Convert from fabric to asyncssh. #101

kurtschelfthout · 2022-07-12T16:16:07Z

run_map 20 tasks, 20 concurrent tasks to existing,allocated machines:

Timings with fabric:

[111.38197560000117,
 109.04082189998007,
 107.4787314000132,
 108.30456080002477,
 110.00857060000999,
 109.59617080001044,
 108.20245420001447,
 109.48294259997783,
 110.91391830000794,
 108.26663929998176]
Min: 107.48
Median: 109.26
Max: 111.38

Timings with asyncssh:

Times in secs:
[15.38872220000485,
 14.202121799986344,
 15.369842799991602,
 14.279609999997774,
 15.418468999996549,
 15.286187899997458,
 14.796257799986051,
 15.803920600010315,
 14.016181000013603,
 14.156926900002873]
Min: 14.02
Median: 15.04
Max: 15.80

hrichardlee · 2022-07-12T22:43:50Z

src/meadowrun/run_job_core.py

-                # fabric is supposedly not threadsafe, but it seems to work as long as
-                # more than one connection is not being opened at the same time:
-                # https://github.com/fabric/fabric/pull/2010/files
-                result_future: asyncio.Future = asyncio.Future()


so nice to get rid of this!!

- scp on a NamedTemporaryFile that's still open doesn't work--we need to close the file and then delete it manually later - Retry on OSError when trying to make an SSH connection

- scp on a NamedTemporaryFile that's still open doesn't work on Windows--we need to close the file and then delete it manually later - Retry on OSError when trying to make an SSH connection

Convert from fabric to asyncssh.

1871a0a

kurtschelfthout force-pushed the asyncssh branch from ae34616 to 1871a0a Compare July 12, 2022 20:57

kurtschelfthout merged commit d2c2c9e into meadowdata:main Jul 12, 2022

hrichardlee reviewed Jul 12, 2022

View reviewed changes

kurtschelfthout deleted the asyncssh branch July 13, 2022 09:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert from fabric to asyncssh. #101

Convert from fabric to asyncssh. #101

kurtschelfthout commented Jul 12, 2022 •

edited

Loading

hrichardlee Jul 12, 2022

Convert from fabric to asyncssh. #101

Convert from fabric to asyncssh. #101

Conversation

kurtschelfthout commented Jul 12, 2022 • edited Loading

hrichardlee Jul 12, 2022

Choose a reason for hiding this comment

kurtschelfthout commented Jul 12, 2022 •

edited

Loading