SSH error connection when submitting a large number of ensemble runs

[arabnejad]

FabSim3 provided a multi-threading functionality to decrease the total job submission time for a large number of ensemble/replica
however, we the ensemble runs are really high (>30k) due to a high number of SSH connection, the submission process may FAILED

    ...
    raise SSHException("SSH session not active")
paramiko.ssh_exception.SSHException: SSH session not active

there are a number of ways that we can handle this issue, but there will be degradation on total submission time

what do you think @djgroen ? what are your suggestions to tackle this issue?

[djgroen]

Okay, I think this is one part that is responsible for all the SSH connections. There are three run commands right after this line:

FabSim3/base/fab.py

Line 596 in 510dc58

if env.label in ['PJ_PYheader', 'PJ_header']:

Any command with run() in job will scale the number of SSH connections by the job count. These need to be merged such that file staging is done for all jobs (or at least multiple jobs) in one go, rather than separately for each job.

Later on there is also this line:

FabSim3/base/fab.py

Line 659 in 510dc58

run(template("chmod u+x %s" %

That line changes permission for one file for each job. That one may actually be straightforward to refactor, because I think you can just chmod all subdirs in that results dir with a single command at the right time (i.e. after everything is uploaded)?

Lastly, there are several run commands after this line

FabSim3/base/fab.py

Line 696 in 510dc58

# We don't want to go next during replicas and pjm until the final

but those are merged in the case of QCG-PJ ensembles, so I think that part can remain unchanged.

Does this help things @arabnejad ?

[arabnejad]

I reduced the number of SSH connection during the job submission (c8cfadc, d162081)

Federica (@FGugole) could you please confirm that new implementation fixed your problem
also, could you please mention the campaign size that you used, and the total submission time

[FGugole]

Yes, the new implementation fixed my problem and I was able to submit a campaign with 60k samples. The job submission took a long time (at least 10 days), but it was successful!