Job submisison workflow

[arabnejad]

The current multi-threading implementation for job submission is not thread safe when we use high number of threads. The main issue is : some global variables modified by different threads are not protected by mutex lock, therefore the output values of those variables are not predicable and not set as they should be

the possible solutions for this issue are:

• adding those variable blocks in a mutex lock to be protected
• working on a different approach which may requires some changes in the workflow of how we submit a job to remote machine

the main disadvantage of first approach is : due to have more mutex lock area, the total job submission will be increased,
so, I believe the second option is the best one to investigate

[arabnejad]

In the current version, when we call the job() function to submit a job, all steps from preparation, transmission and submission are executed in a single job() function. Even with the current multithreading approach, we still need to have some mutex lock to protect global evn variables in different threads which increase the total execution time of job() function.

Therefore, I am suggesting to split those steps as :

1. job_preparation() : where all the job folders and scripts will be created in the temporary folder (/tmp) <tmp_folder>/{results/,scripts/} . In this function, the config_files folder is already transferred to the remote machine by calling with_config function

   • this part can be executed in parallel. In case of using Multithreading, we need to have some local copy of env variables for each thread. By using Multiprocessing there is no change needed compare to serial mode

2. job_transmission() : transfers all generated files/folders from <tmp_folder>/{results/,scripts/} to <target_work_dir>/{results/,scripts/} with a single rsync command.

3. job_submission() : submits all job scripts to the target remote machine.

   • this function can be also executed in parallel. However, I think having multiple simultaneous SSH tunnels will not be safe due to the system limitation on open/alive SSH connection. Therefore, I think this part should be done in a serial mode.

[arabnejad]

the new job workflow system are implemented with both Multithreading and Multiprocessing approaches

• Multithreading is implemented in branch new_job_submission_workflow
• Multiprocessing is implemented in branch new_job_submission_workflow_v2

Both are tested for thread/process safety.

Also, I did some benchmarking on new job submission workflow and here are the results compared to the current version

Here, PJ refers to the PilotJob functionality, i.e, in case of PJ=True only a single job will be submitted, and if PJ=True each job will be submitted individually to the remote machine scheduler

[arabnejad]

This issue closed by Pull Request #196