Performance issue: slurm commission is too slow

[shchen-idmod]

Running this in our slurm cluster: 10.24.15.109
with example: https://github.com/InstituteforDiseaseModeling/idmtools/blob/dev/examples/native_slurm/python_sims.py

Change sweep a with 30, b with 500:
builder.add_sweep_definition(partial(param_update, param="a"), range(30))
builder.add_sweep_definition(partial(param_update, param="b"), range(500))
...
dry_run=True:
suite.run(platform=platform, wait_until_done=True, max_running_jobs=10,
retries=5, dry_run=True)

run example:
Commissioning Simulations: 5%|███▌ | 768/15000 [11:04<19:53, 11.92simulation/s]
It about commission 500 sims/10mins. I think this is way too slow. To finish commission15000 sims, may need at least 5 hours.

[shchen-idmod]

---

Meanwhile comps only took 5 mins to commssion for same amount of simulations
C:\venv\idmtools_dev_173\Scripts\python.exe C:/github/idmtools/examples/native_slurm/python_sims_comps.py
INI File Used: C:\github\idmtools\examples\idmtools.ini

[SlurmStage]
{
"endpoint": "https://comps2.idmod.org",
"environment": "SlurmStage"
}
Uploading 1 files/613 bytes
Uploading files: 100%|██████████| 1/1 [00:01<00:00, 1.35s/file]
Initializing objects for creation: 0simulation [00:00, ?simulation/s]
The created experiment can be viewed at https://comps2.idmod.org/#explore/Simulations?filters=ExperimentId=5992db17-6697-ed11-92f2-f0921c167864
Simulations are still being created

Creating Simulations on Comps: 100%|██████████| 15000/15000 [05:38<00:00, 44.31simulation/s]

[shchen-idmod]

This may relate to perf issue Anne from NU seeing:

[shchen-idmod]

her example:
perf_example.txt

[ZDu-IDM]

Have discussion with Clinton and it seems like there are several possible reasons that may cause the commission performance issue. We are investigating the issue now but not sure when to get it fixed (as well-known, performance issue is always hard to debug and fix).

[shchen-idmod]

See update perf number in ticket #1996. Fix seems very fast now.

[ZDu-IDM]

Done with idmtools release 1.7.4