Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hotfix Slurm commission performance issue #1996

Merged
merged 21 commits into from Jan 20, 2023
Merged

Hotfix Slurm commission performance issue #1996

merged 21 commits into from Jan 20, 2023

Conversation

ZDu-IDM
Copy link
Collaborator

@ZDu-IDM ZDu-IDM commented Jan 20, 2023

Address ticket
Performance issue: slurm commission is too slow #1829

dependabot bot and others added 9 commits December 9, 2022 11:39
@dependabot
Bump qs from 6.5.2 to 6.5.3 in /idmtools_platform_local/idmtools_webui
8368b55 
Bumps [qs](https://github.com/ljharb/qs) from 6.5.2 to 6.5.3.
- [Release notes](https://github.com/ljharb/qs/releases)
- [Changelog](https://github.com/ljharb/qs/blob/main/CHANGELOG.md)
- [Commits](ljharb/qs@v6.5.2...v6.5.3)

---
updated-dependencies:
- dependency-name: qs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
@dependabot
Bump express in /idmtools_platform_local/idmtools_webui
8af3605 
Bumps [express](https://github.com/expressjs/express) from 4.17.1 to 4.18.2.
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/master/History.md)
- [Commits](expressjs/express@4.17.1...4.18.2)

---
updated-dependencies:
- dependency-name: express
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
@devclinton
Merge pull request InstituteforDiseaseModeling#1983 from Institutefor…
12e5b9f 
…DiseaseModeling/dependabot/npm_and_yarn/idmtools_platform_local/idmtools_webui/express-4.18.2

Bump express from 4.17.1 to 4.18.2 in /idmtools_platform_local/idmtools_webui
@devclinton
Merge pull request InstituteforDiseaseModeling#1976 from Institutefor…
b5a2b11 
…DiseaseModeling/dependabot/npm_and_yarn/idmtools_platform_local/idmtools_webui/qs-6.5.3

Bump qs from 6.5.2 to 6.5.3 in /idmtools_platform_local/idmtools_webui
@ZDu-IDM
Merge remote-tracking branch 'master/dev'
a0ced19
@devclinton
Merge pull request InstituteforDiseaseModeling#1989 from Institutefor…
fce9969 
…DiseaseModeling/release-1.7.x

Release 1.7.3 to master
@ZDu-IDM
Merge remote-tracking branch 'master/release-1.7.x' into fix_slurm_perf
ba7a563
@devclinton
Merge pull request InstituteforDiseaseModeling#1992 from shchen-idmod…
338f428 
…/1.7.3

Add changelog for 1.7.3 release to master
@ZDu-IDM
Get metadata directly from simulatino
9173c00
@shchen-idmod
Copy link
Collaborator

shchen-idmod commented Jan 20, 2023 •
edited

Initial testing with old case (python_sims.py in ticket #1829 looks very promising. Only took 3mins to commission 15000 simulations in QUEST
Commissioning Simulations: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15000/15000 [03:19<00:00, 75.25simulation/s]

@shchen-idmod
Copy link
Collaborator

With Annie's example, new perf data as below: (Only took 18s to commission 800 sims now)
(emodpy_malaria_idmtools) [scj6369@quser23 campaign_sweep]$ python3 perf_example.py -l
INI File Used: /home/scj6369/github/emodpy-malaria/examples/campaign_sweep/idmtools.ini
Creating EMODTask (from files)...
DEBUG: write_config_from_default_and_params invoked with config_path: None, config: True, <function set_config_parameters at 0x2ac8a7809c10>.
For drug Vehicle, could not convert string to float: ''
Generating demographics file demographics.json.

[SLURM_LOCAL]
{
"mode": "local",
"job_directory": "/home/scj6369/example_emodpy_malaria/campaign_sweep"
}
Commissioning Simulations: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 800/800 [00:18<00:00, 43.24simulation/s]
job_id: 1331244
job_directory: /home/scj6369/example_emodpy_malaria/campaign_sweep
suite: 8a341f87-db91-4122-bf7b-3ad9aeefb5db
experiment: c73ce35d-510a-46c6-b720-81235333256d
Waiting on Experiment init_test to Finish running: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 800/800 [05:35<00:00, 2.39simulation/s]
Experiment c73ce35d-510a-46c6-b720-81235333256d succeeded.

shchen-idmod
shchen-idmod approved these changes Jan 20, 2023
View reviewed changes
Copy link
Collaborator

@shchen-idmod shchen-idmod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@shchen-idmod shchen-idmod mentioned this pull request Jan 20, 2023
Closed
@shchen-idmod shchen-idmod merged commit 9cb57ea into InstituteforDiseaseModeling:release-1.7.x Jan 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@devclinton devclinton devclinton approved these changes

@shchen-idmod shchen-idmod shchen-idmod approved these changes

Assignees
No one assigned
Labels
Architecture COMPS Core Hotfix
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@ZDu-IDM @shchen-idmod @devclinton