Hotfix SlutmPlatform memory space issue #2006

ZDu-IDM · 2023-01-27T17:26:56Z

Addressed issues

Improvement
Also improved commission performance further.

New Features

…/1.7.3 Add changelog for 1.7.3 release to dev branch

…/1.7.3.nightly bump dev to 1.7.3.nightly

…DiseaseModeling/release-1.7.x Merge 1.7.4 to master

…/dev_1 Merge release 1.7.4 to dev

# Conflicts: # idmtools_platform_slurm/idmtools_platform_slurm/platform_operations/simulation_operations.py

…to fix_memory

…to fixtest_bump_versions

Bump versions to 1.7.5 and fix tests after code changed

shchen-idmod · 2023-01-27T19:08:59Z

Tested, everything looks good.
Memory dramatically reduced. Tested with 2500 serialized simulations from Anne's script. memory used 2.5G in login node(max memory size is 6G).
Before fix. script failed around 400 sims with over 4G max heapsize

Summary data:
Initializing and commssion took 26mins
Execution: 1:23:24
...
Initializing objects for creation: 99%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 2544/2560 [26:43<00:09, 1.76simulation/s]WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
Commissioning Simulations: 0%| | 0/2560 [00:00<?, ?simulation/s]WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
Commissioning Simulations: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2560/2560 [00:01<00:00, 1741.96simulation/s]
job_id: 1554587
job_directory: /home/scj6369/example_emodpy_malaria/burnin_create_and_use_sweep_larval_habitat1
suite: a5964d87-8b14-4fcc-926d-70c0ccd101cb
experiment: c858fe53-f548-4dea-a164-d78a90d527ec
Waiting on Experiment 4EIR_pickup_PvAd to Finish running: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2560/2560 [1:23:24<00:00, 1.95s/simulation]
Experiment c858fe53-f548-4dea-a164-d78a90d527ec succeeded.

shchen-idmod · 2023-01-27T19:13:31Z

With 640 sims, took 6 mins init/commission and 18mins run:
perf_new_640.txt
Initializing objects for creation: 98%|████████████████████████████████████████████████████████████████████████████████▉ | 624/640 [06:34<00:10, 1.57simulation/s]WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
Commissioning Simulations: 0%| | 0/640 [00:00<?, ?simulation/s]WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
WARNING: During schema-based param purge, Disable_IP_Whitelist not in schema.
Commissioning Simulations: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 640/640 [00:01<00:00, 453.71simulation/s]job_id: 1579154
job_directory: /home/scj6369/example_emodpy_malaria/burnin_create_and_use_sweep_larval_habitat1
suite: d8517fb6-42a1-4b89-949d-974e0fce1fc1
experiment: aaa37eaf-90f8-4a66-a342-052ae750f08a
Commissioning Simulations: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 640/640 [00:15<00:00, 453.71simulation/sExperiment aaa37eaf-90f8-4a66-a342-052ae750f08a succeeded. 100%|████████████████████████████████████████████████████████████| 640/640 [18:43<00:00, 3.28s/simulation]

aaa37eaf90f84a66a342052ae750f08a
Commissioning Simulations: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 640/640 [18:45<00:00, 1.76s/simulation]
Waiting on Experiment 4EIR_pickup_PvAd to Finish running: 100%|████████████████████████████████████████████████████████████| 640/640 [18:44<00:00, 1.76s/simulation]

shchen-idmod and others added 30 commits January 12, 2023 14:02

Merge pull request InstituteforDiseaseModeling#1994 from shchen-idmod…

6bb6ce3

…/1.7.3 Add changelog for 1.7.3 release to dev branch

Bump version: 1.7.3 → 1.7.3+nightly.0

f5aa6ce

Bump version: 1.7.3 → 1.7.3+nightly.0

d10affe

Bump version: 1.7.3 → 1.7.3+nightly.0

1e71911

Bump version: 1.7.3 → 1.7.3+nightly.0

6858411

Bump version: 1.7.3 → 1.7.3+nightly.0

5667cd7

Bump version: 1.7.3 → 1.7.3+nightly.0

bb117f8

Bump version: 1.7.3 → 1.7.3+nightly.0

2f54d2d

bump platform_local to 1.7.3.nightly

b5a986a

Merge pull request InstituteforDiseaseModeling#1995 from shchen-idmod…

238019a

…/1.7.3.nightly bump dev to 1.7.3.nightly

Merge pull request InstituteforDiseaseModeling#1997 from Institutefor…

ad3008d

…DiseaseModeling/release-1.7.x Merge 1.7.4 to master

Merge remote-tracking branch 'master/dev' into Analysis

746b293

Fix memory (also improve time) issue

71318c6

resolve conflicts

57fe980

update more

3a89d9b

Merge pull request InstituteforDiseaseModeling#2003 from shchen-idmod…

9816b5f

…/dev_1 Merge release 1.7.4 to dev

Merge remote-tracking branch 'master/dev' into Analysis

77a8005

# Conflicts: # idmtools_platform_slurm/idmtools_platform_slurm/platform_operations/simulation_operations.py

resolve conflicts

fc463cd

Bump version: 1.7.4 → 1.7.5

69d3d2a

Bump version: 1.7.4 → 1.7.5

d8aa4c8

Bump version: 1.7.4 → 1.7.5

7aa23bc

Bump version: 1.7.4 → 1.7.5

0337c6e

Bump version: 1.7.4 → 1.7.5

2ef8abd

Bump version: 1.7.4 → 1.7.5

9f4533f

Bump version: 1.7.4 → 1.7.5

389aa7c

Bump version: 1.7.4 → 1.7.5

6d197c4

update requirements.txt files

95c12f3

Merge branch 'Analysis' of https://github.com/ZDu-IDM/zdu_idmtools in…

ce56971

…to fix_memory

Remove experiments from suite matadata

efde54a

Merge branch 'Analysis' of https://github.com/ZDu-IDM/zdu_idmtools in…

eec7d8a

…to fix_memory

shchen-idmod and others added 9 commits January 25, 2023 16:45

update version to 1.7.5 and fix tests

7a07361

Add new slurm parameter: constraint

143f370

Merge branch 'Analysis' of https://github.com/ZDu-IDM/zdu_idmtools in…

80a33ad

…to fixtest_bump_versions

Apply kfrey's implementation for new_simulation method

fbae1d5

Merge branch 'Analysis' of https://github.com/ZDu-IDM/zdu_idmtools in…

eb8f3c9

…to fixtest_bump_versions

update test for deep copy change

7127d91

temp comment model tests

6d3620a

Add constraint to platform option

d668f18

make batch_size and max_workers configurable with run method

5441e15

ZDu-IDM requested review from shchen-idmod and devclinton January 27, 2023 17:29

shchen-idmod added 3 commits January 27, 2023 09:37

Merge branch 'Analysis' of https://github.com/ZDu-IDM/zdu_idmtools in…

fd2845c

…to fixtest_bump_versions

enable models tests from previous temp disable

c13f09b

remove

2615824

ZDu-IDM changed the base branch from dev to release-1.7.x January 27, 2023 17:44

ZDu-IDM added 2 commits January 27, 2023 09:47

Merge pull request #6 from shchen-idmod/fixtest_bump_versions

eecf723

Bump versions to 1.7.5 and fix tests after code changed

add constraint to batch file

5ea8b37

shchen-idmod approved these changes Jan 27, 2023

View reviewed changes

devclinton approved these changes Jan 27, 2023

View reviewed changes

shchen-idmod merged commit 15b54ef into InstituteforDiseaseModeling:release-1.7.x Jan 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hotfix SlutmPlatform memory space issue #2006

Hotfix SlutmPlatform memory space issue #2006

ZDu-IDM commented Jan 27, 2023 •

edited

shchen-idmod commented Jan 27, 2023

shchen-idmod commented Jan 27, 2023

Hotfix SlutmPlatform memory space issue #2006

Hotfix SlutmPlatform memory space issue #2006

Conversation

ZDu-IDM commented Jan 27, 2023 • edited

shchen-idmod commented Jan 27, 2023

shchen-idmod commented Jan 27, 2023

ZDu-IDM commented Jan 27, 2023 •

edited