Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multitasking (EWMS_PILOT_CONCURRENT_TASKS) [minor] #199

Draft
wants to merge 100 commits into
base: main
Choose a base branch
from
Draft

Multitasking (EWMS_PILOT_CONCURRENT_TASKS) [minor] #199

wants to merge 100 commits into from

Conversation

ric-evans
Copy link
Member

@ric-evans ric-evans commented Jun 7, 2023 •
edited
Loading

closes #194

@ric-evans ric-evans added enhancement New feature or request CI / Testing About CI and/or testing labels Jun 7, 2023
@ric-evans ric-evans self-assigned this Jun 7, 2023
@ric-evans
Copy link
Member Author

Currently, when running 3 concurrent tasks, the memory is 3x:

non-parallel:

2023-06-08T18:50:25.4251190Z                total        used        free      shared  buff/cache   available
2023-06-08T18:50:25.4251491Z Mem:         7110608     1633212      134272     1387024     5343124     3777236
2023-06-08T18:50:25.4251757Z Swap:        4194300        1292     4193008

3 concurrent tasks:

2023-06-08T18:50:01.6157013Z                total        used        free      shared  buff/cache   available
2023-06-08T18:50:01.6157520Z Mem:         7110604     2641800      175848     1381664     4292956     2776052
2023-06-08T18:50:01.6231001Z Swap:        4194300        3352     4190948

wipacdevbot and others added 30 commits September 26, 2023 16:12
@wipacdevbot
<bot> update dependencies*.log files(s)
d5ca7b6
@ric-evans
Revert "use --ipc="host""
e251147 
This reverts commit 26a9f16.
@ric-evans
use --shm-size=16gb
7e93b79
@ric-evans
fix f-string
313ec03
@ric-evans
use --shm-size=32gb
6b9577d
@ric-evans
temp - ls /dev/shm outside and inside container
ab97c02
@wipacdevbot
<bot> update dependencies*.log files(s)
807223b
@ric-evans
temp - ls /dev/shm outside and inside container - 2
f55de80
@ric-evans
Revert "Revert "use --ipc="host"""
9e738f8 
This reverts commit e251147.
@wipacdevbot
<bot> update dependencies*.log files(s)
d11c836
@ric-evans
Merge remote-tracking branch 'origin/main' into multitasking
74c71af
@ric-evans
misc
5f9b311
@ric-evans
add client/prepare.py and use as init_cmd
ea69258
@ric-evans
clean up if __name__ == "__main__" blocks and Done logging
32067c1
@ric-evans
fix call
f8b0da2
@wipacdevbot
<bot> update setup.cfg
36ab7b1
@ric-evans
use WIPACrepo/wipac-dev-py-setup-action@v2.6
6a499a1
@wipacdevbot
<bot> update setup.cfg
35e3f40
@wipacdevbot
<bot> update dependencies*.log files(s)
4588914
@wipacdevbot
<bot> update dependencies*.log files(s)
10a29e0
@ric-evans
rename to client/pilot subprocesses stderr & stdout
fe8b558
@ric-evans
add sleep for non parallel case
5ff47a2
@ric-evans
add sleep for non parallel case - 2
509c473
@ric-evans
use tasks_concurrency (yes/no) for setting task concurrency
b587266
@ric-evans
add sleep for non parallel case - 3
c3f2a05
@ric-evans
use multiclient/multitask
7c7dc14
@ric-evans
adjust logging & remove LOGGER.*->
51ebb6b
@ric-evans
ls /dev/shm after init
6b25f1d
@ric-evans
add logging around I3PhotoSplineService startup
5238055
@ric-evans
ls /dev/shm after init (at top of reco)
75dfa3e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@ric-evans ric-evans

Labels
CI / Testing About CI and/or testing enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Enable Multitasking Parallelism
2 participants
@ric-evans @wipacdevbot