.circleci: Switch to dynamic MAX_JOBS #44729
Conversation
Switches our MAX_JOBS from a hardcoded value to a more dynamic value so that we can always utilize all of the core that are available to us Signed-off-by: Eli Uriegas <eliuriegas@fb.com> [ghstack-poisoned]
| # This script is run inside Docker.2XLarge+ container that has 20 CPU cores | ||
| # But ncpu will return total number of cores on the system | ||
| export MAX_JOBS=18 | ||
| export MAX_JOBS=$(( $(nproc) - 2 )) |
There was a problem hiding this comment.
I thought nproc was suppose to return # of available CPU on the host instead of the container? @malfet
There was a problem hiding this comment.
We had a conversation about it and we think the issue with the jobs stemmed from the use of stdbuf and not necessarily from the nproc invocation
There was a problem hiding this comment.
@seemethere can you please delete comments on lines 8-9 then?
@walterddr yes, that is correct, but actually spawning more build jobs than number of cores is fine.
| # This script is run inside Docker.2XLarge+ container that has 20 CPU cores | ||
| # But ncpu will return total number of cores on the system | ||
| export MAX_JOBS=18 | ||
| export MAX_JOBS=$(( $(nproc) - 2 )) |
There was a problem hiding this comment.
@seemethere can you please delete comments on lines 8-9 then?
@walterddr yes, that is correct, but actually spawning more build jobs than number of cores is fine.
Switches our MAX_JOBS from a hardcoded value to a more dynamic value so that we can always utilize all of the core that are available to us Signed-off-by: Eli Uriegas <eliuriegas@fb.com> [ghstack-poisoned]
|
@seemethere alternatively, perhaps we could try installing |
There was a problem hiding this comment.
LGTM. According to this issue, the available cpus within a docker from the file /sys/fs/cgroup/cpuset/cpuset.cpus.
Codecov Report
@@ Coverage Diff @@
## gh/seemethere/23/base #44729 +/- ##
=========================================================
- Coverage 68.08% 68.08% -0.01%
=========================================================
Files 384 384
Lines 49768 49768
=========================================================
- Hits 33885 33883 -2
- Misses 15883 15885 +2
Continue to review full report at Codecov.
|
|
@seemethere merged this pull request in 6006e45. |
Summary: Pull Request resolved: #44729 Switches our MAX_JOBS from a hardcoded value to a more dynamic value so that we can always utilize all of the core that are available to us Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Test Plan: Imported from OSS Reviewed By: walterddr Differential Revision: D23759643 Pulled By: seemethere fbshipit-source-id: ad26480cb0359c988ae6f994e26a09f601b728e3
6 participants
Stack from ghstack:
Switches our MAX_JOBS from a hardcoded value to a more dynamic value so
that we can always utilize all of the core that are available to us
Signed-off-by: Eli Uriegas eliuriegas@fb.com
Differential Revision: D23759643