Autodetect number of jobs with --jobs auto#605
Conversation
|
|
||
| def _auto_detect_cpus(): | ||
| try: | ||
| return len(os.sched_getaffinity(0)) |
There was a problem hiding this comment.
This return the number of CPUs the current process is restricted to on Linux.
There was a problem hiding this comment.
What's wrong with os.cpu_count()?
There was a problem hiding this comment.
On Linux len(os.sched_getaffinity(0)) returns the number of CPUs available to the current process, where as os.cpu_count() will return the total number of CPUs on the host system which might be more than the current process can use.
There was a problem hiding this comment.
answering my question - it's the number of available cores (not clear whether this includes hyperthreading cores).
More details: https://stackoverflow.com/questions/1006289/how-to-find-out-the-number-of-cpus-using-python/14840102
There was a problem hiding this comment.
not clear whether this includes hyperthreading cores
As far as I know this includes hyperthreading cores. On my dual-core macbook --jobs auto will result in 4 parallel jobs.
rchen152
left a comment
rchen152
left a comment
There was a problem hiding this comment.
Thanks! Just a few nits related to the Google style guide.
Fixed in 4e48291 |
4 participants
This adds the possibility to autodetect the number of jobs based on the number of available CPU cores on the host system using
pytype --jobs auto. This allows for a usage similar topytest-xdistwhich can be useful when usingpytypein scripts that might run on different systems.