-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use /proc/loadavg on Linux #2218
Comments
To have some numbers, I have a Jenkins slave with 64 cores. When pushing to a repository, multiple jobs are triggered. Those jobs are independent, and there is no easy way to have a common "jobserver" to limit the number of parallel tasks. All the makes/ninjas are run with "-j 64 -l 128". With GNU Make 4.4 the peak 1 minute average load is 132, and the peak memory usage 55 GiB. With GNU Make 4.4 it takes a 1.7% less time to complete (clean builds). |
I'm not creating a PR because I'm not sure about the licence implications of me "basing" it on the GPL make patch. Maybe /proc/loadavg should be kept open the whole time instead of opening and closing in continuously, the parsing could maybe be made more robust, etc. But this does the trick.
0001-Use-proc-loadavg-on-Linux.log I'm using it in production and my builds are now a 6.7% faster than without the patch (plus the OOM killer never kills them). |
There is none if you don't the copy code. If you just take the idea and reimplement it in a different program then that is fine. |
…d#2218) The /proc/loadavg file includes the traditional 1-, 5- and 15-minute loads plus an instantaneous count of processes that are ready to run. This lets us respond faster to new processes being launched by other processes.
…d#2218) The /proc/loadavg file includes the traditional 1-, 5- and 15-minute loads plus an instantaneous count of processes that are ready to run. This lets us respond faster to new processes being launched by other processes.
Yeah, this would be pretty convenient. Given that ninja still doesn't support the GNU jobserver (#1139) , supporting |
GNU Make uses /proc/loadavg on Linux to decide whether to start another task: https://lwn.net/Articles/913253/ / https://git.savannah.gnu.org/cgit/make.git/commit/src/job.c?id=d8728efc80b720c630e6b12dbd34a3d44e060690. I would like Ninja to do the same.
Since /proc/loadavg provides "number of processes currently runnable (running or on ready queue)" (https://www.kernel.org/doc/html/latest/filesystems/proc.html), Ninja would be able to take better decisions than by looking only at the info provided by getloadavg().
The text was updated successfully, but these errors were encountered: