[Windows] emulate UNIX os.getloadavg()

[giampaolo]

I'm not sure I understand whether we can really emulate the UNIX counterpart, but this looks interesting:
https://bugs.python.org/issue34060
python/cpython@e16467a
https://github.com/python/cpython/blob/master/Lib/test/libregrtest/win_utils.py

Part of the confusion is that I currently don't really understand how getloadavg() should be interpreted (need to read some literature) . My hope is to land something like this:

def getloadvg():
     if POSIX:
        return os.getloadavg()
    else:
        return _windows_impl()

...but returning 1 value (on Windows, average since last call) vs. 3 values (on UNIX, representing the average of last 1, 5, 15 minutes) maybe suggests these are 2 different things so we should call it differently?

[giampaolo]

CC-ing @ammaraskar just in case he wants to provide advice/direction.

[ammaraskar]

Hey, it's entirely possible to do the 3 values.

The best literature to read would definitely be: https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation which explains really well how the value is calculated.

Essentially its the weighted moving average of the number of processes in the run queue of the OS over 1 minute, 5 minutes and 15 minutes.

You can find Linux's load average calculation here: https://github.com/torvalds/linux/blob/345671ea0f9258f410eb057b9ced9cefbbe5dc78/include/linux/sched/loadavg.h#L18-L38 which is what is emulated in https://github.com/python/cpython/blob/master/Lib/test/libregrtest/win_utils.py#L97

As the Linux comment eludes, the constants can be calculated by:
1 / exp(sampling interval in seconds / window size in seconds)

for example, in CPython we only care about the 1 minute window and we sample every 5 seconds, so the constant comes from:
1 / exp(5 / 60) = 0.92004441462

You can calculate out the constants for the other windows (5 minutes and 15 minutes) and track those separately as well like Linux does in order to fully emulate it: https://github.com/torvalds/linux/blob/345671ea0f9258f410eb057b9ced9cefbbe5dc78/kernel/sched/loadavg.c#L356-L358

Hope this helps and lemme know if you have any other questions.

[ammaraskar]

Oh also, I should say the calculations are the easy part. Getting the actual number of processes in the run queue for windows is an absolute pain and we ended up running the command:

typeperf "\System\Processor Queue Length" -si 5

in order to grab the value. There's some win32 apis that can achieve the same thing but are way more involved in the C side.

[giampaolo]

Thanks for the info @ammaraskar. It appears this could theoretically be good for inclusion but I suspect that running typeperf in a subprocess would be problem as it'd be returned by Process(os.getpid()).children()) and killed by reap_children() on each unit-test run. I suppose there's no way other than using a subprocess or thread (see "using native Win APIs")?

There's some win32 apis that can achieve the same thing but are way more involved in the C side.

Do you know which ones?

[ammaraskar]

You can find the C based implementation I did for CPython here python/cpython@master...ammaraskar:windows_load

Internally, this windows API spins up a thread to do the sampling.

[giampaolo]

Sweet, thanks. I'll take a look at that and I will probably ping you later on with some questions. =)

[giampaolo]

Some questions:

1. other than LOADAVG_FACTOR_1F 0.9200444146293232478931553241 do you know the magic number for 5F and 15F intervals?
2. should I register 3 different callbacks (RegisterWaitForSingleObject), 1 for each value?
3. I copy/pasted your code in a branch (worked immediately):

• diff: https://github.com/giampaolo/psutil/compare/win-getloadavg?expand=1
• C code: https://github.com/giampaolo/psutil/blob/b68b6dc48174ba5c37cad2b96333a23ea7c7f999/psutil/arch/windows/wmi.c
  I wonder if you're interested in contributing / authoring this yourself (I can also do it with your guidance but since this is your code... =))

[ammaraskar]

1. The constants would be:

1 minute: 0.92004441462932324789315532405371723167318753495997420170
5 minutes: 0.659240630200443746254760411013414882365778498066511225174
15 minutes: 0.286504796860190100324885426647837602793150792328251047088

2. Nah, just one callback will do, only thing that changes is the constant for the calculation

3. Sure! I would love to contribute! Just gimme a few hours to get caught up with the project.

[giampaolo]

Cool, thanks. To quickly set you up:

• git clone ...; cd psutil
• make.bat setup-dev-env # install deps
• make.bat test foo.py (if you want to test/call the new function via a script (import psutil; print(psutil.getloadavg()))
• long version (but I suppose you don't need it): https://github.com/giampaolo/psutil/blob/master/docs/DEVGUIDE.rst

[giampaolo]

...last: you may want to change the Python path in make.bat:
https://github.com/giampaolo/psutil/blob/master/make.bat#L22-L28