sanitycheck doesn't keep my cores busy

[andrewboie]

Describe the bug
There was a recent change to sanitycheck to no longer use a GNU Make jobserver to parcel out tasks, in favor of scheduling jobs directly in the Python code. Unfortunately, this has introduced a performance regression on my build machine with 16 cores/32 threads.

What I am seeing is that sanitycheck will parcel out jobs to the CPUs to build/run, but seems to be waiting for all of them to complete for sending new work, rather than scheduling new work on demand as individual jobs get done.

This specifically seems to break down for test cases that time out. What I am seeing is that my CPU usage will spike up, then drop down to 0 for quite a while, then spike up again.

To Reproduce

1. Modify a test case, or a few test cases, such that they will hang forever instead of completing
2. Run sanitycheck on a machine with lots of cores. On my 32 thread machine, I ran sanitycheck -j48 -p qemu_x86_64 -T tests/kernel
3. Monitor CPU usage during the run. Note periods of extremely low CPU activity while sanitycheck sits there waiting for a test to time out

Expected behavior
CPU cores at or near full utilization during the sanitycheck run, tapering off only at the end when work to do runs out.

Impact
Wasted developer time.
Prolonged CI jobs.

Additional context
I can try to prepare a branch which demonstrates this, if desired.

[carlescufi]

Could this be the fault of the GIL?

[andrewboie]

Could this be the fault of the GIL?

Don't think so, unless the GIL routinely gets held for many seconds at a time.
There seems to be custom job scheduling code, there may be an opportunity to simplify it using ProcessPoolExecutor (see https://docs.python.org/3/library/concurrent.futures.html)

[stephanosio]

Had the same problem when running sanitycheck on my 224-core machines.

I see short bursts of 100% CPU usage; but, for most of the time, it stays at not even 10% -- which is an enormous waste of the expensive CPU time.

At least, it is much better for now to have 14x 16-core machines than single 224-core super machine.

[nashif]

I think I found the issue, looking into it.

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.

[andrewboie]

sigh. this is not stale.

[nashif]

I think I finally got something here, work in progress.