Incredibly high CPU load during test running

[bhallstein]

Hi, I'm encountering some strange behaviour. I'm using an (I think) fairly standard ava + enzyme + babel setup.

I initially noticed my machine was running poorly — almost totally locked up — while ava tests were running. Investigating, I was astonished to find that CPU load was continuously over 100 during test running. The machine is so intensely locked up that Activity Monitor won't refresh any of its CPU or memory info until ava has finished running the tests. That 100 figure isn't a percentage, that's the actual CPU load figure, as a % it would be 10,000% CPU load. I thought I must be dreaming it, but it really is true.

My first thought was to try without concurrency (-c 1), and that results in CPU load around 1 (100%) as expected. Test running takes ~40s with -c 1 and CPU load of about 1, compare with ~30s without -c and CPU load of about 100, system all but unusable.

Thinking about this, and 3 things seem possible explanations:

1. ava is getting the number of CPU cores available on the machine wrong, spawning too many threads/processes.
2. a 'thread bomb' situation, where lots of threads are spawned continuously.
3. inefficient inter-process communication is occurring. I wondered if poor multithreading support in one of the libraries, e.g. Enzyme, could have this effect? Or if there is a locking system for non-multithreaded shared libraries or some part of the test execution process that might be spin locking?

Or it could be something else. As I've never seen anything quite like this before, I'd really appreciate anyone's thoughts if they have further ideas.

Code: link to repo

• Tests are in src/test
• package.json
• npm run t is simply ava src/test/*

The machine is a MacBook Air, 1.3GHz, 2 real cores and SMT.

[bhallstein]

To try and get more info about what's going on, I just tried a few different levels of concurrency:

• without -c: 30s, CPU load ~= 100
• -c 1: 40s, CPU load ~= 1
• -c 2: 30s, CPU load ~= 50
• -c 4: 30s, CPU load ~= 100 (perhaps a little lower than with concurrency auto-set, it's a bit hard to tell)

This suggests first that ava might be spawning too many processes: perhaps the number of logical rather than physical cores, while the workload does not spread well over SMT cores. However, even that wouldn't fully explain such high load figures: the jump from 1 at -c 1 to 50 at -c 2. What I think this suggests is that there is also some kind of inter-process communication issue, perhaps such as as busy waiting, but I could be very mistaken.

ava is by far my favourite testing library and I'd love to carry on using & recommending it! For now I have disabled concurrency on this test suite.

[novemberborn]

@babel/register might result in each worker process recompiling the code (because IIRC @babel/register compiles the first time a file is loaded in that process). That'll hit your CPU.

We use node -pe 'os.cpus().length' to determine concurrency. What is that on your node machine?

[bhallstein]

Thanks @novemberborn. os.cpus().length returns 4, which roughly gels with what's above. I wonder if using the physical number of cores would dramatically improve efficiency without losing much (any?) performance on machines with SMT. Looks like that can be queried but it's not exactly pretty.

Thanks again, I'll see if I can get some figures on @babel/register's contribution to this.

[novemberborn]

Looks like that can be queried but it's not exactly pretty.

No, that looks pretty bad.

For I/O heavy tests, using all (hyper-threaded) cores can work well. But if the tests are extremely CPU heavy that might backfire. I suspect that's relatively rare though.

(I'm closing this issue for housekeeping purposes, but let's keep the conversation going.)

[XhmikosR]

I'm also hitting what I think is the same issue, I noticed it on xojs/xo#606 (comment).

Basically whatever value of concurrency bigger than 1 spawns too many processes making the system unresponsive...

I'm using a VM with 4 threads, but even if I use -c=2 I'm getting tons of spawned processes.

[novemberborn]

@XhmikosR that looks like the processes aren't shutting down after tests are complete. Could be due to XO's test setup though. Perhaps see if you can reproduce this with a fresh project and perhaps 10 test files, each with a single passing test?

[XhmikosR]

They do shut down, but the problem is that too many processes are spawned initially... Anything other than using --serial or -c 1 has this.

I'd be happy to try to look into it more, but I don't have a fresh project to test. I just came across this while having to run xo's tests locally.

[novemberborn]

These are XO CLI processes, I haven't checked but it's likely that the test suite runs those in child processes:

[bhallstein]

That's very curious.

Re my original issue, I do often see CPU load of around 100; though I see this with a lot of node/npm related tasks, such as parcel builds, npm install, and so on, so I think it's unlikely it has much to do with ava.

[XhmikosR]

No, this is clearly due to ava. I only ran ava when I took the screenshot.