task thrashing

[jesse99]

I don't have a good repro for this, but I have observed CPU usage going through the roof on my web server. It starts out fine: 27 threads (as reported by top) and 0% CPU. Then I click on a thingy which opens up two server-sent events causing 2 tasks to spawn. Threads goes to 29 and CPU to 200%. Then I start up two more SSEs and threads goes to 35 and CPU to 400%. This can go on even further but my Mac starts sounding like a turbine engine at that point.

It's a fairly complex exe with lots of tasks, and tasks that block, but there's nothing special about the SSEs: several are spawned when a client first connects to the server. Haven't been able to glean as much about the state of the world using Apple's Instruments as I'd like, but memory is fine and there don't seem to be any functions or threads sucking up CPU.

This leads me to think that the problem might be lock contention. Or possibly #3095, but I very much doubt that my tasks are doing much of anything at all (and Instruments backs that up).

This was on an 8-core Mac with rust from Sep 22, 2012.

[jesse99]

I'd like to poke at this some more. In particular I was wondering why there were so many threads, though I may wait for #3584 to be fixed.

[brson]

I would be interested in seeing the code.

[jesse99]

It's a fair amount of code but it's here and here. You can in theory build it and run it (use make run-db), but there are quite a few dependencies. However they should all be in cargo (except you need my forks of rust-mustache and rust-socket). If you do that then you should be able to repro the CPU spiking by clicking on the big discs in the browser window it spawns (rub-db isn't perfectly reliable because it kicks off a browser page and then starts the server: sometimes chrome gives up too quickly).

gnos commit 89f2d70b553d170b9d1ac5c10a8cefe336118c9e and rwebserve commit 49cabbf56bc34f9b9acfcc6da0c0c70b4292eba8 are the bad ones. I'm going to try and remove some calls to spawn_sched that aren't doing what I thought.

[jesse99]

Cleaned up the scheduling and it still spikes CPU.

[jesse99]

This turned out to be a bug on my end.

Not sure why Instruments.app didn't clue me in but I looked at the top results more closely and noticed BSD syscalls were crazy high (24 million). dtruss -p <pid> -s -c showed me exactly what was happening: my tcp read code was checking buffer length instead of bytes read length so when the peer closed down my task went into a tight loop constantly reading zero bytes.