Overloading the driver

[pmembrey]

Hi,

The process that actually writes to the database can be easily over loaded. If MongoDB is busy or slow for whatever reason, the messages can back up in the process' queue to the point where it cannot recover. Even a short burst can cause this problem and although MongoDB recovers almost immediately, the driver process (due to the selective receives) never recovers to the point where it can clear out the queue resulting in the app ultimately crashing.

When this occurs, the throughput goes down from around 6,000 messages per second (and being able to handle spikes of 10,000 with no problem) to about 30 per second - far less than the number coming in, which of course ultimately makes things even slower.

If I kill the producer, the process will recover eventually but it can take hours for that to happen.

Not sure what additional information to provide but I think it's caused by a selective receive and the message queue growing too fast.

Cheers,

Pete