Master service rates drop the longer it runs

[cloudshao]

When a server is started, it seems to go at a rate of about 5000 requests serviced per hour. After the first two hours, the rate seems to go down drastically. This is the rate history output from Dec 13 (from typing 'rate'):

rates:
(5436, 5089)
(5458, 5447)
(911, 740)
(669, 508)
(648, 497)
(518, 350)
(506, 366)
(306, 279)
connections, responses this hour: 719, 411

This might be due to "error: can't start new thread" that doesn't seem to happen when we first start the server, but appears sometime later.

[cloudshao]

After more investigation this was due to disk writes to the database taking longer and longer as the database grew. As disk writes fell behind, workers were being blocked waiting for writes to finish, resulting in hundreds of threads waiting. We implemented producer-consumer pattern where workers are never blocked and as a result, number of threads do not get out of control. Rates are steady when not trying to run in full-tilt mode.

Closing.