geard stop/delete tasks in rapid order can hang geard

[smarterclayton]

The dispatcher queue is being filled up by stop jobs, and is either hanging in the queue, or systemd itself is hanging. The end result is the gear daemon stops serving API requests. The queue is really small right now (4) so if necessary we can bump it up to get around it for now, but this needs to be debugged.

Jun 02 13:44:54 localhost.localdomain gear[25222]: job START *jobs.StoppedContainerStateRequest, spehM4t_p3GuFvWxSqLsAA: &{Id:demo-lb-1}
Jun 02 13:44:54 localhost.localdomain gear[25222]: job END spehM4t_p3GuFvWxSqLsAA
Jun 02 13:44:54 localhost.localdomain gear[25222]: 2014/06/02 13:44:54 202 2.67ms PUT /container/demo-lb-1/stopped
Jun 02 13:44:54 localhost.localdomain gear[25222]: job START *jobs.StoppedContainerStateRequest, kVWGaWWgHNoAca3L5srEvw: &{Id:demo-backend-1}
Jun 02 13:45:09 localhost.localdomain gear[25222]: alter_container_state: Timeout waiting for stop completion
Jun 02 13:45:09 localhost.localdomain gear[25222]: journal: Done
Jun 02 13:45:09 localhost.localdomain gear[25222]: job END kVWGaWWgHNoAca3L5srEvw
Jun 02 13:45:09 localhost.localdomain gear[25222]: 2014/06/02 13:45:09 202 15018.21ms PUT /container/demo-backend-1/stopped
Jun 02 13:45:09 localhost.localdomain gear[25222]: job START *jobs.StoppedContainerStateRequest, DcWonT51q6uAeKch52t9Zg: &{Id:demo-backend-2}
Jun 02 13:45:24 localhost.localdomain gear[25222]: alter_container_state: Timeout waiting for stop completion
Jun 02 13:45:24 localhost.localdomain gear[25222]: journal: Done
Jun 02 13:45:24 localhost.localdomain gear[25222]: job END DcWonT51q6uAeKch52t9Zg
Jun 02 13:45:24 localhost.localdomain gear[25222]: 2014/06/02 13:45:24 202 15004.78ms PUT /container/demo-backend-2/stopped
Jun 02 13:45:24 localhost.localdomain gear[25222]: job START *jobs.StoppedContainerStateRequest, RpacOJxSpi5AAYQwjeGCUA: &{Id:demo-backend-3}
Jun 02 13:45:24 localhost.localdomain gear[25222]: job END RpacOJxSpi5AAYQwjeGCUA
Jun 02 13:45:24 localhost.localdomain gear[25222]: 2014/06/02 13:45:24 202 2.09ms PUT /container/demo-backend-3/stopped
Jun 02 13:45:24 localhost.localdomain gear[25222]: job START *jobs.StoppedContainerStateRequest, m9nZHM4j6hXTj3W8jwZ3Sg: &{Id:demo-db-1}
Jun 02 13:45:39 localhost.localdomain gear[25222]: alter_container_state: Timeout waiting for stop completion
Jun 02 13:45:39 localhost.localdomain gear[25222]: journal: Done
Jun 02 13:45:39 localhost.localdomain gear[25222]: job END m9nZHM4j6hXTj3W8jwZ3Sg
Jun 02 13:45:39 localhost.localdomain gear[25222]: 2014/06/02 13:45:39 202 15004.99ms PUT /container/demo-db-1/stopped
Jun 02 13:45:39 localhost.localdomain gear[25222]: job START *jobs.DeleteContainerRequest, _jJ-u8mFK7K0nJECi_0JTw: &{Id:replset-db-1}
lines 202-239/239 (END)

Reproduce by quickly stopping and deleting > 4 containers in order.