Monitoring of the logplex cluster revealed that post this change set the binary memory usage climbed. IMO this doesn't make sense because the drain buffer process will still retain references to any binaries cleaned up by forcing a hibernate here. But, graphs do not lie. This commit reverts the changes in an attempt to reclaim that binary memory. For reference this original commit that converted these b551562
This test no longer relies on flaky timing and waiting for mocked calls to trigger the next series of events. Instead the events are all sent at once and the mocked out http server controls when the state flips from a 500 status to a 200 status.
The shrink test incorrectly asserted the old behaviour of resizing the queue as soon as socket is connected. Once the test was fixed I discovered and fixed an issue in the logplex_http_drain code that still set the service level back to normal as soon as the socket is connected instead of when a successfull HTTP request is sent.
While fixing a failing test I noticed that the drain buffer would cycle between the shrunk sizing (10) and the default size (1024) rapidly between reconnect attempts. This is an edge case surrounding a drain that has never connected before. I altered the last_good_time atom to be more clear in it's intension and simplified the maybe_shrink code a little.