Dynamic buf size #62

Merged
merged 6 commits into from Oct 30, 2013

2 participants

@ferd

Most of the messages in drains in production systems turn out to be
stored for disconnected drains for a given period of time, and will thus
have a real cost on memory for the overall node.

This pull request makes drain buffers shrink down to 10 entries when
they are disconnected for 5 minutes or more, and go back up in size
once they can properly connect.

All drain losses should be accurately reported for the entire time.

ferd added some commits Oct 24, 2013
@ferd ferd Dynamic resize
If a buffer is inactive for too long, its size gets shrunk to 10 instead
of 1024, and gets re-established when reconnecting.
3ce9cf3
@ferd ferd Fix buffer resizing bug
When a buffer gets resized and needs to drop data, the data is dropped,
but the maximal value of the buffer is never brought down. This patch
makes it so the function works as expected.

This fix also removes all the unused conversion functions that were used
in a past deploy.
07d80fc
@ferd ferd Introduce ENV variable for tcp drain buffer size bfa68f6
@ferd ferd Add tests for TCP drain resizing 6cd725a
@ferd ferd Force shrinking on never-connected drains
Only a successful connection will grow the drain size.
4b39434
@ferd ferd shrinking of HTTP drains
Inactive drains (disconnected for too long) get shrunk down, and resized
when going back online.
a4f2df5
@dpiddy dpiddy commented on the diff Oct 29, 2013
src/logplex_http_drain.erl
@@ -39,7 +41,10 @@
client :: pid(),
out_q = queue:new() :: queue(),
reconnect_tref :: reference() | 'undefined',
- drop_info :: drop_info() | 'undefined'
+ drop_info :: drop_info() | 'undefined',
+ %% Last time we connected or successfully sent data
+ last_good_time :: 'undefined' | erlang:timestamp(),
@dpiddy
Heroku member
dpiddy added a line comment Oct 29, 2013

Is erlang:timestamp() the same as os:timestamp()? Or is it the bad one?

@tsloughter
tsloughter added a line comment Oct 29, 2013

What? I don't think there is an erlang:timestamp()... os:timestamp/0 is what should be used.

@tsloughter
tsloughter added a line comment Oct 29, 2013

OOOOO, haha. That is the datatype. Nevermind.

@dpiddy
Heroku member
dpiddy added a line comment Oct 29, 2013

Ah ok, I see now. Sorry for the noise!

@omarkj
omarkj added a line comment Oct 29, 2013

So use os:timestamp/0 not now/0.

@ferd
ferd added a line comment Oct 29, 2013

erlang:timestamp() in this context is the type specification for the time format used by functions such as now() or os:timestamp() ({MegaSecs, Secs, MicroSecs}) so that using the Dialyzer application, we can type check that our code uses the proper time format.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@omarkj

+1 from me.

@ferd ferd merged commit 843447f into master Oct 30, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment