Resolve graphite growth problem, introduce hopper 0.1.2 #181
Conversation
We noticed that there was something fishy going on with graphite. In particular, the number of lines push to the programmable_filter grew _way_ too quickly for the inputs. It turns out, that fancy line buffer we pass in to avoid allocing small strings repeatedly was not being cleared and so everything that came in would be appended there. This commit also includes the work done as a part of postmates/hopper:#5 by updating to 0.1.2. All messages will be delivered in order--as they should have been--and hopper is now quite a bit faster to boot as it can keep cramming in low-memory when there's space. Signed-off-by: Brian L. Troutwine <blt@postmates.com>
|
This resolves #181 |
| loop { | ||
| time::delay(attempts); | ||
| match recv.next() { | ||
| None => attempts += 1, | ||
| Some(event) => { | ||
| attempts = 0; | ||
| match self.process(event) { | ||
| Ok(events) => { | ||
| for ev in events { |
There was a problem hiding this comment.
Wow..talk about nonintuitive.
There was a problem hiding this comment.
This one was tricky! This change here avoids having to continually alloc a new vector for every call to self.process, worth +/- 10ns per call. This is safe because we're calling drain on the vector. That'll remove all the values stashed in it.
But! A similar change to pass in a string buffer–which I'll point out–did not have something similar.
| .map(|m| metric::Event::new_log(*m)) | ||
| .chain(pyld.metrics.into_iter().map(|m| metric::Event::new_telemetry(*m))) | ||
| .collect()) | ||
| for lg in pyld.logs { |
There was a problem hiding this comment.
Can you describe the changes here and why they were made?
There was a problem hiding this comment.
Absolutely! What this would do previously is two things:
- alloc a new vector filed with
metric::Eventand - stash that vector inside a
Result
To avoid the first step we're now passing in a pre-allocated vector res. This means we've got to explicitly push metric::Events into the vector. The successful return type is now Ok(()), which () is the unit type.
| @@ -90,6 +90,7 @@ fn handle_stream(mut chans: util::Channel, tags: Arc<metric::TagMap>, stream: Tc | |||
| &mut chans, | |||
| metric::Event::Telemetry(Arc::new(Some(m)))); | |||
| } | |||
| line.clear(); | |||
There was a problem hiding this comment.
@pulltab Here's the culprit. Without these clears the line buffer will grow and grow.
We noticed that there was something fishy going on with graphite.
In particular, the number of lines push to the programmable_filter
grew way too quickly for the inputs. It turns out, that fancy
line buffer we pass in to avoid allocing small strings repeatedly
was not being cleared and so everything that came in would be
appended there.
This commit also includes the work done as a part of
postmates/hopper:#5 by updating to 0.1.2. All messages will be
delivered in order--as they should have been--and hopper is now
quite a bit faster to boot as it can keep cramming in low-memory
when there's space.
Signed-off-by: Brian L. Troutwine blt@postmates.com