Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Clear gauges after flush...? #95

Closed
recursify opened this Issue Jun 6, 2012 · 11 comments

Comments

Projects
None yet
7 participants

I realize that gauges were designed to be sticky: statsd continues to write the last known value for every flush interval. #62

Is there a reason why you'd want to do this from statsd instead of using Graphite's builtin "Keep Last Value" function? If you do it from statsd, then there is no way to differentiate between receiving no new data and getting the same value over and over. If you're worried about values not propagating to higher retention levels, you can always set the xFilesFactor in storage-aggregation.conf like so:

[stats_gauges]
pattern = ^stats\.gauges.*
xFilesFactor = 0

Personally I think I'm going to patch my local version of statsd to clear gauges after every flush... Thoughts?

recursify added a commit to recursify/statsd that referenced this issue Jun 7, 2012

Owner

mrtazz commented Jun 8, 2012

Wouldn't this make them practically identical to counters?

Similar... counters sum values within the same flush interval. They also write a value of "0" if no data is received within a flush interval. This is reasonable for a true counter, as zero new "things" were counted in that interval. The "Keep Last Value" function doesn't work when you start writing zeros in for no data (it expects None values).

Gauges and counters flush something to graphite, regardless if data was received for that stat (last value seen and zero, respectively). I'm looking for a dumber, more direct access to graphite.... if no data is seen for an interval, I don't want statsd to write anything.

My argument for clearing gauges after the flush interval is that it seems like you can get all the "keep and hold last value" functionality through graphite, so why do it in statsd?

You're also losing information by writing the last value to graphite. Take the example of logging CPU utilization. If a machine suddenly dies and stops sending metrics, you would have no idea. By only writing values when data is received, you could determine (ie: through the graphite json API) "this machine hasn't updated it's CPU utilization in a while... something is up".

Although I'm only considering the Graphite backend... not sure if other backends provide the "keep last value" functionality.

Contributor

Dieterbe commented Jun 8, 2012

if I understand this correctly, this can be paraphrased as:

  • only submit the gauge value once, on each update, because that's really the only thing needed. (with the backends we have now)
  • after submitting, reset (forget about) the gauge value, because given the above, statsd doesn't need to remember the value.

I agree with the above (and personally I wouldn't worry about other backends that are "dumb", we don't even know which other backends we would support, let alone how well they work. we can always add a "cache_gauges" or "explicit_gauges" config option that reintroduces this behavior if we ever need it. In fact, the backend plugin could cache and explicitly resubmit the values itself)

That said, I would like to point to #84 , I think your gauge behavior is the same as the "raw" type in that one. and "raw" is probably the better name for this kind of thing.

That's correct. And yes, it looks like the "raw" type is what I'm describing and is probably a more suitable name.

Contributor

Dieterbe commented Jun 9, 2012

I've thought a bit about it further and realized gauge is the better name. "raw" merely denotes the implementation detail for the graphite backend, if we would have this "raw" type and no "gauge" type, and we would then want to implement this type for a new backend where the implementation is a bit more involved than simple passing on the value (such as periodically writing the same value if the backend doesn't support "keep last value") the name "raw" would not make sense anymore. So I think the better thing to do is keep the "gauge" name and just implement it like how you described, and forget about the "raw" type because it's basically a gauge implemented properly but with a worse (backend-specific) name

mrzor commented Oct 10, 2012

Is there a way to achieve this without using @recursify branch?

The librato backend would also benefit from this improvement, an example of another backend which is not "dumb" about its (possibly reset) inputs.

Would love to use this for graphing deploys and using drawAsInfinite. Right now I am using a counter which will graph no deploy as 0. I would rather have it be null.

mrzor added a commit to mrzor/statsd that referenced this issue Oct 18, 2012

delete untouched meters, timers on flush
deletes gauges on flush (includes recursify's fix for #95)
Contributor

draco2003 commented Apr 10, 2013

Closing this issue since I believe you can get this functionality by enabling the recent deleteGauges config option.

@draco2003 draco2003 closed this Apr 10, 2013

Great, thanks!

TIL that this was the default behavior in StatsD gauges, and it made me 😿. I'm probably not stating anything that hasn't been said before, but to me, gauges represent a rate of change at a point in time. It makes zero sense imho for that to be artificially replicated in the next flush. Not only does this misrepresent what's actually being reported (or not being reported) to the collector, it completely misleads people into believing their running sum is greater than it actually is (e.g. with Graphite's integral).

I don't expect this will change soon; nor should it. If there's one thing I hate almost as much as bad data, it's introducing behavioral regressions for anyone who understands this is the Way Things Are™ and uses it appropriately. Perhaps this should change, but with a major release and blinky tags highlighting it in the release notes.

My $0.02.

@lswith lswith referenced this issue in prometheus/statsd_exporter Oct 20, 2016

Open

Metric buckets are not clearing #21

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment