Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

"timing" has nothing to do with timing #98

Closed
Dieterbe opened this Issue Jun 11, 2012 · 34 comments

Comments

Projects
None yet
Contributor

Dieterbe commented Jun 11, 2012

the metric type "timing" has a misleading choice of a name, because it doesn't have anything to do with timings per se (although timings may be a common, or even the most common use of it).

fundamentally, this metric takes multiple numbers in a short time interval, and calculates the avg/percentile/min/max. that's it.

let's try to devise a better name.
here's a suggestion: analyzer (or interval-analyzer but the fact that it operates on a time interval is fairly obvious as statsd is an aggregator, and can be explained in the metric type's documentation)

Owner

mrtazz commented Jun 11, 2012

The name derives from the fact that you record timings of an action in ms with it. So I think the name is fairly adequate to what you actually capture with it. And I think analyzer is a lot more confusing than timing.

I've been quite annoyed with timing as well. And while I'm at it - that timing and gauges are put in sub-categoies in whisper, but counters aren't.

Contributor

Dieterbe commented Jun 12, 2012

@mrtazz you are making the assumption that people only use it for timings. my point is it can be used for so much more.
for example look what this guy is doing with the "timing" metric http://railstips.org/blog/archives/2011/03/21/hi-my-name-is-john/ (go to "Fun Use Case").
@msiebuhr what do you use the timing metric for, other than timings?

While we don't have statsd in production yet, we intend to watch the size-/responsetime-distributions of various types of requests in our systems.

So far, we've been putting performance-data like that into the syslog, where we easily can look at both distribution and grand totals; so far, it seem that failures are better correlated to spikes in distribution, rather than the grand totals.

Owner

mrtazz commented Jun 12, 2012

@Dieterbe yeah, but if I don't misunderstand what he's doing, that would be a use case for gauges now and not something to use timings for. My point is, that timings are for timings and we now have gauges for arbitrary values.

@msiebuhr yeah the inconsistent subcategories are not really ideal. There are already some pull requests and ideas how to change that.

Contributor

Dieterbe commented Jun 12, 2012

you should categorize metrics on their characteristics and behavior, not on what you can use them for. just like programming languages have types such as strings, ints, floats, not financials, fictions and counts (example is a bit silly but I think it's clear). given a number of metrics with each specific characteristics and behavior, it's up to users of statsd to pick the metric that is most suitable for each use case at hand. (that said, sometimes certain metric types can be recommended for common cases, but this should not be the norm). Counters and gauges are generic enough names to not become awkward for everything people do with them. But the "timing" name is too tighly coupled to the concept timing as my and now also @msiebuhr's example show.

And no, the characteristics that currently set statsd's timing metrics apart from gauges is the fact
min/max/avg/percentile/count are calculated/recorded (of the total group of received metrics in the flush interval). the gauge is something completely different: it records every metric separately and does not compute these statistics.

so if a user wants min/max/avg/percentile/count numbers for a whole bunch of metrics in a short interval, only the timing metric has the characteristics that fit the bill.

Contributor

iamcal commented Jun 13, 2012

Agree that timing is bad, but analyzer and interval-analyzer are not great names.

Perhaps range or group might be better? Hmm.

liesen commented Jun 13, 2012

How about observation? Too broad?

Owner

mrtazz commented Jun 13, 2012

@Dieterbe now I get where you're coming from. Was a bit too set on the timing usage. I agree it makes more sense to name the bucket after its behavior than intended usage. I still don't particularly like the analyzer names. range or interval are better for what StatsD is actually doing with it imho.

Contributor

Dieterbe commented Jun 13, 2012

@mrtazz phew :)
I'm looking for a name that describes the act of "taking a bunch of values and calculating the numbers such as min/max/avg/95thpercentile on it". because that's what this particular metric type is all about.
that's what I was attempting with (interval-)analyzer. @liesen's suggestion of observation is also fitting but too broad.
I also thought of statistic but it's also way too broad.

I don't understand why range, group or interval would be fitting names as they don't reflect the actions performed on the numbers, they only hint towards the fact that we deal with N metrics per flush-interval at a time, but that's the same for the counter metric.

I've been looking around on http://thesaurus.com and came up with more suggestions:

  • process
  • crunch (from number crunching)
  • distill, extract, dissect, synthesize

these are all more or less synonyms and are imho more representative of what this metric is about. I quite like crunch. It's short, slightly ludicrous and fairly representative of the metric.

@Dieterbe - I actually like the term stats. You're taking a bunch of numbers for a given interval and calculating statistics on them such as min/max/avg/nth percentile (you could imagine adding more... like standard deviation, median, etc...). counters count things and gauges are raw values. Although if you were to call it something like stats you probably would want to change the graphite backend prefix from stats to something like "data" :

data
  -stats
  -counters
  -gauges

I quite like that, @recursify.

Another possibility would be to suffix the names in the back-end; then data collected in any given sub-system would turn up together (i.e. server.subsystem.requests (counter) and server.subsystem.response_size (stat)).

Contributor

Dieterbe commented Jun 14, 2012

"Statistics is the study of the collection, organization, analysis, and interpretation of data" (http://en.wikipedia.org/wiki/Statistics)
"The practice or science of collecting and analyzing numerical data in large quantities." (google).

by definition, the entire statsd and graphite projects fall under the scope of the term statistics. This is especially confirmed by the name of the project "statsd". Naming this metric type "stats" would be confusing and inaccurate.

I think crunch or extract are the best choices so far.

@msiebuhr I don't understand what you mean. suffix the names of the metric types ? what's that syntax with parentheses? that won't work in graphite.

Owner

mrtazz commented Jun 26, 2012

I think we should probably go with range. That's really what that metric in StatsD is, an (ordered) range of values. That we are calculating upper, lower, mean, sum (and possibly other values in the future) shouldn't influence the name too much. It unnecessarily makes naming it more complicated than it already is, we really shouldn't try to be too clever here.

Contributor

otac0n commented Aug 7, 2012

How about we just make it configurable, and let everyone configure their own prefixes?
There are already several pull requests to make prefixes configurable (mine being the latest, since I failed to check the issue log before submitting).
I'm not really sure that ANY prefix is necessary or even beneficial.

Contributor

Dieterbe commented Aug 8, 2012

this discussion is not about prefixes, it's about the name of the "timing" metric type. these concerns are orthogonal.

Contributor

otac0n commented Aug 8, 2012

@Dieterbe I was assuming that you were wanting to change the name of the folder in graphite to something other than "timers". Was I incorrect?

Contributor

Dieterbe commented Aug 8, 2012

i'm talking about the timing metric type. that means basically all occurences of the word 'timing' in the statsd source code and documentation, how you categorize data on submission to statsd (foo:123|ms) and also the name of the folder in graphite (which is now timers).

Bump.

Have you seen Metrics?
They are using term "histogram" for that sort of data http://metrics.codahale.com/getting-started/#histograms
They have timers as well, but a little bit more powerful

timer will measure the amount of time it takes to process each request in milliseconds and provide a rate of requests in requests per second

+1 for histograms. We support histograms in our fork of Statsd. They're treated exactly like timers except the names aren't changed to include "timing". Here's an example:

# sending
my.histogram:100|h

# produces the following metrics
my.histogram.median
my.histogram.max
etc ...

We copied the name from codehale's metrics library and i think it's a good one, because histograms measure the statistical distribution of a set of values. The stats metrics produced by Statsd do the same thing ... avg, percentiles, etc.

Contributor

iamcal commented Sep 19, 2012

This name gets my +1 too, for what it's worth

Owner

mrtazz commented Sep 19, 2012

yup, I think histogram is a very suitable name.

Contributor

Dieterbe commented Sep 26, 2012

afaict a histogram is a frequency distribution over predefined buckets (class intervals), or using the mathematical definition from wikipedia: (http://en.wikipedia.org/wiki/Histogram)
"a histogram is a function mi that counts the number of observations that fall into each of the disjoint categories (known as bins)". it also explicitly states "a histogram represents a frequency distribution by means of rectangles whose widths represent class intervals"

what we are doing here is similar, but different: we adjust the buckets so that they correspond to predefined frequencies. (50%, 95%). this is not the same as class intervals, and hence what we are doing is not histograms.

Contributor

Dieterbe commented Sep 28, 2012

I've been reading up further on statistics, and the only properly fitting category in the field of statistics I could find is summary statistics (see http://en.wikipedia.org/wiki/Summary_statistic#Percentiles)
So I suggest to rename to summary

Contributor

mheffner commented Sep 28, 2012

I do think that trying to standardize, somewhat, with nomenclature in the metrics space would be beneficial. As @tjackowiak pointed to, the popular codahale Metrics library does have both histograms and timers. The latter actually generates two metrics: a distribution of the timer durations and the rate in which they are called.

I think part of the dislike in "timing" in statsd is not to do with the name, but that there is not an equivalent first-class histogram/distribution support in statsd. This leads users to use timing to track the distribution of multiple sample values, which I agree is odd when those samples have nothing to do with timing.

Contributor

Dieterbe commented Oct 1, 2012

agreed that adding histogram support somehow would be nice (though, not sure how that would work with graphite). but this issue stays the same: the timing metric we currently use has nothing to do with timing, and histogram would be a bad name because a histogram is something different (as explained in my previous post). my earlier suggestion(s) still stand..

Contributor

mheffner commented Oct 2, 2012

I'm confused, if there were a first-class type that took multiple samples and calculated avg/min/max/percentile (your original requirement), then you wouldn't use the timing metric. Therefore, it becomes a question of whether timing is an important-enough metric type to keep on its own. IMO, timing portions of code/requests/etc. is such a fundamental use of metrics that it really should be kept as a first-class metric type.

The other half of the discussion is what do we call the new first-class, multi-sample metric. I once again suggest that we should have a really good reason for not choosing terminology that aligns with existing popular metrics libraries. Coda's Metrics library is a pattern that has been copied numerous times and the term "histogram" has been used to describe the type of metric you are looking for. I recommend reading the code for the histogram collection in Coda's library and the multiple types of reservoir sampling methods it supports. Most of the implementations also calculate multiple percentiles.

Contributor

Dieterbe commented Oct 5, 2012

@mheffner re your first paragraph. the current timing metric type does exactly what you describe; taking samples and calculating avg/min/max/percentile. what i'm saying is that 'timing' is a bad name for this. 'timing' is a description of one of the several use cases for this datatype, but it's not a name that describes what the metric type actually does.

a good reason for not using the term 'histogram' for our current implementation of 'timing' is because according to my research (see above) the current metric behavoir has nothing to do with histograms, as histograms split up samples over class intervals, which is not what is happening here. although if we were to add further analysis to this metric in the form of histograms, then of course I would be fine with using the name 'histogram'

Contributor

Dieterbe commented Oct 5, 2012

and btw, i've pondered further about actual histogram support, and i think we can do it, but we would just need to have the user define the class intervals in advance. the problem is, after stats are aggregated and condensed into a histogram, you cannot change your class interals anymore.

Contributor

Dieterbe commented Oct 8, 2012

i'm getting so tired of this discussion, that i actually just implemented histogram support. see #162
with that change, the timer metric gains "bin categorisation" that can be used for histograms but is actually more powerful, in addition to the features it already has (which have nothing to do with histograms).
if #162 gets accepted, I will provide a patch that renames "timer" to "summarize" because we're doing summary statistics, part of which are histograms. histogram support is mentioned alongside it in the README (as this case thanks to #162) and that should be enough. we should not give the wrong name to things just because some other project does it.

ctoomey commented Oct 12, 2012

How 'bout "aggregate" as the name of this new type? From wikipedia: "An aggregate is a collection of items that are gathered together to form a total quantity." and "In statistics, aggregate data describes data combined from several measurements. When you aggregate data, you replace groups of observations with summary statistics based on those observations."

I think it fits better with counter, timer, gauge, set than "summarize" because aggregrate is a noun. Plus in verb form it describes what's being done -- aggregating measurements into summary statistics.

Contributor

Dieterbe commented Oct 15, 2012

sounds good to me... looking forward to write the patch for this as soon as @mrtazz +1's

Contributor

Dieterbe commented Nov 26, 2012

mrtazz gave the okay on irc for an implementation as long as it allows backwards compatibility for now. stay tuned....

Owner

mrtazz commented Jan 22, 2015

Closing this since the issue is super old and we haven't found a better name. And timing has served us well in the past years.

@mrtazz mrtazz closed this Jan 22, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment