Support gathering "top" percentile statistics #200

Merged
merged 1 commit into from Mar 23, 2013

Conversation

Projects
None yet
3 participants
Contributor

zwily commented Dec 5, 2012

Adds support for collecting statistics on top percentiles, instead of
the default bottom percentiles. You specify a top percentile by
using a negative number - so -10 will collect the top 10% of data. It
will emit: mean_top10, lower_top10, and sum_top10.

Using a negative number may seem hacky, but it's convenient and there
is a precedent - referencing an array from the end in some languages
can be done with negative indexes.

Owner

mrtazz commented Jan 14, 2013

What's the use case for this? It seems like you would only get outliers of your data then, which doesn't really seem useful?

Contributor

zwily commented Jan 14, 2013

Sometimes you want the outliers. I want to know what the worst experience people are getting is like, as well as what's typical. Werner Vogels talked about this at re:Invent last year - he said at Amazon they religiously look at the upper 99.9% performance as one of their key metrics, and obviously suggested everyone do the same.

I think it makes sense.

abh commented Mar 13, 2013

👍 for this; I have a similar need for my application (DNS servers for http://www.ntppool.org/ ).

The outliers are really extreme compared to the regular load but very regular and I have to do my capacity planning based on those outliers.

Owner

mrtazz commented Mar 16, 2013

makes sense yeah. @zwily would you be up to basing this onto the newest master to make it easier to merge?

@zwily zwily Support gathering "top" percentile statistics
Adds support for collecting statistics on top percentiles, instead of
the default bottom percentiles. You specify a top percentile by
using a negative number - so -10 will collect the top 10% of data. It
will emit: mean_top10, lower_top10, and sum_top10.

Using a negative number may seem hacky, but it's convenient and there
is a precedent - referencing an array from the end in some languages
can be done with negative indexes.
f369dfa
Contributor

zwily commented Mar 16, 2013

@mrtazz - Yep. Rebased on master.

Owner

mrtazz commented Mar 23, 2013

perfect, thanks for contributing this!

mrtazz merged commit 045b1de into etsy:master Mar 23, 2013

1 check passed

default The Travis build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment