nsqd: track stats of end-to-end message processing time

[mreiferson]

It would be incredibly useful information for nsqd to keep track of end-to-end statistics of how long it's taking to process messages (from ns time of PUB to ns time of FIN) in percentiles, for each topic/channel.

This would need to take a fast/efficient stream approach of approximating these values so as not to add significant memory/CPU overhead.

They should be available via the /stats endpoint and pushed to statsd for context over time.

cc @jehiah @michaelhood

[mynameisfiber]

Might I suggest using a multilevel ε-summary?

[mreiferson]

but is it FAST

[mreiferson]

also cc @davemarchevsky

[mynameisfiber]

O(log(log(N))) inserts and query where N can be kept small with some forgettable-like decaying

[mreiferson]

@mynameisfiber wouldn't it be awesome if you came up with a clever name and created an easy to use go package so that I didn't have to read that PDF? 🎉

[mynameisfiber]

i'm leaving this one in @davemarchevsky's capable hands... although i think aliGOzam has a nice ring to it.

[mynameisfiber]

@mreiferson or this... http://godoc.org/github.com/bmizerany/perks/quantile (care of @danielhfrank)

[mreiferson]

hi5 @danielhfrank

[mreiferson]

see #280