consider merging with armon's go-metrics

[josephholsten]

@armon is a maintainer of serf and statsite. He also has a project with the same name and goal as yours: https://github.com/armon/go-metrics

Perhaps you might consider combining effort?

[rcrowley]

Possibly. Why do you think they should be merged, @josephholsten?

The APIs aren't terribly compatible and I'm concerned that indicates a deeper philosophical incompatibility. I believe processes should, in most cases, emit statistics, not event streams, so rcrowley/go-metrics helps users aggregate statistics properly. The original statsd was intended to run locally next to PHP because PHP couldn't aggregate in-process.

What do you think, @armon?

[armon]

As @rcrowley said, this version focusses on aggregation to provide statistics where my version is mostly built to export event streams to external systems (statsite, in my use case). That said, it would be fairly trivial to write a sink for my package that would write-through to this version. It would effectively be an enhanced version of the "InmemSink" that is already there.

[tsenart]

In any case, I think having both packages with the same name fosters
confusion in the space. :(
On Feb 6, 2014 12:49 AM, "Armon Dadgar" notifications@github.com wrote:

As @rcrowley https://github.com/rcrowley said, this version focusses on
aggregation to provide statistics where my version is mostly built to
export event streams to external systems (statsite, in my use case). That
said, it would be fairly trivial to write a sink for my package that would
write-through to this version. It would effectively be an enhanced version
of the "InmemSink" that is already there.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/39#issuecomment-34280793
.

[josephholsten]

I guess my dream metrics library would work something like log4j: allow multiple subscribers to the event stream, each of which can filter or process as they like, and output in a variety of formats.

And my ideal architecture would have applications send events directly to a system-wide stat server (as with syslogd), which can filter, aggregate, rollup, store locally and/or forward to a centralized aggregator, which then forwards to a time-series database.

I personally think that a number of meters, which are attached to one or more sinks, which then may have attached aggregating emitters is a reasonable API; and reminiscent of AMQP's exchanges, bindings and queues.

@rcrowley & @armon: does that sound over-engineered? Not worth the effort? Totally uninteresting? Or otherwise bad?

[armon]

@josephholsten That is basically exactly what I do. The app emits a metrics stream, I think sink everything to statsite to do the aggregation and rollup, and then it finally dumps everything into graphite from there.

[rcrowley]

Here's the thing about event streams and the reason why rcrowley/go-metrics will continue along its current path: Event streams become prohibitively expensive in terms of network I/O when you start timing every cache lookup and database query, requests to other services, individual method invocations, and tracking connection pool usage. Observer effects start to impact user experience.

If you do go the event stream route and don't set your switches on fire I would still argue you're going to need something that looks a lot like rcrowley/go-metrics in each process that consumes your event stream.

[josephholsten]

I totally agree about setting switches on fire (or at least dropping metrics to the point of unsuitability).

This thread from aphyr conveys the point well: https://twitter.com/ReinH/status/431871352857849858.

If you guys are keen on doing things differently, hugs all around. @armon if there's anything you'd want added to @rcrowley's codebase (or vice versa), then please let me know and I'll help. Otherwise I'll gladly drop it.

[armon]

@josephholsten I guess I'm not clear on what you are looking for, since the packages serve different needs. I think your dream library sounds very similar to streaming raw metrics to a locally running aggregator, which then forwards to some storage system sounds like it could reasonably be built using either library.

I wrote my version of go-metrics because I had some metrics that I wanted aggregated with very high reliability, so they are streamed over TCP to statsite. I also collect a small amount (10's of data points per second) of runtime data that is similarly streamed. My main concerns being reliability of certain counters and a some insight into what is happening.

This version of go-metrics is much more suited for profiling an application, doing an in-process rollup, and exporting the data to a number of sinks. Correct me if I'm wrong @rcrowley.

[rcrowley]

This version of go-metrics is much more suited for profiling an application,
doing an in-process rollup, and exporting the data to a number of sinks.
Correct me if I'm wrong @rcrowley.

Bingo.

I'm going to go ahead and close this since there's no work to be done. Thanks all.