Add tracer health metrics

[delner]

To monitor the health and performance of the tracer, this pull request adds health metrics to the tracer and other core components, which are sent as Statsd metrics under the datadog.tracer.* prefix. These metrics can then be graphed to evaluate the health, stability and impact of the trace library within a Ruby application.

By default, these metrics are enabled, if dogstatsd-ruby >= 3.3.0 is available. They can be disabled by setting the DD_DEBUG_HEALTH_METRICS_ENABLED ENV var to 0.

List of metrics added with this pull request:

datadog.tracer.api.errors
datadog.tracer.api.requests
datadog.tracer.api.responses
datadog.tracer.queue.accepted
datadog.tracer.queue.accepted_lengths
datadog.tracer.queue.accepted_size
datadog.tracer.queue.dropped
datadog.tracer.queue.length
datadog.tracer.queue.max_length
datadog.tracer.queue.size
datadog.tracer.queue.spans
datadog.tracer.traces.filtered # Added but not implemented
datadog.tracer.writer.cpu_time # Added but not implemented

[delner]

This pull request is still a work in progress; need to add the actual instrumentation into the tracer code and tests to verify its behavior. Should also run this in a sandbox application to verify it works end-to-end.

[marcotc]

🎉 It looks very good!

[brettlangdon]

looks like a good start

[delner]

Implemented some queue metrics, too.

Right now it's pretty aggressive, and will send stats with each trace push. This might be too much ultimately, but I want to measure the upper bounds on what we can send, because if we can send for each push, we can get some nice distribution metrics about things like "whats the distribution of spans per trace."

If we can't, then we can apply sample rates to Statsd, or rework these metrics to only compute/submit metric aggregations on flush, which would significantly reduce the number of stat calls, but also the granularity at which the data can be explored; a trade off we might have to take for sake of scale.

Consider this a first iteration, to experiment with and explore the upper bounds of what could be done with stats, but with the caveat that it might have to be reworked slightly to scale it back.

[delner]

Also opened this additional branch, which implements a more conservative "aggregate at flush time" approach; its on the other end of the scale in relation to this original implementation. It definitely has flaws of its own (e.g. queue.accepted_* and queue.* metrics are identical/redundant) but it also is much less verbose.

Feel free to compare the differences here and open it as a PR if necessary: https://github.com/DataDog/dd-trace-rb/compare/feature/debug_metrics...refactor/aggregated_queue_metrics?expand=1

[marcotc]

For these two sub-metrics of datadog.tracer.queue.accepted, would it make sense to have the metric name separator like so: datadog.tracer.queue.accepted.lengths datadog.tracer.queue.accepted.size?

[delner]

Yeah I think that could make sense; something we'd want to reconcile with our standards though.

[marcotc]

Pushed a rebase with latest master.

[marcotc]

This seem like a good compromise between a very shallow estimate (only ::ObjectSpace.memsize_of(object)) and a full recursive memory measurement. 👍