statsd source not supporting Dogstatsd distribution type of metrics

[szibis]

https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=python#distribution

https://docs.datadoghq.com/developers/metrics/types/?tab=distribution#definition

All tested on latest nightly.

Simple example script in python

from datadog import initialize, statsd
import time
import random

options = {
    'statsd_host':'127.0.0.1',
    'statsd_port':8125
}

initialize(**options)

tags_list=["environment:dev"]

@statsd.timed('ss.test_metric.timer', tags=tags_list)
def my_function():
  time.sleep(random.randint(0, 10))

i = 0
while(1):
  i += 1
  statsd.set('ss.test_metric.set', i, tags=tags_list)

  my_function()

  statsd.increment('ss.test_metric.increment', tags=tags_list)
  statsd.decrement('ss.test_metric.decrement', tags=tags_list)

  statsd.gauge('ss.test_metric.gauge', i, tags=tags_list)

  statsd.increment('ss.test_metric.count_rate', sample_rate=0.5, tags=tags_list)

  statsd.distribution('ss.test_metric.distribution', random.randint(0, 20), tags=tags_list)

  statsd.histogram('ss.test_metric.histogram', random.randint(0, 20), tags=tags_list)

  time.sleep(2)

Will produce for distribution emit data this in Vector logs for statsd source

May 14 14:05:26 ip-10-105-195-187 vector[30693]: May 14 14:05:26.721 ERROR source{name=statsd type=statsd}: vector::sources::statsd: Statsd parse error: UnknownMetricType("d")
May 14 14:05:36 ip-10-105-195-187 vector[30693]: May 14 14:05:36.728 ERROR source{name=statsd type=statsd}: vector::sources::statsd: Statsd parse error: UnknownMetricType("d")

Implementation

DataDog distribution type of metrics will be supported by expanding Vector distribution metric to, besides histogram statistic, support summary statistic. In sinks, this statistic differs only in two things from histogram:

• Aggregation of distribution points is done with sample statistic. This statistic mirrors Datadog distribution by producing aggregates defined at https://docs.datadoghq.com/developers/metrics/types/?tab=distribution#definition. enhancement: Add DataDog's distribution metric #2913.
• 'datadog metric' sink maps summary to Datadog distribution metric. Sending distributions to DataDog #3130

While we accept it through:

• statsd source tagged with d which corresponds to Datadog distribution metric tag. enhancement: Add DataDog's distribution metric #2913.
• log_to_metric transform. enhancement(log_to_metric transform): Add support for summary metric #3726

Originally I thought we could reduce the amount of work by making summary to only differ from histogram in datadog metric sink, but that would make this feature inconsistent.

Alternatives

Support the metric only in datadog metric' sink and statsd source/sink, while in other places default to histogram behavior. In documentation we would only add that we support d tag in statsd source/sink which is passed to datadog metric sink.

Todo

Support of Dogstatsd distribution type of metrics

• Add support to log_to_metric transform enhancement(log_to_metric transform): Add support for summary metric #3726

• Update influxdb_metrics sink. enhancement(influxdb_metrics sink): Support summary statistic #3846

• Update prometheus sink. enhancement(prometheus sink): Add support for summary metric #4032

• Update docs. Waiting for implementations to be done.

  • Consider better naming
  • data-model/metric
  • log_to_metric transform
  • datadog_metrics sink
  • influxdb_metrics sink
  • prometheus sink
  • recheck others

[binarylogic]

Thanks @szibis, we're prioritizing this. @ktff let us know if you need more detail around this.

[ktff]

There is one more small improvements that can be made for aws_cloudwatch_metrics and datadog_metrics, at the moment they are sending aggregates as defined by histogram, but they can be changed to be as defined by https://docs.datadoghq.com/developers/metrics/types/?tab=distribution#definition. But this also depends on how much we would like to integrate this metric.

[binarylogic]

@ktff thanks.

but they can be changed to be as defined by docs.datadoghq.com/developers/metrics/types/?tab=distribution#definition

Right. Isn't that the problem though? According to @lukesteensen in #2913 (comment), we can't send this data to DataDog. They are using an undocumented API for sending sketch data types. We'll need to reach out to them to get information on how to do this.

[ktff]

@binarylogic by changing the aggregates that we send. That mostly means adding more percentile aggregations than the histogram does. But it seams better to wait for #3130 than add this.

[ktff]

@jamtur01 @binarylogic the opening comment has been changed to reflect the current state and direction.

[binarylogic]

Thanks. The checklist looks good. I assume you’ll continue work on those? Are you blocked on any of that?

[ktff]

I assume you’ll continue work on those?

Yes, and

Are you blocked on any of that?

no.

[ktff]

This will be done with merger of timberio/vector-website#128