Parse namespace from incoming events for `datadog_agent` source

[jszwedko]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Problem:

A user encountered a great many unexpected custom metrics being reported in Datadog when they introduced Vector with the datadog_agent source and datadog_metrics sink. This appeared to be due to a couple of factors:

• They copy/pasted the default configuration for the datadog_metrics sink which includes default_namespace = "service"
• The datadog_agent source does not set a namespace for incoming metrics

This caused all of the metrics the agent normally collects to be namespaced under an additional service. namespace and be counted as a custom metric.

Attempted Solutions

Leaving out the default_namespace configuration option on the datadog_metrics sink remedies this issue, but this seems like a footgun for the common datadog_agent -> datadog_metrics pipeline.

Proposal

We'd like to parse out the namespace of incoming metrics from the Datadog Agent. Datadog Agent namespaces the metrics with <namespace>.<metric_name> (e.g. system.fs.inodes.total would have a namespace of system).

This would also allow for manipulation of the namespace, easily, in remap.

Open questions:

• Given a metric of system.fs.inodes.total is system the namespace? Or system.fs?

References

• Parse out namespace #4863

Version

vector 0.22.0

[binarylogic]

I'm just realizing this might be complicated with the namespace delimiter. For example, if we parse out the namespace from system.load.5 as system, then we'll name the metric as system_load.5 when sending to Datadog since we use _ for the delimiter. The proper solution is to choose the right delimiter in each sink which would be a breaking change.

[jszwedko]

We discussed offline, but just noting that we do vary the namespace delimiter in the sink (. for the datadog_metrics sink).