| Status | |
|---|---|
| Distributions | contrib |
| Issues |   |
| Code Owners | @mx-psi, @dineshg13, @ankitpatel96 |
| Emeritus | @gbbr |
| Exporter Pipeline Type | Receiver Pipeline Type | Stability Level |
|---|---|---|
| traces | metrics | beta |
| traces | traces | beta |
The Datadog Connector is a connector component that computes Datadog APM Stats pre-sampling in the event that your traces pipeline is sampled using components such as the tailsamplingprocessor or probabilisticsamplerprocessor.
The connector is most applicable when using the sampling components such as the tailsamplingprocessor, or the probabilisticsamplerprocessor in one of your pipelines. The sampled pipeline should be duplicated and the datadog connector should be added to the the pipeline that is not being sampled to ensure that Datadog APM Stats are accurate in the backend.
To use the Datadog Connector, add the connector to one set of the duplicated pipelines while sampling the other. The Datadog Connector will compute APM Stats on all spans that it sees. Here is an example on how to add it to a pipeline using the [probabilisticsampler]:
| Before | After |
# ...
processors:
# ...
probabilistic_sampler:
sampling_percentage: 20
# add the "datadog" processor definition
datadog:
exporters:
datadog:
api:
key: ${env:DD_API_KEY}
service:
pipelines:
traces:
receivers: [otlp]
# prepend it to the sampler in your pipeline:
processors: [batch, datadog, probabilistic_sampler]
exporters: [datadog]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [datadog] |
# ...
processors:
probabilistic_sampler:
sampling_percentage: 20
connectors:
# add the "datadog" connector definition and further configurations
datadog/connector:
exporters:
datadog:
api:
key: ${env:DD_API_KEY}
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [datadog/connector]
traces/2: # this pipeline uses sampling
receivers: [datadog/connector]
processors: [batch, probabilistic_sampler]
exporters: [datadog]
metrics:
receivers: [datadog/connector]
processors: [batch]
exporters: [datadog] |
Here we have two traces pipelines that ingest the same data but one is being sampled. The one that is sampled has its data sent to the datadog backend for you to see the sampled subset of the total traces sent across. The other non-sampled pipeline of traces sends its data to the metrics pipeline to be used in the APM stats. This unsampled pipeline gives the full picture of how much data the application emits in traces.
connectors:
datadog/connector:
traces:
## @param ignore_resources - list of strings - optional
## A blacklist of regular expressions can be provided to disable certain traces based on their resource name
## all entries must be surrounded by double quotes and separated by commas.
#
# ignore_resources: ["(GET|POST) /healthcheck"]
## @param span_name_remappings - map of key/value pairs - optional
## A map of Datadog span operation name keys and preferred name valuues to update those names to. This can be used to
## automatically map Datadog Span Operation Names to an updated value, and is useful when a user wants to
## shorten or modify span names to something more user friendly in the case of instrumentation libraries with
## particularly verbose names.
#
# span_name_remappings:
# io.opentelemetry.javaagent.spring.client: spring.client
# instrumentation:express.server: express
# go.opentelemetry.io_contrib_instrumentation_net_http_otelhttp.client: http.client
## @param span_name_as_resource_name - use OpenTelemetry semantic convention for span naming - optional
## Option created to maintain similarity with the OpenTelemetry semantic conventions as discussed in the issue below.
## https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/trace/semantic_conventions
## https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/1909
#
# span_name_as_resource_name: true
## @param compute_stats_by_span_kind - enables APM stats computation based on `span.kind` - optional
## If set to true, enables an additional stats computation check on spans to see they have an eligible `span.kind` (server, consumer, client, producer).
## If enabled, a span with an eligible `span.kind` will have stats computed. If disabled, only top-level and measured spans will have stats computed.
## NOTE: For stats computed from OTel traces, only top-level spans are considered when this option is off.
#
# compute_stats_by_span_kind: true
## @param peer_tags_aggregation - enables aggregation of peer related tags in Datadog exporter - optional
## If set to true, enables aggregation of peer related tags (e.g., `peer.service`, `db.instance`, etc.) in Datadog exporter.
## If disabled, aggregated trace stats will not include these tags as dimensions on trace metrics.
## For the best experience with peer tags, Datadog also recommends enabling `compute_stats_by_span_kind`.
## If you are using an OTel tracer, it's best to have both enabled because client/producer spans with relevant peer tags
## may not be marked by Datadog exporter as top-level spans.
## If enabling both causes Datadog exporter to consume too many resources, try disabling `compute_stats_by_span_kind` first.
## A high cardinality of peer tags or APM resources can also contribute to higher CPU and memory consumption.
## You can check for the cardinality of these fields by making trace search queries in the Datadog UI.
## The default list of peer tags can be found in https://github.com/DataDog/datadog-agent/blob/main/pkg/trace/stats/concentrator.go.
#
# peer_tags_aggregation: false
## @param trace_buffer - specifies the number of outgoing trace payloads to buffer before dropping - optional
## If unset, the default value is 1000.
## If you start seeing log messages like `Payload in channel full. Dropped 1 payload.` in the datadog exporter, consider
## setting a higher `trace_buffer` to avoid traces being dropped.
#
# trace_buffer: 1000
## @param peer_tags - [BETA] Optional list of supplementary peer tags that go beyond the defaults. The Datadog backend validates all tags
## and will drop ones that are unapproved. The default set of peer tags can be found at
## https://github.com/DataDog/datadog-agent/blob/505170c4ac8c3cbff1a61cf5f84b28d835c91058/pkg/trace/stats/concentrator.go#L55.
#
# peer_tags: ["tag"]
## @param resource_attributes_as_container_tags - enables the use of resource attributes as container tags - Optional
## A list of resource attributes that should be used as container tags.
#
# resource_attributes_as_container_tags: ["could.availability_zone", "could.region"]NOTE: compute_stats_by_span_kind and peer_tags_aggregation only work when the feature gate connector.datadogconnector.performance is enabled. See below for details on this feature gate.
In case you are experiencing high memory usage with Datadog Connector, similar to issue, use the feature gate connector.datadogconnector.performance. With the feature gate enabled, Datadog Connector takes OTLP traces and produces OTLP metric with the name dd.internal.stats.payload. This Metric has an attribute dd.internal.stats.payload that contains the bytes for StatsPayload. With the feature gate, we can use Datadog Connector only in conjunction with Datadog Exporter. Please enable the feature only if needed for performance reasons and higher throughput. Enable the feature gate on all collectors (especially in gateway deployment) in the pipeline that sends data to Datadog. We plan to refactor this component in the future so that the signals produced are usable in any metrics pipeline.