10.0.0, in progress
- The Splunk span sink no longer reports an internal error for timeouts encountered in event submissions; instead, it reports a failure metric with a cause tag set to
submission_timeout. Thanks, antifuchs!
- The Splunk span sink now honors
Connection: keep-alivefrom the HEC endpoint and keeps around as many idle HTTP connections in reserve as it has HEC submission workers. Thanks, antifuchs!
- The metric
veneur.forward.post_metrics_totalwas being emitted both as a gauge and a counter. The errant gauge was removed. Thanks, gphat!
- The Splunk span sink can be configured with a sample rate for non-indicator spans with the
splunk_span_sample_ratesetting. Thanks, aditya!
- The splunk span sink now has configuration parameters
splunk_hec_connection_lifetime_jitterto regulate how long HTTP connections can be kept alive for. Thanks, antifuchs!
- The SignalFx sink can now filter metric names by prefix with
signalfx_metric_name_prefix_dropsand tag literals (case-insensitive) with
signalfx_metric_tag_literal_drops. Thanks gphat!
- Histograms and timers now support global scope. Histograms and timers tagged with "veneurglobalonly" will now emit all metrics from the global veneur. The default behavior is to emit aggregates like max, min locally and percentiles globally. Thanks, clin!
ssf.spans.root.received_totalglobal counter tracks the number of traces (root spans) processed system-wide. Thanks, aditya!
- The README's Metrics section has been updated, as it referred to some missing metrics. Thanks, gphat!
- Various references to Datadog were removed from the README, Veneur is vendor agnostic. Thanks, gphat!
- The metrics
veneur.flush.worker_duration_nswere removed, please use the per-sink
veneur.sink.metric_flush_total_duration_nsto monitor flush durations.
- The metrics
veneur.gc.mallocs_objects_totalmetrics were removed. Also from veneur proxy. Thanks, gphat!
- The metric
veneur.flush.other_samples_duration_nswas removed. Thanks, gphat!
- Metrics can be forwarded over gRPC using veneur-proxy (and Consul). Thanks, noahgoldman and Quantcast!
- Added tracer.InjectHeader convenience function for... convenience! Thanks, mikeh!
- Veneur has a new sink that can be configured to send spans as events into a Splunk HEC endpoint. Thanks, antifuchs and aditya!
- Go 1.11 is now supported and used for all public Docker images. Thanks, aditya!
veneur/tracepackage now supports setting the indicator bit on a span manually. Thanks, aditya!
validate-config-strictflags will make veneur exit appropriately after checking the specified (
-f) config file. Thanks, sdboyer!
veneur-emitwill now exit with an error if no data would have been sent. Thanks, sdboyer!
- The trace client can now correctly parse trace headers emitted by Envoy. Thanks, aditya!
- Go 1.9 is no longer supported.
veneur-emitnow takes a new option
-span_tagsfor tags that should be applied only to spans. This allows span-specific tags that are not applied to other emitted values. Thanks gphat!
-tagflag now applies the supplied tags to any value emitted, be it a span, metric, service check or event. Use other, mode specific flags (e.g. span_tags) to add tags only to those modes. Thanks gphat!
- Isolated a potential resource starvation issue. Added new configuration options for
veneur-proxyto configure its http.Transport:
idle_connection_timeoutfor controlling how long connections may idle before timing out, corresponds to
max_idle_connsfor controlling the maximum number of idle connections in total
max_idle_conns_per_hostfor controlling the maximum number of idle connections per host. Not that this now defaults to
- Added configuration options and improved defaults for the following tracing client parameters:
tracing_client_capacityfor controlling the depth of a buffer that holds tracing spans when they can't be emitted, defaults to
1024, up from
tracing_client_flush_intervalfor controlling how often the tracing client's backing buffer will be emptied (as an alternative to when it is full), defaults to
tracing_client_metrics_intervalfor controlling how often thew tracing client will send metrics about it's own operations, defaults to
1sand is unchanged
veneur-prometheusno longer crashes when the metrics host is unreachable. Thanks, arjenvanderende!
veneur-proxynow only logs forward counts at Debug level, drastically reducing log volume.
- Metrics can be imported over gRPC if the
grpc_addressparameter is set. Thanks, noahgoldman and Quantcast!
- When timing commands,
veneur-emitnow passes stdin, stdout and stderr through to the child process unmodified. Thanks antifuchs and sdboyer!
- Metrics can be forwarded over gRPC (currently only to a single Veneur) using
forward_use_grpc. Thanks, noahgoldman and Quantcast!
- Two new options,
debug_flushed_metricsmake veneur log (at level DEBUG) information about the metrics and spans it processes. Thanks, antifuchs!
veneur-emitnow takes a new option
-setwith a string argument, which allows counting how many unique values were reported in veneur's flush interval. Thanks, antifuchs!
- Fix a possible crash-before-panic when unable to open UDP socket. Thanks, gphat
tracer.Tracerwill default to the provided
operationNameif provided. This function is provided for compatibility with OpenTracing, but the package-level
trace.StartSpanFromContextfunction is recommended for new users.
- When creating timer metrics from indicator spans, veneur no longer prefixes
indicator_span_timer_namewith the string
veneur.. Thanks, antifuchs!
veneur-prometheusnow exports Histograms properly, with a statsd tag for each bucket
- Environment config for
VENEUR_PROXY_as a prefix. Previously used
VENEUR_which was a bug!
- Metric sampler parse function now looks for
veneurglobalonlyby prefix instead of direct equality for times where value can't/shouldn't be excluded even if it's blank. Thanks joeybloggs
veneur-prometheusnow exports a tag for each quartile rather than a seperate metric
- The tag
span_namehas been removed from the timer metric generated for indicator spans. Thanks, aditya!
- Added a timeout for sink ingestion to all sinks, which prevents a single slow sink from blocking ingestion on other span sinks indefinitely. Thanks, aditya!
trace.SetDefaultClientto handle overridding the default trace client, and closing the existing one. Thanks, franklinhu
- Veneur's key performance indicator metrics for metrics processing are reported through the statsd client. This way, KPI metrics are affected only by the metrics pipeline, not the tracing pipeline as well.
- SignalFX sink can now handle and convert ssf service checks (represented as a gauge). Thanks, stealthcode!
- Converted the grpsink to use unary instead of stream RPCs. Thanks, sdboyer!
- Bumped version of SignalFx go client to prevent accidental removal of
-from tag keys. Thanks, gphat!
- Switched to a faster consistent hash implementation. Thanks, aditya and noahgoldman!
- Reduce mutex contention around the default RNG in the math/rand standard library package. Thanks, aditya!
- Improve performance of
ssf.RandomlySampleby a factor of one million. Thanks, aditya!
- Added additional metrics for internal memory usage. Thanks, aditya!
ignored-metricsflags for veneur-prometheus will filter no metrics or labels if no filter is specified. Thanks, arjenvanderende!
- Fixed problem where all Datadog service checks were set to
OKinstead of the supplied value. Thanks, gphat!
- Added a timeout to the Kafka sink, which prevents the Kafka client from blocking other span sinks. Thanks, aditya!
- Official support for building Veneur's binaries with Go 1.8 has been dropped. Supported versions of Go for building Veneur 1.9, 1.10, or tip.
- Veneur's trace client library can still be used in applications that are built with Go 1.8, but it is no longer tested against Go 1.8.
veneur.ssf.received_totalmetric has been removed, as it is mostly redundant with
veneur.ssf.spans.received_total, and was not reported consistently between packet and framed formats.
veneur.ssf.spans.received_totalmetric now tracks all SSF data received, in either packet or framed format, whether or not a valid span was extracted.
- Receiving SSF in UDP packets now happens on
num_readersgoroutines. Thanks, antifuchs
- Updated SignalFx library dependency so that compression is enabled by default, saving significant time on large metric bodies. Thanks, gphat
- Decreased logging output of veneur-proxy. Thanks, gphat!
- Better warnings when invalid flag combinations are passed to
veneur-emit. Thanks, sdboyer!
- Revamped how sinks handle DogStatsD's events and service checks. Thanks, gphat
veneur.worker.checks_flushed_totalhave been replaced by
veneur.flush.event_worker_duration_nshas been replaced by
- The new
tags_excludeparameter can be used to strip tags from all metrics Veneur processes, either for all supported sinks or a subset of sinks. Thanks, aditya!
- The SSF client now defaults to opening 8 connections in parallel to avoid blocking client code. Thanks, antifuchs!
- New config settings
span_channel_capacitythat allow you to customize the parallelism of span ingestion. Thanks, antifuchs!
- New span sink utilization metrics - Thanks, antifuchs:
veneur.sink.span_ingest_total_duration_nsgives the total time per
sinkspent ingesting spans
veneur.worker.span_chan.total_capacitygives the utilization of the sink ingestion channel.
- Introduce a generic gRPC streaming backend for trace spans. Thanks, sdboyer!
- New config keys
signalfx_per_tag_api_keyswhich which allow sending signalfx data points with an API key specific to these data points' dimensions. Thanks, antifuchs!
- veneur-proxy now reports runtime metrics (with the prefix
veneur-proxy.) at a configurable interval controlled by
runtime_metrics_interval. It defaults to 10s. Thanks gphat!
- Allow specifying trace start/end times on
veneur-emit. Thanks, sdboyer!
span_channel_capacityto a non-zero value so we don't drop most spans in a minimal configuration. Thanks, gphat!
- Added tests for parsing floating point timers and histograms, just in case! Thanks gphat!
ignored-metricsflags added to veneur-prometheus to selectively restrict exports to Veneur. Thanks, yolken!
- These deprecated configuration keys are no longer supported and will cause an error on startup if used in a veneur config file:
api_hostname- replaced in 1.5.0 by
key- replaced in 1.5.0 by
trace_address- replaced in 1.7.0 by
trace_api_address- replaced in 1.5.0 by
ssf_address- replaced in 1.7.0 by
udp_address- replaced in 1.7.0 by
- These metrics have changed names:
- Datadog, MetricExtraction, and SignalFx sinks now emit
veneur.sink.metric_flush_total_duration_nsfor metric flush duration and tag it with
- Datadog, Kafka, MetricExtraction, and SignalFx sinks now emits
sink.metrics_flushed_totalfor metric flush counts and tag it with
- Datadog and LightStep sinks now emit
veneur.sink.span_flush_total_duration_nsfor span flush duration and tag it with
- Datadog, Kafka, MetricExtraction, and LightStep sinks now emit
sink.spans_flushed_totalfor metric flush counts and tag it with
- Datadog, MetricExtraction, and SignalFx sinks now emit
- Veneur's internal metrics are no longer tagged with
veneurlocalonly. This means that percentile metrics (such as timers) will now be aggregated globally.
- LightStep sink was hardcoded to use plaintext, now adjusts based on URL scheme (http versus https). Thanks gphat!
- The Datadog sink will no longer panic if
flush_max_per_bodyis not configured; a default is used instead. Thanks silverlyra!
- The statsd source will no longer reject all packets if
metric_max_lengthis not configured; a default is used instead. Thanks silverlyra!
veneur-emitnow infers parent and trace IDs from the environment (using the variables
VENEUR_EMIT_PARENT_SPAN_ID) and sets these environment variables from its
parent_span_idwhen timing commands, allowing for convenient construction of trace trees if traced programs call
veneur-emitthemselves. Thanks, antifuchs
- The Kafka sink for spans can now sample spans (at a rate determined by
kafka_span_sample_rate_percent) based off of traceIDs (by default) or a tag's values (configurable via
kafka_span_sample_tag) to consistently sample spans related to each other. Thanks, rhwlo!
- Improvements in SSF metrics reporting - thanks, antifuchs!
ssf.RandomlySamplethat takes an array of samples and a sample rate and randomly drops those samples, adjusting the kept samples' rates
- New variable
ssf.NamePrefixthat can be used to prepend a common name prefix to SSF sample names.
trace/metrics, containing functions that allow reporting metrics through a trace client.
- New type
ssf.Samplesholding a batch of samples which can be submitted conveniently through
trace.(*Trace).Add, which allows adding metrics to a trace span.
veneur-proxyhas a new configuration option
forward_timeoutwhich allows specifying how long forwarding a batch to global veneur servers may take in total. Thanks, antifuchs!
- Add native support for running Veneur within Kubernetes. Thanks, aditya!
- Updated Datadog span sink to latest version in Datadog tracing agent. Thanks, gphat!
- The semantics around
veneur-emitcommand timing have changed:
-shellCommandargument has been renamed to
-commandis now gone. The only way to time a command is to provide the command and its arguments as separate arguments, the method of passing in a shell-escaped string is no longer supported.
- A SignalFx sink has been added for flushing metrics to SignalFx. Thanks, gphat!
- A Kafka sink has been added for publishing spans or metrics. Thanks, parabuzzle and gphat!
- Buffered trace clients in
github.com/stripe/veneur/tracenow have a new option to automatically flush them in a periodic interval. Thanks, antifuchs!
- Gauges can now be marked as
veneurglobalonlyto be globally "last write wins". Thanks gphat!
-indicatorarguments. These (combined with
-ssf) allow submitting spans for tracing when recording timing data for commands. In addition,
-ssfnow works, too.
github.com/stripe/veneur/ssfpackage now has a few helper functions to create samples that can be attached to spans:
Timing. In addition, veneur now has a span sink that extracts these samples and treats them as metrics. Thanks, antifuchs
- Spans sent to Lightstep now have an
indicatortag set, indicating whether the span is an indicator span or not. Thanks, aditya!
veneur-emit -commandnow streams output from the invoked program's stdout/stderr to its own stdout/stderr. Thanks, antifuchs!
- Veneur now supports the tag
veneursinkonly:<sink_name>on metrics, which routes the metric to only the sink specified. See the docs here. Thanks, antifuchs!
- Veneur now emits a timer metric giving the duration (in nanoseconds) of every "indicator" span that it receives, if you configure the setting
indicator_span_timer_name. Thanks, antifuchs!
- All sinks have been moved to their own packages for smaller code and better interfaces. Thanks gphat!
- Removed noisy Sentry events that duplicated Datadog error reporting. Thanks, aditya!
- Veneur now reuses HTTP connections for forwarding and Datadog flushes. Furthermore each phase of the HTTP request is traced and can be seen using the trace sink of your choice. This should drastically improve performance and reliability for installs with large numbers of instances. Thanks gphat!
- Veneur now tracks statsd metrics for SSF spans concerning its own operation. This means that the
veneur.ssf.spans.received_totalcounter and the
veneur.ssf.packet_sizehistogram again reflect trace spans routed internally. Thanks, antifuchs!
- New 'blackhole' sink for testing and benchmark purposes. Thanks gphat!
- Veneur no longer requires the use of Datadog as a target for flushes. Veneur can now use one or more of any of its supported sinks as a backend. This realizes our desire for Veneur to be fully vendor agnostic. Thanks gphat!
- The package
github.com/stripe/veneur/tracenow depends on fewer other packages across veneur, making it easier to pull in
traceas a dependency. Thanks antifuchs!
- A Veneur server with tracing enabled now submits traces and spans concerning its own operation to itself internally without sending them over UDP. See the "Upgrade Notes" section below for metrics affected by this change. Thanks antifuchs!
- veneur-prometheus and veneur-proxy executables are now included in the docker images. Thanks jac
- All Veneur executables are now in $PATH in the docker images. Thanks jac
- When using Lightstep as a tracing sink, spans can be load-balanced more evenly across collectors by configuring the
trace_lightstep_num_clientsoption to multiplex across multiple clients. Thanks aditya!
- sync_with_interval is a new configuration option! If enabled, when starting, Veneur will now delay its first metric to be aligned with an
intervalboundary on the local clock. This will effectively "synchronize" Veneur instances across your deployment assuming reasonable clock behavior. The result is a metric timestamps in your TSDB that mostly line up improving bucketing behavior. Thanks gphat!
- Cleaned up some linter warnings. Thanks gphat!
- Tests no longer depend on implicit presence of a Datadog metric or span sink. Thanks gphat!
- Refactor internal HTTP helper into its own package fix up possible circular deps. Thanks gphat!
- Fix a panic when using
veneur-emitto emit metrics via
-ssfwhen no tags are specified. Thanks myndzi
- Remove spurious warnings about unset configuration settings. Thanks antifuchs
- Removed the InfluxDB plugin as it was experimental and wasn't working. We can make a sink for it in the future if desired. Thanks gphat!
Notes for upgrading from previous versions
setdata structure serialization format for communiation with a global Veneur server has changed in an incompatible way. If your infrastructure relies on a global Veneur installation, they will drop
setdata from non-matching versions until the entire fleet and the global Veneur are all at the same version.
- The metrics for SSF packets (and spans) received have changed names: They used to be
veneur.packet.spans.received_total, respectively, and they are now named
- New configuration option
statsd_listen_addresses, a list of URIs indicating on which ports (and protocols) Veneur should listen on for statsd metrics. This deprecates both the
tcp_addresssettings. Thanks antifuchs!
- New package
github.com/stripe/veneur/protocol, containing a wire protocol for sending/reading SSF over a streaming connection. Thanks antifuchs!
github.com/veneur/tracenow contains customizable
Clients that support streaming connections.
- veneur-prometheus now has a
-poption for specifying a prefix for all metrics. Thanks gphat!
- New metrics
ssf.packet_sizetrack the distribution of tags per span and total packet sizes, respectively.
- Our super spiffy logo, designed by mercedes, is now included at the top of the README!
- Refactor internals to use a new intermediary metric struct to facilitate new plugins. Thanks gphat!
- The BUILD_DATE and VERSION variables can be set at link-time and are now exposed by the
- A new HyperLogLog implementation means
sets are faster and allocate less memory. Thanks, martinpinto and seiflotfy!
- Introduced a new
metricSinkwhich provides a common interface for metric backends. In an upcoming release all plugins will be converted to this interface. Thanks gphat!
veneur-emitno longer supports the
-configargument, as it's no longer possible to reliably detect which statsd host/port to connect to. The
-hostportoption now takes a URL of the same form
statsd_listen_addressestakes to explicitly tell it what address it should send to.
SSFSpanCollectiongot removed, as it is superseded by the wire protocol. If you need to send multiple spans in bulk, we recommend setting up a buffered
- Veneur-emit can now time any shell command and emit its duration as a Timing metric. Thanks redsn0w422!
- Config options can now be provided via environment variables using envconfig for Veneur and veneur-proxy. Thanks gphat!
- SSF now includes a boolean
indicatorfield for signaling that this span is useful as a Service Level Indicator for its service.
- A type
SSFSpanCollectionhas been added but is not yet used.
veneur-prometheuscommand can be used to scrape prometheus endpoints and emit those metrics to Veneur. Thanks gphat and jvns
As a result of the efficiency improvements in this release, we've seen ~50% reduction in memory usage by way of measuring the allocated heap.
Secondly, the shift in not buffering spans on their way to LightStep should be noted. This changes behavior in Veneur which has traditionally done everything in 10s increments.
- If possible, initialization errors when starting Veneur will now be reported to Sentry. Thanks chimeracoder!
- Check return value of LightStep flush. Thanks chimeracoder!
- No longer using a fork of Logrus that fixed a race condition. Thanks chimeracoder!
- Updated to latest version of LightStep's tracing library, which drastically improves the success of spans on hosts with high span rates. Thanks gphat!
- No longer buffers spans for LightStep, they are dispatched directly to the LightStep client. Thanks gphat!
- Reuse an existing buffer when parsing incoming spans, reducing allocations. Thanks gphat!
- Use gogo protobuf for code generation of SSF's protobuf, resulting in faster and less memory span ingestion. Thanks gphat!
- veneur-proxy no longer balks at using static hosts for tracing and metrics. Thanks gphat!
operationfield has been deprecated in favor of the field
- SSF spans with a tag
namewill have that name placed into the SSF span
namefield until 2.0 is released.
- Correctly parse
Setmetrics if sent via SSF. Thanks redsn0w422!
- Return correct array of tags after parsing an SSF metric. Thanks redsn0w422!
- Don't panic if a packet doesn't have tags. Thanks redsn0w422!
- Fix a typo in the link to veneur-emit in the readme. Thanks vasi!
- Adds events and service checks to
veneur-emit. Thanks redsn0w422!
- Switch to dep for managing the
vendordirectory. Thanks chimeracoder!
- Remove support for
govendor. Thanks chimeracoder!
- Emit GC metrics at flush time. This feature is best used with Go 1.9 as previous versions cause GC pauses when collecting this information from go. Thanks gphat!
- Allow configuration of LightStep's reconnect period via
trace_lightstep_reconnect_periodand the maximum number of spans it will buffer via
trace_lightstep_maximum_spans. Thanks gphat!
- Switch to dep for managing the
vendordirectory. Thanks chimeracoder!
- Remove support for
govendor. Thanks chimeracoder!
- Added harmonic mean as an optional aggregate type. Use
hmeanas an option to the
aggregatesconfig option to enable. Thanks non!
- Added tests for
parseMetricSSF. Thanks redsn0w422!
veneur-emitflag usage to make testing easier. Thanks redsn0w422!
- Minor text fixes in the README. Thanks an-stripe!
- Restructured SSF parsing logic and added more tests. Thanks redsn0w422!
servicefrom the span. Thanks chimeracoder!
- Better document how to configure Veneur as a DogStatsD replacement. Thanks gphat with assist from stangles!
- Fixed an error in graceful shutdown of the TCP listener. Thanks evanj!
- Don't hang if we call
log.Fataland we aren't hooked up to a Sentry. Thanks evanj!
- Fix flusher_test being called more than once resulting in flappy failure. Thanks evanj!
- Improve flusher test to not start Veneur, fixing flapping test. Thanks evanj!
veneur-emitcan now emit metrics using the SSF protocol. Thanks redsn0w422!
- Documentation for SSF. Thanks gphat!
- It is no longer required to emit a
sumto get an
avgwhen configuring what aggregations to emit for a histogram. Thanks cgilling!
- Tags added in the
tagsconfig key are now applied to trace spans. Thanks chimeracoder!
- Additional documentation for
veneur-proxy. Thanks gphat!
- Revamped configuration file organization and comments. Thanks gphat!
- Changed some config keys to have more specific names to facilitate future refactoring. Thanks gphat!
- Adjust the flush loop to listen for server shutdown to improve test consistency. Thanks evanj!
- Veneur can now, experimentally, ingest metrics using the SSF protocol. Thanks redsn0w422!
- Reresolve the LightStep trace flusher on each flush, accomodating Consul-based DNS use and preventing stale sinks. Thanks chimeracoder!
- The following configuration keys are deprecated and will be removed in version 2.0 of Veneur:
- Require Go 1.8+ and stop building against 1.7 Thanks Thanks chimeracoder!
- Decrease logging level for proxy's "forwarded" messages. Thanks gphat!
- Failed discovery refreshes now log the service name. Thanks gphat!
- Proxy no longer requires a trace service name, since it's not wired up. Thanks gphat!
- No longer allow clients to pass in
-infas a value for a metric, as this caused errors on flush. Thanks gphat!
veneur-proxyto provide HA features with consistent hashing. See the Proxy section of the README
- Fix flusher_test to properly shutdown HTTP after handling. Thanks evanj!
- Verify that
trace_max_length_bytesis properly set. Thanks evanj!
- Fix some race conditions in testing.
- Document performance cost of TLS with RSA and ECDH keys. Thanks evanj!
- Reduce logging of tracing information to
debuglevel to decrease unnecessary logging.
- Reduce common TCP error logs to
infolevel. Thanks evanj!
- Deal with server shutdown without inspecting errors strings. Thanks evanj!
- Decrease the number of things we send to Sentry as "errors".
- Detect and emit a metric
reason:toolongfor metrics that exceed the metric max length.
- Emit a metric
reason:zerolengthfor metrics have no contents.
- Correct unnecessary allocation / goroutine in TCP connections that was leaking memory. Thanks evanj!
- Close idle TCP connections after 10 minutes. Thanks evanj!
- Fixed a lot of go lint errors.
- Add a metric
veneur.sentry.errors_totalfor number of errors we send to Sentry.
- New plugin
flush_filefor writing metrics to a flat file.
/healthcheck/tracingendpoint that returns 200 if this Veneur instance is accepting traces.
- Refactor tests to use a more shareable test fixture. Thanks evanj!
Server's constructor to not start any goroutines and add a
Start()that takes care of that, making for easier tests.
- Hostname and device name tags are now omitted from JSON generated for transmission to Datadog at flush time. Thanks evanj!
- Fix panic when an error is generated and Sentry is not configured. Thanks evanj!
- Fix typos in README
omit_empty_hostnameoption. If true and
hostnametag is set to empty, Veneur will not add a host tag to its own metrics. Thanks evanj!
- Support "all interfaces" addresses (
:1234) for listening configuration. Thanks evanj!
- Add support for receiving statsd packets over authenticated TLS connections. Thanks evanj!
- [EXPERIMENTAL] Add InfluxDB support.
- [EXPERIMENTAL] Add support for ingesting traces and sending to Datadog's APM agent.