Skip to content

feat(mezmo_datadog_agent_parser): Create MezmoDatadogAgentParser#49

Merged
biblicalph merged 3 commits intomasterfrom
biblicalph/LOG-22997
Jan 28, 2026
Merged

feat(mezmo_datadog_agent_parser): Create MezmoDatadogAgentParser#49
biblicalph merged 3 commits intomasterfrom
biblicalph/LOG-22997

Conversation

@biblicalph
Copy link
Contributor

@biblicalph biblicalph commented Jan 26, 2026

This PR adds support for transforming Datadog agent events into logs, metrics, traces and sketches.
This allows us to deprecate VRL handling of metrics while adding support for Datadog traces and sketches.

For reference the following protos apply:

  1. Logs: https://github.com/DataDog/agent-payload/blob/master/proto/logs/agent_logs_payload.proto
  2. Metrics: https://github.com/DataDog/agent-payload/blob/master/proto/metrics/agent_payload.proto#L91
  3. Traces: https://github.com/DataDog/datadog-agent/blob/main/pkg/proto/datadog/trace/agent_payload.proto

Upstream of vector, the agent payload is intercepted, decoded and normalized. The transform thus expects TracerPayload item for trace events, Sketch for sketch metrics, MetricSeries for metrics and Log for logs.

The implementation is mostly based on the existing Datadog agent source implementation; we don't use that source because it exposes an http server to collect metrics

Ref: LOG-22997

Summary

Vector configuration

How did you test this PR?

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Notes

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
  • Some CI checks run only after we manually approve them.
    • We recommend adding a pre-push hook, please see this template.
    • Alternatively, we recommend running the following locally before pushing to the remote branch:
      • cargo fmt --all
      • cargo clippy --workspace --all-targets -- -D warnings
      • cargo nextest run --workspace (alternatively, you can run cargo test --all)
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

@biblicalph biblicalph self-assigned this Jan 26, 2026
@biblicalph biblicalph marked this pull request as draft January 26, 2026 22:03
@biblicalph biblicalph force-pushed the biblicalph/LOG-22997 branch 2 times, most recently from 39a967b to 89057ec Compare January 26, 2026 22:59
@biblicalph biblicalph marked this pull request as ready for review January 26, 2026 22:59
The transform receives a log event and outputs a single
or multiple events to one of logs, metrics, traces or
 _unmatched ouptuts.
The Datadog agent data is not decoded by this transform;
the incoming event is assumed to be already decoded.
For traces, each event is assumed to be an entry of the
tracerPayloads or traces (legacy) field of the agent
payload. For metrics, each event is assumed to be an
entry of the series field of the agent payload.
The implementation of the transform is mostly based on
the Datadog agent source.

Ref: LOG-22997
@biblicalph biblicalph force-pushed the biblicalph/LOG-22997 branch from 89057ec to 36ebe26 Compare January 26, 2026 23:08

parser.build_events_from_messages(event, output_messages)
parser
.build_events_from_payloads(event.clone(), output_messages)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clone is still needed here because the method mutates the event

1. Reduce cost of cloning event when emitting
multiple events for a received metric or trace
2. Implement trait for transforming event
3. Move event into transform to reduce cloning.
4. On error, move the error into the event
5. Route the original event to unmatched if
transforming fails. Previously this was done
for logs only.

Ref: LOG-22997
@biblicalph biblicalph force-pushed the biblicalph/LOG-22997 branch 4 times, most recently from 1313aa2 to 6e1bf69 Compare January 28, 2026 12:58
The vector documentation now includes the two flags for
Datadog logs and metrics - likely in newer Vector versions.

This matches the implementation.

Ref: LOG-22997
@biblicalph biblicalph force-pushed the biblicalph/LOG-22997 branch from 6e1bf69 to 6b1e115 Compare January 28, 2026 13:04
Copy link
Member

@mdeltito mdeltito left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good for the first cut 👍

@biblicalph biblicalph merged commit 4eefdd2 into master Jan 28, 2026
2 checks passed
@biblicalph biblicalph deleted the biblicalph/LOG-22997 branch January 28, 2026 14:46
@logdnabot
Copy link
Contributor

🎉 This PR is included in version 6.7.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants