Migrate dogstatsd from libdatadog #2

duncanpharvey · 2025-03-19T19:18:13Z

What does this PR do?

Migrate dogstatsd crate from libdatadog to this repo.

Motivation

Develop library crates used solely by Serverless independently from libdatadog.

https://datadoghq.atlassian.net/browse/SVLS-6327

Additional Notes

Removed versioning of dogstatsd crate since we won't be versioning crates for now but rather will be adding dependencies by the commit hash
Generated fresh Cargo.lock file with the latest versions of dependencies pulled in by dogstatsd

Commands to migrate dogstatsd crate along with commit history from libdatadog.

cd libdatadog
git subtree split -P dogstatsd -b duncan-harvey/split-dogstatsd
git checkout duncan-harvey/split-dogstatsd
git push

cd serverless-components
git checkout -b duncan-harvey/dogstatsd
git subtree add -P crates/dogstatsd git@github.com:DataDog/libdatadog.git duncan-harvey/split-dogstatsd

Describe how to test/QA your changes

Update dogstatsd dependency in Cargo.toml to reference the latest commit in this feature branch.

dogstatsd = { git = "https://github.com/DataDog/serverless-components/", rev = "78dc404c342b5df7f1371441252d438745ca1f79" }

For testing Azure Functions I updated the dependency in the datadog-serverless-trace-mini-agent crate, built the libdatadog binaries, deployed them to Azure, and confirmed both custom and runtime metrics are sent to Datadog successfully.

* feat: upstream dogstatsd from datadog lambda extension * add: stand alone dogstatsd binary as example * fix: imports * fix: make test utils public * fix: more test utils public * fix: make entry public * fix: clippy * fix: make test config public * Revert "fix: make test config public" This reverts commit 7dc979979d85c665767906030f91253725850f30. * fix: make fields public for tests and assertions * use reqwest 0.12.4 in Cargo.lock * add copyright headers * apply formatting * update license * upgrade rust to 1.77.2 * install protoc binary for lint, test, and miri workflows * install protoc binary for cross on centos7 * add dogstatsd to alpine build and upgrade alpine version * install protoc binary for coverage * use rust 1.76.0 and forked saluki-backport repo * remove package specific toolchain * refactor protoc install script to pass install path as an argument * install protoc for benchmark ci job * remove unzip install since it is now part of base image * add dogstatsd integration test * fix license * fix merge * fix lint error * use localhost address for dogstatsd test * use port 18125 for dogstatsd test * disable dogstatsd test for miri * dogstatsd cleanup * fix license * fix merge conflicts * update alpine base image to 3.19.3 --------- Co-authored-by: alexgallotta <5581237+alexgallotta@users.noreply.github.com>

* feat: Support DD_HTTP_PROXY and DD_HTTPS_PROXY * fix: fmt * feat: honor dd_proxy

* feat: upstream dogstatsd from datadog lambda extension * add: stand alone dogstatsd binary as example * fix: imports * fix: make test utils public * fix: more test utils public * fix: make entry public * fix: clippy * fix: make test config public * Revert "fix: make test config public" This reverts commit 7dc979979d85c665767906030f91253725850f30. * fix: make fields public for tests and assertions * refactor: remove unused error * refactor: do one step parsing * fix: duplicate dependencies from conflicts * refactor: use type for sorted tags and ustr * fix: statsd parsing and tests * format: format and clippy test: fix test format with # for tags. It also changed the byte wise test expected result * fix: use only minimum regexp * fix: increase to 10k context size, lost in refactors/moving * refactor: update license * fix: remove custom toolchain * style: fix comment too long not picked up by rustfmt * fix: restore previous cargo.lock * fix: missing error dep * fix: use empty tags * fix: update license * fix: nightly rustc 1.83 format * fix: clippy * fix: do not use a map, support duplicate tag values --------- Co-authored-by: Taegyun Kim <taegyun.kim@datadoghq.com>

* feat: support HTTPS_PROXY for traces * fix: https_proxy only. * fix: maybe native-tls works in lambda? * feat: try rust-tls * fix: rusttls oops * fix: reee it's rustls why * fix: default-features must be false * WIP: debug log, will revert * fix: reqwest honors system proxies, hyper doesn't seem to. Only proxy https traffic * fix: revert hyper-proxy, just use system proxy * fix: revert proxy, use system * fix: revert hyper-proxy, use system proxy * fix: revert senddata proxy change from tests * Revert "fix: revert senddata proxy change from tests" This reverts commit 105000853f86dd46c39914434907452d9ca05b60. * Revert "fix: revert hyper-proxy, use system proxy" This reverts commit d9ebdc7e3cd68f046a4e8d981ac0564b34458983. * Revert "fix: revert proxy, use system" This reverts commit e4a8e18c14b56f8b889aec23b40cf3ccaf511639. * Revert "fix: revert hyper-proxy, just use system proxy" This reverts commit f8ed3005f75bdb3a4a4fb35dc52f499206b8d2b9. * fix: re-commit tests * fix: fmt * feat: license * feat: Wrap proxy in feature flag * fix: fmt * fix: not sure why the arg is needed in create_send_data * fix: no api changes for public interfaces * fix: None * fix: allow unused * fix: None for update_send_results_example * fix: remove unused error import

* feat: Map over sketches attr instead of assigning in map, which Levi thinks avoids a memcpy * fix: lint

* fix: id public and use optional for tags * change: utility to easily create values

This way we're more explicit about upgrades, rather than upgrades happening "accidentally" if the `Cargo.lock` gets refreshed.

* feat: Prefer DD_PROXY_HTTPS over HTTPS_PROXY * fix: no proxy on ints * fix: clippy thx

* show dogstatsd logs in serverless mini agent * increase buffer size to 8192 bytes * update metric pattern to exclude service checks and to account for container ids * exclude events from being parsed * add comment for 8KB max buffer size matching default value in Go Agent * lazily initialize static regex for dogstatsd metrics * minor refactors * add unit tests * remove explicit drops

* add fixed timeout to verify behavior * add number of series and distro to flush * tmp: add info on trace retries * add more tmp debug log * fix debug log * Revert "fix debug log" This reverts commit 4b44bb37ec4cc658d37fbeea820d399a4e924788. * Revert "add more tmp debug log" This reverts commit 536917b8c2f990365058c90a580937ee5e5f1894. * Revert "tmp: add info on trace retries" This reverts commit 5e4790794decea0f3f7e1ab523a9ab36e39820db. * add timeout for client * add todo * fmt * add constant for timeout duration --------- Co-authored-by: jordan gonzález <30836115+duncanista@users.noreply.github.com>

* use saluki in dogstatsd * update to latest saluki commit sha

* to not log error on unsupported metric type * avoid log altogether * fix: better error handling and log warn for ignored metric types

* hardcode filter metric * update `cc` to be `^1` * add todo

Co-authored-by: jordan gonzález <30836115+duncanista@users.noreply.github.com>

* resources are only dd.internal.resource tags * improve test

* fix: support tag that are not key-value pairs * fix comment * add test for tag with multiple columns * use more efficient string concatenation * clippy

* tmp: disable flushing * try flushing only once on 500 error and do sketch/distro shipping in parallel * remove stale comment * avoid panic * make sure the client has timeout, or explicitely communicate failure * formatting * formatting * wait for flush to terminate * remove unused depdency * use inspect err instead of verbose match * propagate result * format * use map_err * cleanup style

Co-authored-by: Levi Morrison <levi.morrison@datadoghq.com>

* feat: set timestamp when metric is parsed * feat: Ten second rolling buckets * debuggin * remove debug line * fix: remove time from aggr * feat: Clean up parse code, refactor floored buckets to metrics * feat: comment docs * feat: Cleanup impl * fix: call method

Add clippy warnings and allows for panic macros to most crates Add an extension crate for mutex in ddcommon to isolate the unwrap to one location in code to avoid the allow annotations. We aren't going to stop unwrapping mutexes anytime soon. Replace use of lazy_static with OnceLock for ddcommon, ddtelemetry, live-debugger, tools, sidecar

(serverless) feature: Push retry strategy to libdatadog to avoid re-compressing

dogstatsd - optimize `parse` function * `MAX_TAGS` to `100` based on internal docs, the limit for custom tags is 100 --------- Co-authored-by: Edmund Kump <edmund.kump@datadoghq.com>

…dbbbcecf' git-subtree-dir: crates/dogstatsd git-subtree-mainline: ab46521 git-subtree-split: e9e3fcb

duncanpharvey and others added 30 commits September 9, 2024 10:42

adds readme for dogstatsd (#621)

512cb24

feat: Support DD_HTTP_PROXY and DD_HTTPS_PROXY (#631)

95f1f14

* feat: Support DD_HTTP_PROXY and DD_HTTPS_PROXY * fix: fmt * feat: honor dd_proxy

test: ignore dogstatsd tests under miri (#661)

66277a3

* feat: Map over sketches attr instead of assigning in map, which Levi thinks avoids a memcpy * fix: lint

Fix dogstatsd aggregator (#654)

b56ca77

* fix: id public and use optional for tags * change: utility to easily create values

Explicitly pin version of deps from saluki-backport repo

088060e

This way we're more explicit about upgrades, rather than upgrades happening "accidentally" if the `Cargo.lock` gets refreshed.

feat: Prefer DD_PROXY_HTTPS over HTTPS_PROXY (#673)

a14916a

* feat: Prefer DD_PROXY_HTTPS over HTTPS_PROXY * fix: no proxy on ints * fix: clippy thx

[chore] Use elapsed() if possible when calculating durations (#750)

7a3b41d

fix key val resource order in dogstatsd (#803)

accf31b

[dogstatsd] Use Saluki as a dependency (#804)

97bf33b

* use saluki in dogstatsd * update to latest saluki commit sha

to not log error on unsupported metric type (#777)

ecee809

* to not log error on unsupported metric type * avoid log altogether * fix: better error handling and log warn for ignored metric types

[dogstatsd] hardcode filter metric (#761)

1048f57

* hardcode filter metric * update `cc` to be `^1` * add todo

chore: [SVLS-5992] clearer dogstatsd Flusher API (#822)

a05e5dc

Co-authored-by: jordan gonzález <30836115+duncanista@users.noreply.github.com>

resources are only dd.internal.resource tags (#829)

bf89fd1

* resources are only dd.internal.resource tags * improve test

fix: support tag that are not key-value pairs (#826)

717f491

* fix: support tag that are not key-value pairs * fix comment * add test for tag with multiple columns * use more efficient string concatenation * clippy

feat(dogstatsd): compress metrics (#901)

163b081

Co-authored-by: Levi Morrison <levi.morrison@datadoghq.com>

Push retry strategy into client (#940)

a34de4c

(serverless) feature: Push retry strategy to libdatadog to avoid re-compressing

[dogstatsd] optimize parse function in SortedTags (#943)

e9e3fcb

dogstatsd - optimize `parse` function * `MAX_TAGS` to `100` based on internal docs, the limit for custom tags is 100 --------- Co-authored-by: Edmund Kump <edmund.kump@datadoghq.com>

Add 'crates/dogstatsd/' from commit 'e9e3fcb9be11b07c1cf870b27d1a5530…

b31f6a6

…dbbbcecf' git-subtree-dir: crates/dogstatsd git-subtree-mainline: ab46521 git-subtree-split: e9e3fcb

remove version from dogstatsd Cargo.toml

1073537

use relative link in dogstatsd readme

3e70c29

Merge branch 'main' into duncan-harvey/dogstatsd

fe5839f

generate Cargo.lock for dogstatsd crate

78dc404

duncanpharvey marked this pull request as ready for review March 20, 2025 17:35

duncanpharvey changed the title ~~Migrate dogstatsd~~ Migrate dogstatsd from libdatadog Mar 20, 2025

duncanista approved these changes Mar 24, 2025

View reviewed changes

duncanpharvey merged commit e86f522 into main Mar 24, 2025

duncanpharvey deleted the duncan-harvey/dogstatsd branch March 24, 2025 19:24

duncanpharvey mentioned this pull request Mar 25, 2025

Remove Serverless Specific DogStatsD Server Implementation DataDog/libdatadog#973

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Migrate dogstatsd from libdatadog #2

Migrate dogstatsd from libdatadog #2

Uh oh!

duncanpharvey commented Mar 19, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Migrate dogstatsd from libdatadog #2

Migrate dogstatsd from libdatadog #2

Uh oh!

Conversation

duncanpharvey commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

Describe how to test/QA your changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

duncanpharvey commented Mar 19, 2025 •

edited

Loading