Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/datadog] Refactor Datadog Exporter to use logs agent exporter #32327

Merged
merged 69 commits into from
May 6, 2024
Merged
Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
bf2f37b
Add logsagentexporter and implementation of logs agent init
liustanley Apr 11, 2024
1c94ca9
Address PR comments
liustanley Apr 11, 2024
4aa0290
Edit go modules
liustanley Apr 15, 2024
1eca1a7
Edit go modules
liustanley Apr 15, 2024
4d9c020
Update go modules
liustanley Apr 16, 2024
3d8b883
Add config options for compression
liustanley Apr 16, 2024
522e400
Add logs agent config
liustanley Apr 17, 2024
a39bd8a
Add tests
liustanley Apr 18, 2024
ee00145
Add new logs config options in collector.yaml
liustanley Apr 18, 2024
68fec8b
Merge branch 'main' into stanley.liu/logsagentexporter
liustanley Apr 18, 2024
fc53dc5
go mod tidy
liustanley Apr 18, 2024
72d2f55
Fix tests
liustanley Apr 18, 2024
ba8b385
Update go version and datadog-agent commit
liustanley Apr 19, 2024
ca2bb09
Merge branch 'main' into stanley.liu/logsagentexporter
liustanley Apr 19, 2024
a20ea1a
make gotidy
liustanley Apr 19, 2024
2a7a388
Add tests
liustanley Apr 19, 2024
a097514
go mod tidy
liustanley Apr 19, 2024
9c3b51b
make goporto
liustanley Apr 22, 2024
c545a49
Fix test data race
liustanley Apr 22, 2024
28682fb
Fix lint
liustanley Apr 22, 2024
1c170c1
go mod tidy
liustanley Apr 22, 2024
5b7a344
Merge branch 'main' into stanley.liu/logsagentexporter
liustanley Apr 22, 2024
651d789
go mod tidy
liustanley Apr 22, 2024
58aeea4
Fix datadog-api-client-go version
liustanley Apr 22, 2024
1e3f45f
Fix datadog-api-client-go version
liustanley Apr 22, 2024
83ec28f
Move logsagent files to new directory
liustanley Apr 22, 2024
f557abc
Change to if else to switch
liustanley Apr 22, 2024
14bd647
Sync datadog-agent module versions
liustanley Apr 22, 2024
4d72a74
Remove PeerServiceAggregation
liustanley Apr 22, 2024
80df01a
Change build endpoints
liustanley Apr 22, 2024
13e6371
Add changelog
liustanley Apr 22, 2024
b7a6817
go mod tidy
liustanley Apr 22, 2024
0a316fe
Revert makefile
liustanley Apr 22, 2024
f8645b5
Move logs agent code to datadog-agent
liustanley Apr 24, 2024
8912a02
Update dependencies and PR feedback
liustanley Apr 24, 2024
68cce51
PR feedback
liustanley Apr 29, 2024
c193163
Merge branch 'main' into stanley.liu/logsagentexporter
liustanley Apr 29, 2024
97e8e29
go mod tidy
liustanley Apr 29, 2024
ecfbd78
go mod tidy
liustanley Apr 29, 2024
5a73215
Fix tests
liustanley Apr 29, 2024
31f348a
New changelog
liustanley Apr 29, 2024
b3e53fe
Move code to logs_exporter.go
liustanley Apr 29, 2024
968e0c1
PR feedback
liustanley Apr 29, 2024
456cd70
PR feedback
liustanley Apr 29, 2024
e9dbc2f
Fix tests
liustanley Apr 29, 2024
81f86c2
Fix tests
liustanley Apr 29, 2024
6990a5a
Add retry on network error test
liustanley Apr 30, 2024
8ebafb4
Change go version
liustanley Apr 30, 2024
cc9c8b2
Change trace agent version
liustanley Apr 30, 2024
2334d67
PR feedback
liustanley Apr 30, 2024
3f6edd8
Merge branch 'main' into stanley.liu/logsagentexporter
liustanley Apr 30, 2024
cc3230e
go mod tidy
liustanley Apr 30, 2024
49427ac
Fix config override
liustanley Apr 30, 2024
11ec80f
Change config
liustanley Apr 30, 2024
23624fb
Merge branch 'main' into stanley.liu/logsagentexporter
liustanley May 1, 2024
f7cd734
go mod tidy
liustanley May 1, 2024
9ddefe5
Revert peer service aggregation changes
liustanley May 2, 2024
6c8695b
Merge branch 'main' into stanley.liu/logsagentexporter
liustanley May 2, 2024
379fb38
go mod tidy
liustanley May 2, 2024
d3b95d1
PR feedback
liustanley May 2, 2024
b68cda9
Fix tests
liustanley May 2, 2024
3603277
Fix tests
liustanley May 2, 2024
e209761
Fix lint
liustanley May 2, 2024
c098248
Apply suggestions from code review
liustanley May 3, 2024
bb612e0
PR feedback
liustanley May 3, 2024
8df5497
PR feedback
liustanley May 3, 2024
1db4d1a
Remove hostname service err
liustanley May 3, 2024
815a763
Fix config error
liustanley May 3, 2024
8477f0b
One last module update
liustanley May 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .chloggen/datadogexporter_logsagentexporter.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: enhancement

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: exporter/datadog

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Introduces the logs agent exporter for exporting logs to Datadog under the "exporter.datadogexporter.logsagentexporter" feature flag.
liustanley marked this conversation as resolved.
Show resolved Hide resolved

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [32327]
13 changes: 13 additions & 0 deletions .chloggen/datadogexporter_peerserviceaggregation.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: breaking

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: exporter/datadog

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Removes the PeerServiceAggregation config option which was previously deprecated. Please use PeerTagsAggregation instead.
liustanley marked this conversation as resolved.
Show resolved Hide resolved

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [32327]
72 changes: 63 additions & 9 deletions cmd/configschema/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -218,23 +218,67 @@ require (
github.com/ClickHouse/clickhouse-go/v2 v2.23.0 // indirect
github.com/Code-Hex/go-generics-cache v1.3.1 // indirect
github.com/DataDog/agent-payload/v5 v5.0.114 // indirect
github.com/DataDog/datadog-agent/pkg/obfuscate v0.52.1-0.20240321095122-a3c5dbb936ae // indirect
github.com/DataDog/datadog-agent/pkg/proto v0.52.1-0.20240321095122-a3c5dbb936ae // indirect
github.com/DataDog/datadog-agent/pkg/remoteconfig/state v0.52.1-0.20240321095122-a3c5dbb936ae // indirect
github.com/DataDog/datadog-agent/pkg/trace v0.52.1-0.20240321095122-a3c5dbb936ae // indirect
github.com/DataDog/datadog-agent/pkg/util/cgroups v0.52.1-0.20240321095122-a3c5dbb936ae // indirect
github.com/DataDog/datadog-agent/pkg/util/log v0.52.1-0.20240321095122-a3c5dbb936ae // indirect
github.com/DataDog/datadog-agent/pkg/util/pointer v0.52.1-0.20240321095122-a3c5dbb936ae // indirect
github.com/DataDog/datadog-agent/pkg/util/scrubber v0.52.1-0.20240321095122-a3c5dbb936ae // indirect
github.com/DataDog/datadog-agent/comp/core/config v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/comp/core/hostname/hostnameinterface v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/comp/core/log v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/comp/core/secrets v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/comp/core/telemetry v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/comp/def v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/comp/logs/agent/config v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/comp/otelcol/logsagentpipeline v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/comp/otelcol/logsagentpipeline/logsagentpipelineimpl v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/comp/otelcol/otlp/components/exporter/logsagentexporter v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/collector/check/defaults v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/config/env v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/config/model v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/config/setup v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/config/utils v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/logs/auditor v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/logs/client v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/logs/diagnostic v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/logs/message v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/logs/metrics v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/logs/pipeline v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/logs/processor v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/logs/sds v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/logs/sender v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/logs/sources v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/logs/status/statusinterface v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/logs/status/utils v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/obfuscate v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/proto v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/remoteconfig/state v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/status/health v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/telemetry v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/trace v0.54.0-rc.1.0.20240430175334-c06093ab9650 // indirect
github.com/DataDog/datadog-agent/pkg/util/backoff v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/cgroups v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/executable v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/filesystem v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/fxutil v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/hostname/validate v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/http v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/log v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/optional v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/pointer v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/scrubber v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/startstop v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/statstracker v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/system v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/system/socket v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/winutil v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/version v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-api-client-go/v2 v2.25.0 // indirect
github.com/DataDog/datadog-go/v5 v5.5.0 // indirect
github.com/DataDog/dd-sensitive-data-scanner/sds-go/go v0.0.0-20240419161837-f1b2f553edfe // indirect
github.com/DataDog/go-sqllexer v0.0.9 // indirect
github.com/DataDog/go-tuf v1.0.2-0.5.2 // indirect
github.com/DataDog/go-tuf v1.1.0-0.5.2 // indirect
github.com/DataDog/gohai v0.0.0-20230524154621-4316413895ee // indirect
github.com/DataDog/opentelemetry-mapping-go/pkg/inframetadata v0.15.0 // indirect
github.com/DataDog/opentelemetry-mapping-go/pkg/otlp/logs v0.15.0 // indirect
github.com/DataDog/opentelemetry-mapping-go/pkg/quantile v0.15.0 // indirect
github.com/DataDog/sketches-go v1.4.4 // indirect
github.com/DataDog/viper v1.13.2 // indirect
github.com/DataDog/zstd v1.5.2 // indirect
github.com/GehirnInc/crypt v0.0.0-20200316065508-bb7000b8a962 // indirect
github.com/GoogleCloudPlatform/grpc-gcp-go/grpcgcp v1.5.0 // indirect
Expand Down Expand Up @@ -283,9 +327,11 @@ require (
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.28.6 // indirect
github.com/aws/smithy-go v1.20.2 // indirect
github.com/benbjohnson/clock v1.3.5 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/blang/semver/v4 v4.0.0 // indirect
github.com/bmatcuk/doublestar/v4 v4.6.1 // indirect
github.com/briandowns/spinner v1.23.0 // indirect
github.com/buger/jsonparser v1.1.1 // indirect
github.com/cenkalti/backoff v2.2.1+incompatible // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
Expand Down Expand Up @@ -398,6 +444,7 @@ require (
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/hashicorp/nomad/api v0.0.0-20240306004928-3e7191ccb702 // indirect
github.com/hashicorp/serf v0.10.1 // indirect
github.com/hectane/go-acl v0.0.0-20190604041725-da78bae5fc95 // indirect
github.com/hetznercloud/hcloud-go/v2 v2.6.0 // indirect
github.com/iancoleman/strcase v0.3.0 // indirect
github.com/imdario/mergo v0.3.16 // indirect
Expand All @@ -424,6 +471,7 @@ require (
github.com/jpillora/backoff v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/julienschmidt/httprouter v1.3.0 // indirect
github.com/kardianos/osext v0.0.0-20190222173326-2bc1f35cddc0 // indirect
github.com/karrick/godirwalk v1.17.0 // indirect
github.com/klauspost/compress v1.17.8 // indirect
github.com/klauspost/cpuid/v2 v2.2.5 // indirect
Expand Down Expand Up @@ -462,6 +510,7 @@ require (
github.com/moby/sys/mountinfo v0.6.2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect
github.com/mongodb-forks/digest v1.1.0 // indirect
github.com/montanaflynn/stats v0.7.0 // indirect
github.com/mostynb/go-grpc-compression v1.2.2 // indirect
Expand Down Expand Up @@ -548,6 +597,7 @@ require (
github.com/ovh/go-ovh v1.4.3 // indirect
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
github.com/paulmach/orb v0.11.1 // indirect
github.com/pelletier/go-toml v1.9.3 // indirect
github.com/pelletier/go-toml/v2 v2.1.0 // indirect
github.com/philhofer/fwd v1.1.2 // indirect
github.com/pierrec/lz4 v2.6.1+incompatible // indirect
Expand Down Expand Up @@ -588,8 +638,10 @@ require (
github.com/spf13/afero v1.11.0 // indirect
github.com/spf13/cast v1.6.0 // indirect
github.com/spf13/cobra v1.8.0 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/spf13/viper v1.18.2 // indirect
github.com/stormcat24/protodep v0.1.8 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635 // indirect
Expand Down Expand Up @@ -679,6 +731,8 @@ require (
go.opentelemetry.io/otel/trace v1.26.0 // indirect
go.opentelemetry.io/proto/otlp v1.2.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/dig v1.17.0 // indirect
go.uber.org/fx v1.18.2 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.0 // indirect
golang.org/x/crypto v0.22.0 // indirect
Expand Down
Loading
Loading