datadog exporter panic

[sean-snyk]

Component(s)

exporter/datadog

What happened?

Description

datadog exporter panics

Steps to Reproduce

🤷 run the exporter, connected to vector

Expected Result

no panic

Actual Result

panic, see log

Collector version

0.70.0

Environment information

otelcol-contrib docker image

OpenTelemetry Collector configuration

No response

Log output

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x25591b3]

goroutine 247 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter/internal/clientutil.isNonRetriable(...)
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter@v0.70.0/internal/clientutil/error_converter.go:32
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter/internal/clientutil.WrapError(...)
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter@v0.70.0/internal/clientutil/error_converter.go:25
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter.(*metricsExporter).pushSketches(0xc0016e6500, {0x780ef70, 0xc08aa40000}, {0xc0bfb03000?, 0xc0367d9098?, 0xcfe88a?})
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter@v0.70.0/metrics_exporter.go:171 +0x353
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter.(*metricsExporter).PushMetricsData.func4({0x780ef70?, 0xc08aa40000?})
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter@v0.70.0/metrics_exporter.go:251 +0x3c
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter/internal/clientutil.(*Retrier).DoWithRetries(0xc0016d5840, {0x780ef70, 0xc08aa40000}, 0xc0367d9500)
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter@v0.70.0/internal/clientutil/retrier.go:65 +0x189
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter.(*metricsExporter).PushMetricsData(0xc0016e6500, {0x780ef70, 0xc08aa40000}, {0xc0367d9868?})
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter@v0.70.0/metrics_exporter.go:250 +0xc0c
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter.(*metricsExporter).PushMetricsDataScrubbed(0xc0016e6500, {0x780ef70?, 0xc08aa40000?}, {0x0?})
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter@v0.70.0/metrics_exporter.go:182 +0x28
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsRequest).Export(0x424607?, {0x780ef70?, 0xc08aa40000?})
	go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/metrics.go:65 +0x34
go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send(0xc0016b2468, {0x782ce50, 0xc090bb7b90})
	go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/common.go:208 +0x96
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send(0xc0016e25a0, {0x782ce50?, 0xc090bb7b90?})
	go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/queued_retry.go:358 +0x185
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send(0xc0016d6210, {0x782ce50, 0xc090bb7b90})
	go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/metrics.go:136 +0x88
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1({0x782ce50, 0xc090bb7b90})
	go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/queued_retry.go:205 +0x39
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1()
	go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/internal/bounded_memory_queue.go:61 +0xb6
created by go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers
	go.opentelemetry.io/collector@v0.70.0/exporter/exporterhelper/internal/bounded_memory_queue.go:56 +0x45

### Additional context

This was observed on multiple pods after upgrading from 0.66 -> 0.70

[github-actions]

Pinging code owners:

• exporter/datadog: @KSerrania @mx-psi @gbbr @knusbaum @amenasria @dineshg13

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mx-psi]

Thanks for reporting this @sean-snyk, I think this relates to #17386, we don't check if the *http.Response object is nil and it probably is for certain errors.

If I am right, v0.70.0 should be the only version affected, so it should be safe to bump to v0.69.0

[mx-psi]

Fix will be released on v0.71.0, two workarounds until then are to either use v0.69.0 or to disable the native client by using --feature-gates=-exporter.datadogexporter.metricexportnativeclient

[sean-snyk]

We're running with --feature-gates=-exporter.datadogexporter.metricexportnativeclient when we see the crash

[songy23]

Thanks for flagging this and apologies for the confusion - #17386 happened to also change Zorkian sketches (i.e. histograms) export to also call the new function:

opentelemetry-collector-contrib/exporter/datadogexporter/metrics_exporter.go

Line 167 in 0881216

resp, err = exp.client.HttpClient.Do(req)

So setting --feature-gates=-exporter.datadogexporter.metricexportnativeclient won't help. For now the only workaround is to use v0.69.0.