grpc metrics exporter client starts and keeps logging context deadline exceeded after stably running for about 2.5 hours #2217
Labels
area:metrics
Part of OpenTelemetry Metrics
bug
Something isn't working
pkg:exporter:otlp
Related to the OTLP exporter package
Projects
Description
The grpc metrics exporter client keeps consistently logging
context deadline exceeded
after stably running for about 2.5 hours. The server it is reporting to, an open telemetry collector, is still running but the client disconnects and is just not able to reconnect for some reason. If I start another process using the client and connect to the collector otlp grpc receiver endpoint, it's able to connect. Also this happens in while running on an ubuntu based container in k8s (Both the collector and client are running there). I could not reproduce it on mac os(darwin).I switched to the http metrics exporter client and I did not observe this problem. I was previously using v0.18.0 of opentelemetry-go and did not observe this issue.
Environment
Collector log with timestamp for when the receiver started:
Exporter metrics client logs "context deadline exceeded" start happening after about 2.5 hours.
They do not stop till the process running the client is killed.
Steps To Reproduce
Expected behavior
It should continue to happily report metrics to the otel collector for however long it runs.
The text was updated successfully, but these errors were encountered: