Most spans and traces are invalid(zeros) in k8s env

[sbaldin]

We have our own starter that brings sleuth + opentracing for our services(it uses spring boot 2.7.x), on local machine everythings works fine, we able to see correct traceids and spans, but in dev or prod k8s cluser, we are observing zeros traceId and spanId in logs.

Most of the request's logs are having zeroes instead of traceId.

Starter brings following dependencies:

io.opentelemetry:opentelemetry-exporter-otlp:1.28.0
org.springframework.cloud:spring-cloud-sleuth-otel-autoconfigure:1.1.4
org.springframework.cloud:spring-cloud-starter-sleuth:3.1.9

Logs of requests looks like:

{"@timestamp":"2024-03-21T10:46:51.692+03:00","message":"Send request to itself","logger":"com.example.controller.TestTracingController","thread":"http-nio-8080-exec-8","level":"INFO","traceId":"bc2d1f2d61519d91282cc2d582e731c6","spanId":"9b8b5e340f144c3b"}
.....
.....
{"@timestamp":"2024-03-21T10:46:57.945+03:00","message":"terminal call with delay 1.","logger":"com.example.controller.TestTracingController","thread":"http-nio-8080-exec-2","level":"INFO","traceId":"00000000000000000000000000000000","spanId":"0000000000000000"}

I have been unable to reproduce the issue locally. While it is possible to localize the problem, it does not provide a clear understanding of the root cause.

As I understand span and traces created by org.springframework.cloud.sleuth.instrument.web.tomcat.TraceValve which delegates it to org.springframework.cloud.sleuth.otel.bridge.OtelHttpServerHandler. It should creates SdkSpan
however, if creation of span fails, it creates an invalid Span as PropagatedSpan.

We are seeking guidance on what may have been done incorrectly. Any assistance would be greatly appreciated.

[jkwatson]

I think you should open this ticket with the sleuth team. We don't control any of that integration.

[sbaldin]

ok thanks