Open Telemetry dropping child spans between Kafka Connect and Spring KafkaListener Consumer #8126
Replies: 7 comments 2 replies
-
Suspecting on Batch-Listener we have disabled batchListener and made max poll records to 1
concurrentKafkaListenerContainerFactory.batch.listener: false
|
Beta Was this translation helpful? Give feedback.
-
Followed https://github.com/open-telemetry/opentelemetry-java-instrumentation/discussions/7033 and added
and some other properties as well Upgraded to
for above SpringKafkaTelemetry import Still I see a lot of difference in traces count at Consumer end Please suggest any solutions here |
Beta Was this translation helpful? Give feedback.
-
Is this issue related to Because we Configured OpenTelemetry Java agent at Kafka Connect project and traces are getting generated from Kafka Connect only and also configured java agent at Spring Kafka Consumer |
Beta Was this translation helpful? Give feedback.
-
Can anyone suggest a solution to this yet? |
Beta Was this translation helpful? Give feedback.
-
Can you create a small sample application that reproduces the issue? Or is this something that happens only when you send/receive a lot of messages? |
Beta Was this translation helpful? Give feedback.
-
Irrespective of the number of messages sent/received the child spans are always getting dropped. We have checked with 10000 records at Kafka Connect for which around 4000 child records are only matching at Kafka listener in consumer (stage environment) In production, this ratio always fluctuates between 20 to 40 percent with around 10 million records at Kafka Connect on daily basis. For the sample code, I have given most of the screenshots in the chat earlier. Let me know if you need any additional information because the sample code might take some time to reproduce. |
Beta Was this translation helpful? Give feedback.
-
Thank you for your response |
Beta Was this translation helpful? Give feedback.
-
We have DockerFile which uses Jaeger all in one image
We have enabled OpenTelemetry java agent with Kafka Agent as shown below
At Kafka connect producer topic, the count (No. of records at producer topic shown in Akhq tool) is matching with below aggregated elastic search query based on Trace IDs
Now at consumer we have a configured similarly
We have a Kafka Listener at consumer which listens to messages in topic
Also added some interceptors at Kafka Container Factory
consumerProps.put(ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG, Collections.singletonList(TracingConsumerInterceptor.class));
Now
the number of spans at consumer is always less than at producer topic even though the consumption is happening for all the records
Multiple child spans are getting dropped
We are using
Spring Boot 2.3
and
Spring Kafka 2.5.1 Release
Is there any additional Configuration needed or any limitations from Kafka Instrumentation?
Any suggestions would help
Thanks
Beta Was this translation helpful? Give feedback.
All reactions