How to ignore some traces or methods being collected

[kingkongpoon]

Java 8
SpringBoot 2.3.5.RELEASE
otel 1.19.0

my project have a kafka consumer
then it will collect too many traces which I don't want

code like this

@Component
public class KafkaMessageReceive {
    @KafkaListener(topics = "test-kafka")
    public void processMessageString(String content) {
        log.info("test-kafka", content);
    }
}

Operation like this
KafkaMessageListenerContainer$ListenerConsumer$$Lambda$.run

this is otlp metrics for per minute

-Dotel.traces.exporter=otlp -Dotel.exporter.otlp.endpoint=http://xxxx:xx -Dotel.metrics.exporter=otlp 
 -Dotel.service.name=666666  -Dotel.logs.exporter=none 
-Dotel.metric.export.interval=60000 -Dotel.javaagent.exclude-classes="com.example.service.KafkaMessageReceive"  
-javaagent:xxx\opentelemetry-javaagent.jar

how to ignore it?

[mateuszrzeszutek]

Hey @kingkongpoon ,

Can you post full contents of that unwanted span?

[kingkongpoon]

HI @mateuszrzeszutek .
do you mean this?

KafkaMessageListenerContainer$ListenerConsumer$$Lambda$.run
org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer$$Lambda$1163/1414529337

or the original span data in elasticsearch ?

some of my springboot project will consume message from kafka ,and the message can be 6 million per minute .
so too many unused traces/spans be collected.

how to ignore it ?

[kingkongpoon]

Although there is no new message from kafka ,it will still produce new traces from kafka listener consumer,why?

thread.name ThreadPoolTaskScheduler-1

it seems like a Scheduler's traces ?

[mateuszrzeszutek]

Yeah, these spans are created by the spring scheduler.
To work around this issue, can you try running your application with the -Dotel.instrumentation.spring-scheduling.enabled=false option? It'll disable the spring scheduler instrumentation.

[kingkongpoon]

Ok, it works,thanks a lot. :)

[kingkongpoon]

In the same project,why it notice
clock skew adjustment disabled; not applying calculated delta of xxxx ?

[mateuszrzeszutek]

That's a jaeger feature: https://www.jaegertracing.io/docs/1.21/deployment/#clock-skew-adjustment

[kingkongpoon]

OK ,thank you ! 👍

[kingkongpoon]

Hello @mateuszrzeszutek
I think this problem still exist
this is my env prod span data

jaeger

otlp

otlp span collector metrics

it still seem too many traces ?

[mateuszrzeszutek]

Hey @kingkongpoon ,

What version of the agent are you using? The fix I made was included as a part of the 1.22.0 release.

[kingkongpoon]

@mateuszrzeszutek Hello
the kafka problem has sloved,
but the new problem seem different
I update to 1.22.1
but the problem do not slove.
is there any metrics like processedSpans can tell me which method product such tracs/spans?

[mateuszrzeszutek]

The processedSpans counter is defined in the SDK repo: https://github.com/open-telemetry/opentelemetry-java/blob/aa873a05be27137d276de5fa50557add5f78184c/sdk/trace/src/main/java/io/opentelemetry/sdk/trace/export/BatchSpanProcessor.java#L200

If you're missing it, then perhaps there is a bug in the OTel SDK - please file an issue in the opentelemetry-java repo.

[kingkongpoon]

Hey @mateuszrzeszutek
I can find processedSpans
but it only tell me how many spans, it can not tell me why my program product lot's of useless spans.
is there any metrics like this

so I can find out why product so many spans ?

[kingkongpoon]

some spans like this how to exclude ?
may be like -Dotel.instrumentation.spring-scheduling.enabled=false ?

[mateuszrzeszutek]

but it only tell me how many spans, it can not tell me why my program product lot's of useless spans.
is there any metrics like this

No, there are no metrics like that. You can investigate the spans themselves though.

some spans like this how to exclude ?
may be like -Dotel.instrumentation.spring-scheduling.enabled=false ?

These look like SERVER spans; perhaps there is some process that periodically calls your application? Disabling the server instrumentations entirely is not exactly a good idea; you could use a custom sampler to filter these out though - take a look at #1060, might be useful for your case.

Anyway, since this deviated from the original topic, if you run into any other problems please open a new issue.

[MartinEmrich]

What would be the equivalent for -Dotel.instrumentation.spring-scheduling.enabled=false for RuntimeAttach.attachJavaagentToCurrentJvm();?

[trask]

hi @MartinEmrich, can you open a new issue for your question? thx

[MartinEmrich]

@trask done: #10076