Skip to content

Spring Actuator created extra resources in APM #9917

@ffffionn

Description

@ffffionn

Tracer Version(s)

1.54.0, 1.55.0

Java Version(s)

25.0.36

JVM Vendor

Eclipse Adoptium / Temurin

Bug Report

After updating from 1.40.1 to 1.55.0, I noticed a lot of requests for a Resource named OperationHandler.handle inside APM on a Service. I eventually linked it to Spring Actuator requests, mostly from a Kubernetes health check.
We have ".*/actuator.*" set to be excluded at the Datadog Agent level in the cluster, but there seemed to be a new span created for each actuator call.

I gradually reverted the library version and once I got to 1.53.0, the issue went away.

Here's a bit of an excerpt for a single Trace from the logs with DD_TRACE_DEBUG=true:

2025-11-07 14:43:01.907 [tomcat-handler-60] DEBUG datadog.trace.agent.core.propagation.HttpCodec - Extract incomplete context TagContext{baggage={}, }
2025-11-07 14:43:01.907 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpanContext - setResourceName `null`->`null` with priority 0->0 for traceId=3536004845519906260 spanId=8720177333644726388
2025-11-07 14:43:01.908 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpan - Started span: DDSpan [ t_id=3536004845519906260, s_id=8720177333644726388, p_id=0 ] trace=edi-gateway-api/servlet.request/servlet.request tags={_dd.injection.mode=k8s, _dd.profiling.enabled=0, _dd.trace_span_attribute_schema=0, env=dev, language=jvm, process_id=1, runtime-id=297c889b-aaf3-4e12-a504-b8eb641f364b, thread.id=137, thread.name=tomcat-handler-60, version=2.36.0-feature_java_25-SNAPSHOT-2800}, duration_ns=0, forceKeep=false, links=[]
2025-11-07 14:43:01.908 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpanContext - setResourceName `null`->`GET /edi/api/actuator/health` with priority 0->1 for traceId=3536004845519906260 spanId=8720177333644726388
2025-11-07 14:43:01.909 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpanContext - setResourceName `GET /edi/api/actuator/health`->`GET /edi/api/actuator/health` with priority 1->1 for traceId=3536004845519906260 spanId=8720177333644726388
2025-11-07 14:43:01.910 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpanContext - setResourceName `GET /edi/api/actuator/health`->`GET /edi/api/actuator/health` with priority 1->3 for traceId=3536004845519906260 spanId=8720177333644726388
2025-11-07 14:43:01.913 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpanContext - setResourceName `GET /edi/api/actuator/health`->`GET /edi/api/actuator/health` with priority 3->3 for traceId=3536004845519906260 spanId=8720177333644726388
2025-11-07 14:43:01.916 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpanContext - setResourceName `null`->`null` with priority 0->0 for traceId=3536004845519906260 spanId=1749850157596654480
2025-11-07 14:43:01.917 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpan - Started span: DDSpan [ t_id=3536004845519906260, s_id=1749850157596654480, p_id=8720177333644726388 ] trace=edi-gateway-api/spring.handler/spring.handler tags={_dd.injection.mode=k8s, env=dev, thread.id=137, thread.name=tomcat-handler-60, version=2.36.0-feature_java_25-SNAPSHOT-2800}, duration_ns=0, forceKeep=false, links=[]
2025-11-07 14:43:01.918 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpanContext - setResourceName `null`->`OperationHandler.handle` with priority 0->0 for traceId=3536004845519906260 spanId=1749850157596654480
2025-11-07 14:43:01.921 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpan - Finished span (PENDING): DDSpan [ t_id=3536004845519906260, s_id=1749850157596654480, p_id=8720177333644726388 ] trace=edi-gateway-api/spring.handler/OperationHandler.handle *measured* tags={_dd.injection.mode=k8s, component=spring-web-controller, env=dev, language=jvm, span.kind=server, thread.id=137, thread.name=tomcat-handler-60, version=2.36.0-feature_java_25-SNAPSHOT-2800}, duration_ns=4889105, forceKeep=false, links=[]
2025-11-07 14:43:01.924 [tomcat-handler-60] DEBUG datadog.trace.agent.common.writer.RemoteWriter - Enqueued for serialization: [DDSpan [ t_id=3536004845519906260, s_id=8720177333644726388, p_id=0 ] trace=edi-gateway-api/servlet.request/GET /edi/api/actuator/health *measured* tags={_dd.agent_psr=1.0, _dd.injection.mode=k8s, _dd.profiling.enabled=0, _dd.trace_span_attribute_schema=0, _sample_rate=1, component=tomcat-server, env=dev, http.hostname=100.64.82.138, http.method=GET, http.route=/edi/api/actuator/health, http.status_code=200, http.url=http://100.64.82.138:8088/edi/api/actuator/health, http.useragent=ELB-HealthChecker/2.0, language=jvm, peer.ipv4=172.18.31.109, peer.port=16282, process_id=1, runtime-id=297c889b-aaf3-4e12-a504-b8eb641f364b, servlet.context=/, servlet.path=/edi/api/actuator/health, span.kind=server, thread.id=137, thread.name=tomcat-handler-60, version=2.36.0-feature_java_25-SNAPSHOT-2800}, duration_ns=15676391, forceKeep=false, links=[], DDSpan [ t_id=3536004845519906260, s_id=1749850157596654480, p_id=8720177333644726388 ] trace=edi-gateway-api/spring.handler/OperationHandler.handle *measured* tags={_dd.injection.mode=k8s, component=spring-web-controller, env=dev, language=jvm, span.kind=server, thread.id=137, thread.name=tomcat-handler-60, version=2.36.0-feature_java_25-SNAPSHOT-2800}, duration_ns=4889105, forceKeep=false, links=[]]
2025-11-07 14:43:01.924 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpan - Finished span (WRITTEN): DDSpan [ t_id=3536004845519906260, s_id=8720177333644726388, p_id=0 ] trace=edi-gateway-api/servlet.request/GET /edi/api/actuator/health *measured* tags={_dd.agent_psr=1.0, _dd.injection.mode=k8s, _dd.profiling.enabled=0, _dd.trace_span_attribute_schema=0, _sample_rate=1, component=tomcat-server, env=dev, http.hostname=100.64.82.138, http.method=GET, http.route=/edi/api/actuator/health, http.status_code=200, http.url=http://100.64.82.138:8088/edi/api/actuator/health, http.useragent=ELB-HealthChecker/2.0, language=jvm, peer.ipv4=172.18.31.109, peer.port=16282, process_id=1, runtime-id=297c889b-aaf3-4e12-a504-b8eb641f364b, servlet.context=/, servlet.path=/edi/api/actuator/health, span.kind=server, thread.id=137, thread.name=tomcat-handler-60, version=2.36.0-feature_java_25-SNAPSHOT-2800}, duration_ns=15676391, forceKeep=false, links=[]
2025-11-07 14:43:01.996 [dd-remote-config] DEBUG datadog.communication.ddagent.DDAgentFeaturesDiscovery - discovered traceEndpoint=v0.4/traces, metricsEndpoint=v0.6/stats, supportsDropping=true, supportsLongRunning=true, dataStreamsEndpoint=v0.1/pipeline_stats, configEndpoint=null, evpProxyEndpoint=evp_proxy/v4/, telemetryProxyEndpoint=telemetry/proxy/
2025-11-07 14:43:02.942 [dd-trace-processor] DEBUG datadog.trace.agent.common.writer.ddagent.DDAgentApi - Success while sending 1 (size=1KB) traces. Total: 89, Received: 89, Sent: 89, Failed: 0.
2025-11-07 14:43:02.943 [dd-trace-processor] DEBUG datadog.trace.agent.common.writer.PayloadDispatcherImpl - Successfully sent 1 traces to the API

Expected Behavior

No extra Resources in the APM Service page relating to ignored Resources

Reproduction Code

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugBug report and fix

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions