Latency increases experienced with releases 0.76.0 and greater

[tbruketa]

On jetty/jersey service, observed a 3 fold latency increase in response time when compared to 0.65.0. Isolated the problem by reverting from 0.78.2 to 0.65.0. Investigation done so far:

• April 6 2021 Updated the version of the library from 0.65.0 to 0.76.0 - latency jumped from 330us to 1.21ms (p50)
• April 15, 2021 Latency jumped to 1.85ms due to datadog-agent side-car upgrade to 7.27.1
• April 20, 2021 Updated dd-trace-java to 0.78.2- latency dropped to 1.29ms
• April 22, 2021 Reverted dd-trace-java to 0.65.0- latency dropped to 500-700us; it may still improve due jit optimization.

There number are from our QA/staging environment that is lightly loaded. In production, with more load, these numbers are amplified (11ms (dd-trace-java-0.65.0) -> 30ms (dd-trace-java-0.76.0) -> 108ms (dd-trace-java-0.76.0, dd-agent-2.27.1). Therefore, the concern is that this may be catastrophic for services experiencing high workloads.

[richardstartin]

Hi @tbruketa can you give us more information please?

• Which instrumentations activate in your application?
• Which features do you use (e.g. tracing, profiling, jmxfetch)
• Which library versions does your application use? Can you give us your dependency tree (with your internal dependencies redacted)?
• Do you have profiles you can share with us?
• Can we have debug logs (run with -Ddd.trace.debug=true)?

Opening a support ticket by emailing support@datadoghq.com would allow sharing of this information confidentially.

[tbruketa]

Thanks @richardstartin. Just sent an email with the info requested.

[richardstartin]

Thanks @tbruketa, I've seen all that information now. I couldn't find anything interesting in the profile (a netbeans snapshot) but saw that the only instrumentation to activate was jax-rs, so this, along with some of the information already provided, narrows the problem down to the JAX-RS 1 instrumentation itself, and generic handling of server spans. I suspect that this the root cause (but I don't have hard evidence yet) was moving obfuscation of the URL to the application threads.

What you could do to rule this out would be either of the following:

1. Compare 0.67.0 with 0.68.0. If my theory is correct 0.67.0 should have similar latency to 0.65.0, and 0.68.0 should be higher.
2. Disable the URL obfuscation with trace.URLAsResourceName.enabled=false

If this is related to URL obfuscation, resource name handling for all server instrumentations is being reworked at the moment to improve performance, so it's possible that this problem will get fixed incidentally, but we'll create a jax-rs 1/jersey benchmark to be sure.