Issues since upgrading to 0.41.0

[RafiGreenberg]

We recently upgraded from 0.40.0 to 0.41.0. Since then we have seen a huge increase in reported requests and a huge drop in latency. I'm guessing this is due to #1165
We are tracing a Spring MVC application.

Our traffic has been stable, but DD is reporting some major differences:

Specifically, I can attribute it to a JSP resource:

Any idea? Why is it considering the JSP as an "endpoint"? Can this feature be disabled?

[devinsba]

Can you check the component tag on a span from that endpoint to make sure we definitely look at the right module, it doesn't hurt to create a support ticket either support@datadoghq.com

[RafiGreenberg]

component: java-web-servlet

I will open a ticket as well, thanks

[RafiGreenberg]

Can we disable this feature in the current version? dd.integration.servlet-filter.enabled ?
It's causing issues with all of our monitors that use APM metrics for alerts.
Or can we just exclude that endpoint? We have alerts that use average latency

[RafiGreenberg]

We downgraded to 0.40.0 and the issue went away. But we would still like to know why this happened and if there is a solution.

[tylerbenson]

Thanks so much for reporting this issue. We have identified the problem and are working on a fix. If you want, you should be able to disable the "servlet-dispatcher" instrumentation to resolve the issue until we get the fix released.

[tylerbenson]

@mackoftrack any chance you'd be willing to try a snapshot build to confirm the fix? We'd like to include this in our release tomorrow, so please let us know ASAP.

[RafiGreenberg]

@tylerbenson we only run the APM in our production environment, so I'd rather not. but once the release goes out I will deploy it in our next release cycle.

[RafiGreenberg]

@devinsba @tylerbenson the issue appears to be fixed after deploying the 0.42.0 release. One question though - it appears that the endpoints are now grouped differently. E.g. before there was an entry GET /lists/view.* and now it seems to be broken up into individual GET /lists/?/xxxx. It doesn't seem to be problematic, I just wanted to confirm that this is the expected behavior.

[devinsba]

@mackoftrack Are you seeing more, or less, resource names than you were before? ie: is the cardinality higher?

I would theorize that the interaction between these spans and the datadog.opentracing.decorator's is slightly different than before

[RafiGreenberg]

It seems there are more resource names than before. But we are also getting restarted pods

Seems to be OOM restarts. Is dd-trace-java using more RAM than before?

In the meantime we are downgrading back to 0.40.0. We rarely get pod restarts like this and can't attribute this to any code changes in our app (there were no changes to the java code, only to static assets)

[tylerbenson]

Thanks for the update. If you happen to have a heap dump, please file a support ticket so you can provide that directly to us.

[dougqh]

Seems to be OOM restarts. Is dd-trace-java using more RAM than before?
Alas, there's not a simple answer to that question.

In version 0.42, we reduced the size of an internal cache to reduce the JVM's live set size.
However, when things are evicted from the cache, we might end up reloading them which can lead to additional churn especially in young gen.

In our experiments, this approach performs better in terms of memory and CPU when the heap is tightly constrained. However, if the heap max is larger than physical memory or the container limit, this can lead to problems.

Unfortunately, many older JVMs don't have good defaults particularly for containerized environments.