Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sampling override based on request processing time? #3637

Closed
kemuri-9 opened this issue Apr 11, 2024 · 7 comments
Closed

Sampling override based on request processing time? #3637

kemuri-9 opened this issue Apr 11, 2024 · 7 comments

Comments

@kemuri-9
Copy link

Is your feature request related to a problem? Please describe.
On a particular project I've found that most of the ingestion into application insights is from dependency metrics collected.
For us this includes JDBC, HTTP client, and Redis.
For quickly responding requests these dependency metrics aren't actually that desired, as performance evaluation of the request will not be evaluated.
I've been trying to go through the configuration to see how there could be an override to change the sampling rate for dependencies when the request is "quick" (as a concrete example, let's say <5 seconds is "quick").
But so far I've not been able to see anything like this. I have not yet found any span that was the processing time of the request to do this override.

Describe the solution you would like
Is this currently possible somehow? It would be great for us to be able to not sample dependencies for quick requests to reduce our ingestion into insights (and reduce our running cost).

Describe alternatives you have considered
Not interested in turning off all dependency sampling as this is necessary to investigate why slow requests are slow.

Additional context

@kemuri-9
Copy link
Author

Currently using v3.5.1

@heyams
Copy link
Contributor

heyams commented Apr 22, 2024

@kemuri-9 sorry for the late response.
Please try this sample app filter telemetry based on request duration. I believe this will help.

@kemuri-9
Copy link
Author

kemuri-9 commented Apr 24, 2024

Looking at the referred example, it looks like the core functionality being referred to is within https://github.com/Azure-Samples/ApplicationInsights-Java-Samples/tree/main/opentelemetry-api/java-agent/extensions/FilterSpanBasedOnDuration
and this functionality is an open telemetry extension that decides whether to export (or to not export) the span based on its duration.
So the desired functionality is not purely available in the application insights json configuration, but requires a separate code module to be implemented.
Is this understanding correct?

@heyams
Copy link
Contributor

heyams commented Apr 24, 2024

@kemuri-9 please take a closer look at the sample, it doesn't require a json configuration. sampling overrides apply at the beginning of the span. span duration is captured at the end of a span. Extensions give you more control to make decision what to export. You can filter based on any attributes or anything else that can be used uniquely identifying a span.

@kemuri-9
Copy link
Author

Sorry, had a bit of a mistyping in my last comment about the intent, I've edited it to add the missing "not".
Up until now all of the configuration for app insights has been exclusively through the json configuration, there has been no writing of extensions.
The desired clarification was that for the desired behavior, an extension must be created and utilized. there is no applicable json configuration available for application insights to avoid having to maintain open telemetry extensions

@heyams
Copy link
Contributor

heyams commented Apr 24, 2024

Looking at the referred example, it looks like the core functionality being referred to is within https://github.com/Azure-Samples/ApplicationInsights-Java-Samples/tree/main/opentelemetry-api/java-agent/extensions/FilterSpanBasedOnDuration and this functionality is an open telemetry extension that decides whether to export (or to not export) the span based on its duration. So the desired functionality is not purely available in the application insights json configuration, but requires a separate code module to be implemented. Is this understanding correct?

Yes

@kemuri-9
Copy link
Author

Then there is a methodology to achieve the desired behavior, so resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants