Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using Application Insights java Agent is impossible to suppress some handled exceptions #2941

Closed
apescione opened this issue Feb 28, 2023 · 14 comments · Fixed by #3022
Closed
Assignees

Comments

@apescione
Copy link

Context

I'm using a Spring Boot Application (Java) and WebFlux as the web layer (Reactor), and Tomcat is used as an Embedded web server.
In my application, I've Server-Sent-Event API like this:

  @RequestMapping(value = "/messages",
        produces = {"text/event-stream", "application/stream+json"},
        method = RequestMethod.GET)
    public Mono<ResponseEntity<Flux<ServerSentEvent<String>>>> getSseMessages() {

This API sends events to clients using a HOT Stream.
I'm using the application insight java agent to instrument my application deployed into a docker container.
When a client disconnects himself from my application, the Spring application, in the next update through the client raises a
ClientAbortException and inner cause java.io.IOException: Broken pipe.

Expected behavior

Having a way to choose what exception I want to track. So, in my scenario, I don't want to consider this kind of exception "ClientAbortException and inner cause java.io.IOException: Broken pipe." on a particular API, because I know that sooner or later a client will disconnect itself (Close front end application, change page, etc...)

Actual behavior

Reading Microsoft documentation, it seems there is a way to choose the "Sampling" for each metric, so that, if you set percentage to 0, you can suppress the metric and avoid that agent sents it to cloud.
Here's the documentation: https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-sampling-overrides

Despite it is a preview, it works fine for AgentLogExporter, in fact, if the "exception" is coming from a log, it can be managed, but if the metric is from TelemetryItemExporter, it seems there is no way to suppress it.
Here's the metric that I'd like to suppress:

2023-02-27 16:25:38.808Z DEBUG c.a.m.o.e.i.p.TelemetryItemExporter - sending telemetry to ingestion service:
....
{"ver":1,"name":"Exception","time":"2023-02-27T16:25:38.31Z","iKey":"9805179f-a509-4c79-ac20-bf6753b671f9","tags":{"ai.internal.sdkVersion":"java:3.4.9","ai.operation.id":"434698b7492fa6cbc81e233da0f92af4","ai.cloud.roleInstance":"9caa85af7fa8","ai.operation.name":"GET /api/messages","ai.cloud.role":"ANTONIO_TEST_APP_52","ai.user.userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36","ai.operation.parentId":"eb54096db74ef832"},"data":{"baseType":"ExceptionData","baseData":{"ver":2,"exceptions":[{"typeName":"java.io.IOException","message":"Broken pipe","stack":"java.io.IOException: Broken pipe\n\tat sun.nio.ch.FileDispatcherImpl.write0(Native Method)\n\tat sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)\n\tat sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)\n\tat sun.nio.ch.IOUtil.write(IOUtil.java:65)\n\tat sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470)\n\tat org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:135)\n\tat org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1417)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:773)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.flushNonBlocking(SocketWrapperBase.java:744)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:718)\n\tat org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.flush(Http11OutputBuffer.java:573)\n\tat org.apache.coyote.http11.filters.ChunkedOutputFilter.flush(ChunkedOutputFilter.java:157)\n\tat org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:221)\n\tat org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1255)\n\tat org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:402)\n\tat org.apache.coyote.Response.action(Response.java:209)\n\tat org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:306)\n\tat org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:273)\n\tat org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118)\n\tat org.springframework.http.server.reactive.ServletServerHttpResponse.flush(ServletServerHttpResponse.java:211)\n\tat org.springframework.http.server.reactive.ServletServerHttpResponse.access$600(ServletServerHttpResponse.java:50)\n\tat org.springframework.http.server.reactive.ServletServerHttpResponse$ResponseBodyFlushProcessor.flush(ServletServerHttpResponse.java:322)\n\tat org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor$State$3.writeComplete(AbstractListenerWriteFlushProcessor.java:312)\n\tat org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor$State$WriteResultSubscriber.onComplete(AbstractListenerWriteFlushProcessor.java:465)\n\tat org.springframework.http.server.reactive.WriteResultPublisher$State.publishComplete(WriteResultPublisher.java:266)\n\tat org.springframework.http.server.reactive.WriteResultPublisher$State$1.subscribe(WriteResultPublisher.java:171)\n\tat org.springframework.http.server.reactive.WriteResultPublisher.subscribe(WriteResultPublisher.java:77)\n\tat org.springframework.http.server.reactive.AbstractListenerWriteProcessor.subscribe(AbstractListenerWriteProcessor.java:205)\n\tat org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor$State$2.onNext(AbstractListenerWriteFlushProcessor.java:294)\n\tat org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor.onNext(AbstractListenerWriteFlushProcessor.java:120)\n\tat org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor.onNext(AbstractListenerWriteFlushProcessor.java:43)\n\tat org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.onNext(ChannelSendOperator.java:173)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onNext$1(TracingSubscriber.java:62)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:83)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onNext(TracingSubscriber.java:62)\n\tat reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onNext$1(TracingSubscriber.java:62)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:83)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onNext(TracingSubscriber.java:62)\n\tat reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmit(FluxFlatMap.java:543)\n\tat reactor.core.publisher.FluxFlatMap$FlatMapInner.onNext(FluxFlatMap.java:984)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onNext$1(TracingSubscriber.java:62)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:83)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onNext(TracingSubscriber.java:62)\n\tat reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onNext$1(TracingSubscriber.java:62)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:83)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onNext(TracingSubscriber.java:62)\n\tat reactor.core.publisher.FluxInterval$IntervalRunnable.run(FluxInterval.java:125)\n\tat reactor.core.scheduler.PeriodicWorkerTask.call(PeriodicWorkerTask.java:59)\n\tat reactor.core.scheduler.PeriodicWorkerTask.run(PeriodicWorkerTask.java:73)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:750)\n"}]}}}
...

To Reproduce

It's very simple, create with web flux a server sent event api that produce an event every few seconds, connect the client (browser is enough) and close the browser. The next event that cannot be write on stream because the client is disconnected you are going to have a IOException: Broken pipe.

Sample Application

@RestController
@RequestMapping("/api")
public class ServerSentEventApi {
...


    @RequestMapping(value = "/messages",
        produces = {"text/event-stream", "application/stream+json"},
        method = RequestMethod.GET)
    public Mono<ResponseEntity<Flux<ServerSentEvent<String>>>> getSseMessages() {
        Flux<ServerSentEvent<String>> sseFlux = Flux.interval(Duration.ofSeconds(5))
            .map(i -> ServerSentEvent.<String>builder().event("ping")
                .data(i.toString()).id(RandomStringUtils.randomAlphanumeric(8)).build());
        return Mono.just(ResponseEntity.ok().body(sseFlux));
    }
...

}

System information

Please provide the following information:

  • SDK Version: Java 8 Zulu, Spring Boot 2.6.8, Webflux 5.3.20, tomcat 9.0.63
  • OS type and version: Container Linux alphine

Consideration

Avoiding the sending of these metrics, I want to also reduce the cloud cost, because in my scenario I have always a huge number of this unuseful exceptions

Thank you in advance.

@trask
Copy link
Member

trask commented Feb 28, 2023

hi @apescione! what version of Application Insights Java are you using?

see https://github.com/microsoft/ApplicationInsights-Java/releases/tag/3.4.0:

  • Exceptions are no longer captured directly on dependency records for these reasons:
    • in order to reduce ingestion cost
    • dependency exceptions which are uncaught, bubble up to the request-level where they are already captured
    • dependency exceptions which are caught, tend to be logged if they are important, where they are also already captured
      (#2423).

@trask
Copy link
Member

trask commented Feb 28, 2023

oh, nevermind, I see you are using 3.4.9 and these are exceptions on requests (not dependencies)

@heyams
Copy link
Contributor

heyams commented Mar 7, 2023

@apescione can you push a sample app to https://github.com/microsoft/ApplicationInsights-Java-Repros for a repro? thanks.

@apescione
Copy link
Author

Hi @heyams, I have no permission on that repo. Could you create one branch to push the example on it?

@trask
Copy link
Member

trask commented Mar 8, 2023

hey @apescione, can you fork the repo and send PR from your fork?

@apescione
Copy link
Author

I've push in a my repo. You can find an example here.
https://github.com/apescione/applicationinsightTestApps
Let me know if you need more information

@heyams heyams self-assigned this Mar 13, 2023
@heyams
Copy link
Contributor

heyams commented Mar 17, 2023

@apescione

quote:
It's very simple, create with web flux a server sent event api that produce an event every few seconds, connect the client (browser is enough) and close the browser. The next event that cannot be write on stream because the client is disconnected you are going to have a IOException: Broken pipe.

can you help me with generating an IOException? I sent api request using your app and then close the browser. Nothing happens. What do you mean connect the client (browser is enough)? Can you push some repro instructions to your repo? I attached it using `java -javaagent:xxxx.jar -jar ./target/message-0.0.1-SNAPSHOT.jar' and didn't use your docker-compose.

never mind, i was able to repro using http://127.0.0.1:8080/api/messages

@heyams
Copy link
Contributor

heyams commented Mar 20, 2023

@apescione
Copy link
Author

@heyams
I'm trying the snapshot version but using both this configuration:

 "preview": {
    "sampling": {
      "overrides": [
        {
          "telemetryType": "exception",
          "attributes": [
            {
              "key": "message",
              "value": "Broken pipe",
              "matchType": "strict"
            }
          ],
          "percentage": 0
        },

or this


 "preview": {
    "sampling": {
      "overrides": [
        {
          "telemetryType": "exception",
          "attributes": [
            {
              "key": "exception.message",
              "value": "Broken pipe",
              "matchType": "strict"
            }
          ],
          "percentage": 0
        },

it doesn't work. What is wrong in these configuration?

Thank's.

@heyams
Copy link
Contributor

heyams commented Mar 22, 2023

@apescione sorry, I forgot to mention, and it's configured like this:

{
  "preview": {
    "captureRequestException" : false
  }
}

@apescione
Copy link
Author

I've just tried and it works fine. It captures custom exceptions but not the suppress exceptions. When is it going to be available in a stable release?
Thank you very much for the support

@heyams
Copy link
Contributor

heyams commented Mar 23, 2023

Yes, it will be part of our next GA release. Thank you for confirming. @apescione

@heyams
Copy link
Contributor

heyams commented May 1, 2023

@apescione you can use sampling override to filter out exceptions on request. For your test app, here is the config:

"preview": {
    "sampling": {
      "overrides": [
        {
          "telemetryType": "exception",
          "attributes": [
            {
              "key": "exception.message",
              "value": "An established connection was aborted by the software in your host machine",
              "matchType": "strict"
            }
          ],
          "percentage": 0
        }
      ]
    }
  }

attributes can be found if you turn on self-diagnostics log to debug logging level and search for exception.message, or exception.type or exception.stacktrace in the applicationinsights.log. "Broken pipe" is not in any of these properties though. this feature will become available in the next release.

@apescione
Copy link
Author

Thank you very much, "Broken pipe" is when the same application is raised from linux OS, instead your message, probably is from Windows machine. I'll try it just after you release 3.4.13 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants