Is there a way to sample overrides EventHub messages and send events?

[pdefreitas]

Is your feature request related to a problem? Please describe.
I've tried to do sample override for EventHub messages but I wasn't able to do so. I can't find any documentation example allowing me to do so. I've tried attributes like messaging.system strict to eventhubs and servicebus, and even net.peer.name strict targetting the eventhub connection string hostname in question.

Describe the solution you would like
Users would be able to do sample overrides for EventHub dependencies.

Describe alternatives you have considered
I've considering using different fields to filter the dependency like az.namespace strict to Microsoft.EventHub. I've collected available attributes from the debug log but they don't seem to work for this telemetry type.

Additional context
I'm using AppInsights Java 3.4.12.

[heyams]

Hi @pdefreitas can you follow this doc:

https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-sampling-overrides#example-suppress-collecting-telemetry-for-a-noisy-dependency-call

and https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-sampling-overrides#span-attributes-available-for-sampling

this is how you find out the dependency attributes in the debug self-diagnostics logs:

Alternatively, you can submit a sample app at https://github.com/microsoft/ApplicationInsights-Java-Repros, and then I can add code to demonstrate how to apply sampling overrides on your app.

[pdefreitas]

@heyams The exporting span logs was described in the alternatives I've considered.

You can find here two examples for EventHubs.message and EventHubs.send. I've redacted some information for obvious reasons.

2023-06-20 19:01:54.542Z DEBUG c.m.a.a.i.exporter.AgentSpanExporter - exporting span: DelegatingSpanData{spanContext=ImmutableSpanContext{traceId=5320a6ea8f219342a909f8657d3af743, spanId=a7e6d1db1894d7c8, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.19.0, attributes={container.id="REDACTED", host.arch="amd64", host.name="REDACTED", os.description="REDACTED", os.type="linux", process.command_line="REDACTED", process.pid=1, process.runtime.description="Temurin OpenJDK 64-Bit Server VM REDACTED", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="1.8.0_372-b07", service.name="REDACTED", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.25.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=Azure-OpenTelemetry, version=null, schemaUrl=null, attributes={}}, name=EventHubs.message, kind=PRODUCER, startEpochNanos=1687287714529000000, attributes={applicationinsights.internal.item_count=1, az.namespace="Microsoft.EventHub", message_bus.destination="REDACTED", peer.address="REDACTED.servicebus.windows.net", thread.id=96}, events=[], links=[], status=ImmutableStatusData{statusCode=OK, description=}, endEpochNanos=1687287714529060701, hasEnded=true, totalRecordedEvents=0, totalRecordedLinks=0, totalAttributeCount=6}
2023-06-20 19:01:54.643Z DEBUG c.m.a.a.i.exporter.AgentSpanExporter - exporting span: DelegatingSpanData{spanContext=ImmutableSpanContext{traceId=0aa7163a8c45c1f3f084e76d204fd9d0, spanId=264f5fd6563bcdc8, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.19.0, attributes={container.id="REDACTED", host.arch="amd64", host.name="REDACTED", os.description="REDACTED", os.type="linux", process.command_line="REDACTED", process.executable.path="REDACTED", process.pid=1, process.runtime.description="Temurin OpenJDK 64-Bit Server VM REDACTED", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="1.8.0_372-b07", service.name="REDACTED", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.25.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=Azure-OpenTelemetry, version=null, schemaUrl=null, attributes={}}, name=EventHubs.send, kind=CLIENT, startEpochNanos=1687287714529000000, attributes={applicationinsights.internal.item_count=1, az.namespace="Microsoft.EventHub", message_bus.destination="REDACTED", peer.address="REDACTED.servicebus.windows.net", thread.id=96}, events=[], links=[], status=ImmutableStatusData{statusCode=OK, description=}, endEpochNanos=1687287714575070314, hasEnded=true, totalRecordedEvents=0, totalRecordedLinks=0, totalAttributeCount=6}

I've tried the following sampling overrides:

{
  "telemetryType": "dependency",
  "attributes": [
    {
      "key": "messaging.system",
      "value": "eventhubs",
      "matchType": "strict"
    }
  ],
  "percentage": 0
},
{
  "telemetryType": "dependency",
  "attributes": [
    {
      "key": "messaging.system",
      "value": "servicebus",
      "matchType": "strict"
    }
  ],
  "percentage": 0
},
{
  "telemetryType": "dependency",
  "attributes": [
    {
      "key": "az.namespace",
      "value": "Microsoft.EventHub",
      "matchType": "strict"
    }
  ],
  "percentage": 0
},
{
  "telemetryType": "dependency",
  "attributes": [
    {
      "key": "peer.address",
      "value": ".*servicebus\\.windows\\.net.*",
      "matchType": "regexp"
    }
  ],
  "percentage": 0
},

And it still exports the spans above and dependency appears on AppInsights.

[heyams]

can you try this

{
  "connectionString": "...",
  "preview": {
    "sampling": {
      "overrides": [
        {
          "telemetryType": "dependency",
          "attributes": [
            {
              "key": "az.namespace",
              "value": "Microsoft.EventHub",
              "matchType": "strict"
            },
            {
              "key": "peer.address",
              "value": "*.servicebus.windows.net",
              "matchType": "regexp"
            }
          ],
          "percentage": 0
        }
      ]
    }
  }
}

here are your attributes:

  attributes={
    applicationinsights.internal.item_count=1,
    az.namespace="Microsoft.EventHub",
    message_bus.destination="REDACTED",
    peer.address="REDACTED.servicebus.windows.net",
    thread.id=96
  },

[pdefreitas]

@heyams it seems your proposed solution generates an invalid configuration:

*************************
Application Insights Java Agent 3.4.12 startup failed (PID 1)
*************************

Description:
A telemetry filter configuration has an invalid regex: *.servicebus.windows.net

Action:
Please provide a valid regex in the telemetry filter configuration.

So I've tried the following configuration, which still exports EventHub spans. Note that I've tried to filter through az.namespace already without success:

{
  "telemetryType": "dependency",
  "attributes": [
    {
      "key": "az.namespace",
      "value": "Microsoft.EventHub",
      "matchType": "strict"
    },
    {
      "key": "peer.address",
      "value": ".*.servicebus.windows.net",
      "matchType": "regexp"
    }
  ],
  "percentage": 0
},

[heyams]

@pdefreitas can you submit a repro sample app to https://github.com/microsoft/ApplicationInsights-Java-Repros?

[pdefreitas]

@heyams I'm afraid that won't be possible with exact same parameters. What I can state is that we're using Spring Framework with the package com.azure.azure-messaging-eventhubs version 5.10.1.
I expect this issue happens in other scenarios since I don't see any example provided of sampling overrides for EventHub.

[heyams]

@pdefreitas i want to see how the exception is generated. a simple app can be done right? you can remove all the credentials after testing. then i can fill in my own resources.

[pdefreitas]

@heyams what do you mean by exception? I'm trying to do sampling override on a dependency.
If it is in regards to the startup issue, it's because the regex you've provided is not valid.

[heyams]

@pdefreitas sorry, got mixed up on another github issue.
I saw you fixed the regex express

{
"telemetryType": "dependency",
"attributes": [
{
"key": "az.namespace",
"value": "Microsoft.EventHub",
"matchType": "strict"
},
{
"key": "peer.address",
"value": ".*.servicebus.windows.net",
"matchType": "regexp"
}
],
"percentage": 0
},

this config is good. but sampling overrides still didn't work for you. if you provide a sample app, it will be helpful and quicker.

[pdefreitas]

@heyams repro posted here: https://github.com/pdefreitas/ApplicationInsights-Java-Repros

It's so trivial and with microsoft components that I did not expect that I would have to do such a repro 👀

[heyams]

@pdefreitas thanks! I have lots of issues and other priorities to go through. anything that you can contribute to help me speed up the process is appreciated.

[pdefreitas]

@heyams the server is running in port 9001, tested with java 8 and java 11 on linux, check application.properties and fill the necessary EventHub settings. Calling / will trigger sending a EventHub message to the given topic. mvn package will get the agent version in test and copy the configuration file from resources to target as well. The connection string in resources for AppInsights needs to be updated.

[heyams]

@pdefreitas I'm getting this exception:
exception=com.azure.core.amqp.exception.AmqpException: Themessagingentity'sb://<my event hub>/samplingOverridesDebugging'couldnotbefound.

any ideas?

i didn't find any attributes in the log for "peer.address" and "az.namespace".

[heyams]

additionally, getting this

 curl -X POST http://localhost:9001/
{"timestamp":"2023-06-22T18:28:02.287+00:00","status":405,"error":"Method Not Allowed","path":"/"}

[pdefreitas]

@heyams For EventHub connection string for this demo, I'm using Connection string–primary key which starts with Endpoint=.

The REST part I've left the same of the repro. It is a GET / and it sends an EventData payload with string test.

[pdefreitas]

@heyams commands to I've used to build and run

mvn clean
mvn package
java -javaagent:target/agent/applicationinsights-agent.jar -jar target/app.jar

Doing the endpoint request:

curl http://localhost:9001
OK

Logs are stored in target/agent/applicationinsights.log:

2023-06-22 20:07:27.074+01:00 DEBUG c.m.a.a.i.exporter.AgentSpanExporter - exporting span: SpanData{spanContext=ImmutableSpanContext{traceId=f8a619afb0286736eaa913b7eb552b25, spanId=caab0509a774e8ca, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=f8a619afb0286736eaa913b7eb552b25, spanId=980901197c40e8fb, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.19.0, attributes={host.arch="amd64", host.name="ubuntu-1804", os.description="Linux 5.4.0-150-generic", os.type="linux", process.command_args=[/usr/lib/jvm/java-11-openjdk-amd64/bin/java, -javaagent:target/agent/applicationinsights-agent.jar, -jar, target/app.jar], process.executable.path="/usr/lib/jvm/java-11-openjdk-amd64/bin/java", process.pid=2743, process.runtime.description="Ubuntu OpenJDK 64-Bit Server VM 11.0.19+7-post-Ubuntu-0ubuntu120.04.1", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="11.0.19+7-post-Ubuntu-0ubuntu120.04.1", service.name="master", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.25.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=Azure-OpenTelemetry, version=null, schemaUrl=null, attributes={}}, name=EventHubs.send, kind=CLIENT, startEpochNanos=1687460846455461543, endEpochNanos=1687460846992249854, attributes=AttributesMap{data={message_bus.destination=temporary, az.namespace=Microsoft.EventHub, applicationinsights.internal.operation_name=GET /, thread.id=40, peer.address=REDACTED.servicebus.windows.net, applicationinsights.internal.item_count=1, thread.name=http-nio-9001-exec-2}, capacity=128, totalAddedValues=7}, totalAttributeCount=7, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=OK, description=}, hasEnded=true}

With newer versions of azure-messaging-eventhubs this span is no longer exported, probably due to #3084. This is also reproducible by bumping the dependency version to 5.15.6.

[heyams]

@pdefreitas i'm not able to repro your issue

microsoft/ApplicationInsights-Java-Repros#5

i only found one dependency associated with the request.

your app didn't work for me. so i changed the root() implementation. i used the latest azure-messing-eventhubs and latest java agent. can you try to repro using my PR?

[pdefreitas]

@heyams Probably due to the A&A differences between the two demos. Check my message above, for some reason the latest azure-messaging-eventhubs is not exporting spans.

AppInsights version	azure-messaging-eventhubs	EventHub span exported without sampling override?	Sampling override working ?
3.4.12	5.10.1	Yes	No
3.4.12	5.15.6	No	N/A
3.4.14	5.10.1	Yes	No
3.4.14	5.15.6	No	N/A

My repro is still exporting with 5.10.2 with sampling override to 0%:

[heyams]

does it imply that if you use the latest the azure-messaging-eventhubs, it is no longer an issue. no more dependencies and there is nothing to be filtered.

[pdefreitas]

@heyams I would say that not having spans after the upgrade is itself a bug? If someone wants to track these dependencies they're not able to do that anymore. If #3084 is fixed (3.4.9 -> 3.4.10), we might have this issue again because from my new tests with downgraded Application Insights:

AppInsights version	azure-messaging-eventhubs	EventHub span exported without sampling override?	Sampling override working ?
3.4.9	5.15.6	Yes	No

With Application Insights 3.4.9, the span is exporting with sampling overrides of this issue set to 0%.

[heyams]

I will work on #3084 to see if i can repro it.

[heyams]

#3084 is resolved. Do you want to try it again and let me know if i can close this issue?

[pdefreitas]

@heyams thanks for following up, this situation seems to be fixed for now!