API Request -> ServiceBus -> WebJob = not logging end to end transactions

[CraigGraw]

When pushing messages onto a ServiceBus session queue from a HTTP request and then processing the message from a WebJob, Application insights no longer is able to correlate the log information in Azure as an end to end transaction.

Seen when using Azure.Messaging.ServiceBus

It only shows the ServiceBus send part of the transaction, not the dependency httpclient call when processing the message.

The same approach using the package 'Microsoft.Azure.ServiceBus' produces end to end transactions.
As can be seen here, the previous servicebus package when used with WebJob shows the dependency (httpClient call) on the processing of the message.

However the package 'Microsoft.Azure.ServiceBus' is deprecated and no longer supported.
https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/servicebus/Azure.Messaging.ServiceBus/MigrationGuide.md

Tested with Azure Functions and end to end transactions are logging.
Appears to be something wrong in the WebJob SDK integration with Application Insights.

Repro steps

Provide the steps required to reproduce the problem

Sample projects:
https://github.com/CraigGraw/WebJobServiceBus/tree/master/ApplicationInsights (Azure.Messaging.ServiceBus)
https://github.com/CraigGraw/WebJobServiceBus-Deprecated/tree/master/ApplicationInsights (Microsoft.Azure.ServiceBus)

1. Add application insights key and service bus queue to the 'appsettings.json' configs for 'APIService' and 'WebJobApp' projects.
2. Build solution and run the two projects 'APIService' and 'WebJobApp'
3. Call either Get or Post HTTP request {localhost:port}/message
4. Observe that the Webjob console logs the message has been processed.
5. Open Azure Application Insights for the associated account.

Expected behavior

Application Insights shows full end to end transaction details for the request.

Actual behavior

Application Insights shows only the post onto queue part of the transaction for the request.

Known workarounds

Revert to deprecated ServiceBus package 'Microsoft.Azure.ServiceBus'.

Related information

Provide any related information

• Package version
  .NetCore 3.1 (Tested and reproduced with .Net 6.0)
  Microsoft.Azure.ServiceBus 7.7.0
  https://github.com/CraigGraw/WebJobServiceBus/tree/master/ApplicationInsights (Azure.Messaging.ServiceBus)
  https://github.com/CraigGraw/WebJobServiceBus-Deprecated/tree/master/ApplicationInsights (Microsoft.Azure.ServiceBus)

[v-bbalaiagar]

Hi @CraigGraw , Thank you for your feedback! I checked this internally. We would like to know in what step the correlation broken.

[CraigGraw]

@v-bbalaiagar I have updated the original description to describe the difference in behaviour between the two ServiceBus packages when used with the WebJob SDK. Also I have added the source code for both of the tests using the old and new packages. To summerize I would expect application insights to be tracking the logs for both the request which sends the service bus message and the processing of the message from the queue (as it is doing in the previous ServiceBus package). I have tried all the options in the WebJob initialisation options but cannot get the behaviour seen before.

[v-bbalaiagar]

Hi @RohitRanjanMS , Could you please look into this issue

[RohitRanjanMS]

Adding @JoshLove-msft from the SDK team, @JoshLove-msft can you please have a look at this?

[JoshLove-msft]

This is due to the issue described here - microsoft/ApplicationInsights-dotnet#2151

[CraigGraw]

@JoshLove-msft thank you for the link.
How can this be classified as a feature request in a package that is replacing a deprecated library? The package 'https://www.nuget.org/packages/Microsoft.Azure.ServiceBus' has been deprecated for over a year now, consumers using this in a professional environment are unable to migrate to the new package currently as it is operationally incomplete. In a production environment the new library is not fit for purpose without the ability to trace requests through the system.
What is the current recommendation from Microsoft on which ServiceBus NuGet package to use?

[lmolkova]

@CraigGraw

you're right that the only supported SeriveBus SDK does not have proper integration with ApplicationInsights SDK yet. It's a gap and we're working on fixing it.

In the meantime, can you please use a workaround?

1. I think there is a tiny mismatch between your Deprecated and fresh samples in how you configure ApplicationInsights.

   • here, with fresh ServiceBus loggingBuilder.AddApplicationInsightsWebJobs you only configure ApplicationInsights logger and nothing else - there is no ServiceBus tracing happening
   • to fix it, you should do exactly the same you do in APIService - services.AddApplicationInsightsTelemetry(instrumentationKey);. It configures logging in addition to everything else

2. Now it's time for the workaround

   • Please add the following class

     public class ServiceBusTelemetryInitializer : ITelemetryInitializer
     {
       public void Initialize(ITelemetry telemetry)
       {
           var activity = Activity.Current;
           if (activity != null && activity.OperationName.StartsWith("ServiceBus"))
           {
               string endpoint = null;
               string queueName = null;
     
               foreach (var tag in activity.Tags)
               {
                   if (tag.Key == "peer.address")
                   {
                       endpoint = tag.Value;
                   }
                   else if (tag.Key == "message_bus.destination")
                   {
                       queueName = tag.Value;
                   }
               }
     
               if (endpoint == null || queueName == null)
               {
                   return;
               }
     
               string separator = "/";
               if (endpoint.EndsWith(separator))
               {
                   separator = string.Empty;
               }
     
               string eventHubInfo = string.Concat(endpoint, separator, queueName);
     
               if (telemetry is DependencyTelemetry dependency)
               {
                   dependency.Type = "Azure Service Bus";
                   dependency.Target = eventHubInfo;
               }
               else if (telemetry is RequestTelemetry request)
               {
                   request.Source = eventHubInfo;
               }
           }
       }
     }

   • and make sure to add it to register it next to AddApplicationInsightsTelemetry

         services.AddSingleton<ITelemetryInitializer>(new ServiceBusTelemetryInitializer());
         // services.AddApplicationInsightsTelemetry(...);

You should see something like this

Let us know if it helps!

[CraigGraw]

@lmolkova Thank you for the information.

I have applied the recommended changes and committed them to github.

https://github.com/CraigGraw/WebJobServiceBus/tree/master/ApplicationInsights

There appears to be no difference to the Application Insights logs.

Please let me know if the changes are correct.

[lmolkova]

@CraigGraw It looks like there is no telemetry coming from your WebJob Application or it's not correlated. Let's check which case it is:

1. Can you please put a breakpoint into ProcessMessageTask https://github.com/CraigGraw/WebJobServiceBus/blob/main/ApplicationInsights/WebJobApp/Application.cs#L48 and check System.Diagnostics.Activity.Current value there - is it null or what's the OperationName of this activity?

2. Do you see any telemetry collected for your HTTP client call here https://github.com/CraigGraw/WebJobServiceBus/blob/main/ApplicationInsights/WebJobApp/Application.cs#L57 ?

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[CraigGraw]

@lmolkova
The value for System.Diagnostics.Activity.Current is => {Azure.Core.Pipeline.DiagnosticScope.DiagnosticActivity}

The value for OperationName is-> "ServiceBusSessionProcessor.ProcessSessionMessage"

The telemetry for the HTTP client call is recorded but does not correlate.

[Eli-Black-Work]

@CraigGraw Looks like this might be fixed in the next release 🙂 microsoft/ApplicationInsights-dotnet#2593

[Eli-Black-Work]

@CraigGraw ApplicationInsights 2.21.0 has been released, and it looks like there's a fix for this! 🙂

[CraigGraw]

@Bosch-Eli-Black
Is there any update on a new release of the following WebJob Insight package?

https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Logging.ApplicationInsights

Currently it is at version 3.0.33 which includes ApplicationInsights version 2.17.0

[ilmax]

@CraigGraw there's a PR here #2906 waiting to be approved/merged