How to correctly add a custom AI telemetry processor

[acblksun]

Hi there,

We're looking to add a custom AI telemetry processor when using the azure webjobs SDK nuget package
Microsoft.Azure.WebJobs.Logging.ApplicationInsights. We require some filtering for some of the automatically collected telemetry. In this case the telemetry we wish to filter is from Azure table storage. Our App Insights is flooded (and alerts on) 401 failures to table storage items we know might not be there.

We use dotnet core 3.1 for our api as well as webjobs. In the api we have done this in the past using a dependency filter like so:

public void ConfigureServices(IServiceCollection services)
{
	// snip - other service configuration
	
	
	// Add App Insights telemetry
	services.AddApplicationInsightsTelemetry(options =>
	{
		// Instrumentation from "ApplicationInsights:InstrumentationKey"
		options.AddAutoCollectedMetricExtractor = true; // default is true
		options.InstrumentationKey = Configuration.GetValue<string>("ApplicationInsights:InstrumentationKey");
		options.DeveloperMode = Env.IsDevelopment();
	});

	// MyDependencyTelemetryFilter is of type ITelemetryProcessor
	services.AddApplicationInsightsTelemetryProcessor<MyDependencyTelemetryFilter>();
}

Repro steps

For our webjob, we have

var hostBuilder = new HostBuilder()
// other builders skipped
.ConfigureLogging((context, logging) =>
{
	logging
		.AddApplicationInsightsWebJobs(options =>
		{
			options.EnableDependencyTracking = true;
			options.InstrumentationKey = context.Configuration.GetValue<string>("ApplicationInsights:InstrumentationKey");

			// TODO - add ITelemetryProcessor telemetry filter >> here <<
			// logging.Services.AddApplicationInsightsTelemetryProcessor<MyDependencyTelemetryFilter>();

		});
		
	// TODO - add ITelemetryProcessor telemetry filter >> or here <<
	// logging.Services.AddApplicationInsightsTelemetryProcessor<MyDependencyTelemetryFilter>();
}

We have also tried

var hostBuilder = new HostBuilder()
// other builders skipped
.ConfigureServices((context, services) =>
{
	// TODO - add ITelemetryProcessor telemetry filter >> or here <<
	services.AddApplicationInsightsTelemetryProcessor<MyDependencyTelemetryFilter>();
}

Expected behavior

For apps written by using ASP.NET Core or WorkerService, adding a new telemetry processor is done by using the AddApplicationInsightsTelemetryProcessor extension method on IServiceCollection, as shown. This method is called in the ConfigureServices method of your Startup.cs class.

Documented https://docs.microsoft.com/en-us/azure/azure-monitor/app/api-filtering-sampling#create-a-telemetry-processor-c

var hostBuilder = new HostBuilder()
// other builders skipped
.ConfigureServices((context, services) =>
{
	services.AddApplicationInsightsTelemetryProcessor<MyDependencyTelemetryFilter>();
}

Actual behavior

Attempting to add to using services.AddApplicationInsightsTelemetryProcessor<MyDependencyTelemetryFilter>(); does not work and the telemetry filter is unused.

Known workarounds

There are two workarounds

Workaround 1 - using Microsoft.ApplicationInsights.WorkerService

https://docs.microsoft.com/en-us/azure/azure-monitor/app/worker-service

Application Insights is releasing a new SDK, called Microsoft.ApplicationInsights.WorkerService, which is best suited for non-HTTP workloads like messaging, background tasks, console applications etc.

The only time we could get it to work was removing Microsoft.Azure.WebJobs.Logging.ApplicationInsights and replacing it with Microsoft.ApplicationInsights.WorkerService. However it appears the webjobs sdk has a lot of custom filtering that we would prefer not to lose.

Workaround 2 - manual addition to TelemetryProcessorChainBuilder in ConfigureServices

The author of the open issue #2459 which talks about ordering or telemetry processors discusses the following code. I have verified this appears to work, but the author, @digitalfuller, indicates this is not best practice and there's no comments on the issue.

new HostBuilder()
    .ConfigureLogging((context, builder) =>
    {
         builder.AddApplicationInsightsWebJobs(o => 
        {
            o.InstrumentationKey = appInsightsKey;
        }
    }
    .ConfigureServices(services =>
    {
        var sp = services.BuildServiceProvider();
        var tc = sp.GetRequiredService<TelemetryConfiguration>();
        var tpcb = tc.TelemetryProcessorChainBuilder;

        tpcb.Use(next => new MyDependencyTelemetryFilter(next));

        tpcb.Build();
    })

Related information

Provide any related information

• target framework netcoreapp3.1
• Microsoft.Azure.WebJobs version 3.0.16
• Microsoft.Azure.WebJobs.Logging.ApplicationInsights version 3.0.14
• https://github.com/Azure/azure-webjobs-sdk/blob/dev/src/Microsoft.Azure.WebJobs.Logging.ApplicationInsights/Extensions/ApplicationInsightsLoggingBuilderExtensions.cs
• https://github.com/Azure/azure-webjobs-sdk/blob/dev/src/Microsoft.Azure.WebJobs.Logging.ApplicationInsights/Extensions/ApplicationInsightsServiceCollectionExtensions.cs

[tanujnarula]

Do we have any resolution for this? We have exact same issue.

[chinwobble]

We have the same issue too

[sergevm]

Bumping into issues too. We migrated a .net framework solution with web jobs to dotnet core with an update of the app insights SDK's. Because of a breaking behaviour in the way the SDK logs stuff since the update, I wanted to see if I can revert things by using a processor. But the processor doesn't trigger. Tried the second fix mentioned above, but it looks like the Request telemetry that I start for the operation does not invoke the processor, while MetricTelemetry etc do.

As for the breaking behaviour, I see in the Azure portal that the request telemetry adds a bunch of custom properties, amongst which the "OperationName" custom property, which is apparently always set to the name of the method.

The method where the request is handled, is actually triggered in the web job with messages on a service bus, and we want to reflect the name of the message in the logging, instead or the generic method name (the method is choosing the handler for the message). So we use a StartOperation call passing in the name of the message, but while the request name is set accordingly, in the custom properties the method name keeps showing as the operation name.

[NickMSW]

We are running into this issue as well. Are there any updates?

[v-anvari]

Tagging @brettsam for further insights

[wmcainsh]

Same as above, would be good to get a resolution to this as we are getting swamped with synthetic request log entries since moving to Azure Front Door.

[AkshitaKukreja30]

This solution worked for me :

builder.AddApplicationInsightsWebJobs(
loggerOptionsConfiguration: loggerOptions =>
{
loggerOptions.ConnectionString = configuration["ApplicationInsightsConnectionStringBlackSecret"];
loggerOptions.EnableLiveMetrics = true;
},
additionalTelemetryConfiguration: telemetryConfiguration =>
{
telemetryConfiguration.TelemetryProcessorChainBuilder.Use(next => new SuppressDependencyFilter(next));
telemetryConfiguration.TelemetryProcessorChainBuilder.Build();
}
);

SuppressDependencyFilter is my custom Dependency Filter implementing the interface ITelemetryProcessor with the Business Logic to suppress the unwanted dependencies from Telemetry Processor chaining.