Question: Modify dependency telemetry with custom values outside of initializer/processor context

[hiraldesai]

I have previously worked with telemetry initializers and processor to add custom properties to my telemetry but I'm looking to find a way to add custom properties to dependency calls outside the initializer/processor context. I am using dotnet core HttpClientFactory and Refit to do external API calls and I have a setup similar to this:

services.AddRefitClient<IExternalApi>()
.AddHttpMessageHandler<HttpLoggingHandler>()
                    .ConfigureHttpClient(c =>
                    {
                        c.BaseAddress = new Uri(baseUrl);
                    });

Now my HttpLoggingHandler is where I'm intercepting into the outgoing HttpRequestMessage and putting some debug messages in. I want to change it to find the current dependency telemetry entry associated with the outgoing http call, add some custom properties (request/response/headers etc.) to this telemetry entry so that I can see these details of this inside application insights.

    public class HttpLoggingHandler : DelegatingHandler
    {
        protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request,
            CancellationToken cancellationToken)
        {            
            // Find the current dependency telemetry
            // Add request headers to the dependency telemetry
            // Add request content to the dependency telemetry
            
            var response = await base.SendAsync(request, cancellationToken).ConfigureAwait(false);

            // Add response header to the dependency telemetry
            // Add response content to the dependency telemetry
        }
    }

[SergeyKanzhelev]

@lmolkova would Activity.Current be already initialized here?

@hiraldesai if you have this example handy - can you try to access Activity.Current. If it's already one related to dependency - you can add Tags to it and later in telemetry initializer populate properties from Tags

[hiraldesai]

I added the following to HttpLoggingHandler, the value for Activity.Current.OperationName here is Microsoft.AspNetCore.Hosting.HttpRequestIn.

var current = Activity.Current;
current.AddTag("testkey", "testvalue");

Then I added a telemetry initializer like this:

public class HttpTelemetryInitializer : ITelemetryInitializer
{
    public void Initialize(ITelemetry telemetry)
    {
        if (!(telemetry is DependencyTelemetry dependencyTelemetry))
            return;

        var currentActivityTags = Activity.Current?.Tags?.ToList();
        if (currentActivityTags == null || !currentActivityTags.Any())
            return;
        
        dependencyTelemetry.Properties.AddRange(currentActivityTags);
    }
}

This didn't write anything against the dependency so I changed it to this for debug:

public void Initialize(ITelemetry telemetry)
{            
    if (Activity.Current == null)
        return;
    
    Debug.WriteLine($"Operation: {Activity.Current.OperationName} Telemetry: {telemetry.GetType()} Tags: {Activity.Current.Tags.Count()}" );
}

Which produced the following output.

Operation: Microsoft.AspNetCore.Hosting.HttpRequestIn Telemetry: Microsoft.ApplicationInsights.DataContracts.RequestTelemetry Tags: 0
Operation: Microsoft.AspNetCore.Hosting.HttpRequestIn Telemetry: Microsoft.ApplicationInsights.DataContracts.TraceTelemetry Tags: 0
Operation: Microsoft.AspNetCore.Hosting.HttpRequestIn Telemetry: Microsoft.ApplicationInsights.DataContracts.TraceTelemetry Tags: 0
Operation: Microsoft.AspNetCore.Hosting.HttpRequestIn Telemetry: Microsoft.ApplicationInsights.DataContracts.TraceTelemetry Tags: 0
Operation: Microsoft.AspNetCore.Hosting.HttpRequestIn Telemetry: Microsoft.ApplicationInsights.DataContracts.TraceTelemetry Tags: 0
Operation: Microsoft.AspNetCore.Hosting.HttpRequestIn Telemetry: Microsoft.ApplicationInsights.DataContracts.TraceTelemetry Tags: 1
Operation: System.Net.Http.HttpRequestOut Telemetry: Microsoft.ApplicationInsights.DataContracts.DependencyTelemetry Tags: 0
Operation: Microsoft.AspNetCore.Hosting.HttpRequestIn Telemetry: Microsoft.ApplicationInsights.DataContracts.TraceTelemetry Tags: 1
Operation: Microsoft.AspNetCore.Hosting.HttpRequestIn Telemetry: Microsoft.ApplicationInsights.DataContracts.TraceTelemetry Tags: 1
Operation: Microsoft.AspNetCore.Hosting.HttpRequestIn Telemetry: Microsoft.ApplicationInsights.DataContracts.TraceTelemetry Tags: 1
Operation: Microsoft.AspNetCore.Hosting.HttpRequestIn Telemetry: Microsoft.ApplicationInsights.DataContracts.TraceTelemetry Tags: 1
Operation: Microsoft.AspNetCore.Hosting.HttpRequestIn Telemetry: Microsoft.ApplicationInsights.DataContracts.TraceTelemetry Tags: 1
Operation: Microsoft.AspNetCore.Hosting.HttpRequestIn Telemetry: Microsoft.ApplicationInsights.DataContracts.TraceTelemetry Tags: 1
Operation: Microsoft.AspNetCore.Hosting.HttpRequestIn Telemetry: Microsoft.ApplicationInsights.DataContracts.RequestTelemetry Tags: 1

So the Activity.Current inside the HttpHandler isn't set to the HttpRequestOut and hence there's no corresponding DependencyTelemetry with it to have the values from the tags.

[lmolkova]

Activity that tracks outgoing http request is created in the innermost handler therefore is not accessible in any custom handlers.

@hiraldesai would microsoft/ApplicationInsights-dotnet-server#900 work for you?

Starting with Dependency Collector 2.7.0 (stable will be released in a couple of weeks), DependencyTelemetry has request and response object in the OperationDetails property - you can take whatever you want from them in the initializer/processor and stamp on the telemetry item.

[hiraldesai]

@lmolkova - thanks, looks like that would work for me.

Basically what that means is I need to stop trying to do any of this inside a DelegatingHandler but just use the TelemetryInitializer instead. Is that correct?

[lmolkova]

@hiraldesai precisely!

If you have more questions or any issues, please let us know and feel free to reopen this issue.

[lmolkova]

BTW, telemetry initializer may be called several times (2) on the same telemetry, you may rely on OperationDetails presence only in the last call.

[hiraldesai]

Cool thanks - I will wait for the update then!

[lmolkova]

you can try beta - https://www.nuget.org/packages/Microsoft.ApplicationInsights.DependencyCollector/2.7.0-beta4 if you don't mind trying preview packages and stable is on the way.

[hiraldesai]

It's not a burning issue for us - so I am happy to wait for the feature to be available.

[hiraldesai]

So - I ended up writing something like this and it works like a charm. This is against the Interweb's recommendations and I'm sure it will not fly for a very busy app (hence the configuration switch) 😄 We're not expecting a heavy usage on launch, so we will go with it for now.

Question: is there a limit on how much you can stuff inside custom dimensions (DependencyTelemetry.Properties)?

[hiraldesai]

@lmolkova @SergeyKanzhelev @cijothomas - bump. I just realized you may not have read my comment above because the issue is closed. I wanted to know if there is a limit to how much you can store in Properties.

[lmolkova]

I'll let @SergeyKanzhelev ad @cijothomas comment on properties limitations.

Reading response is somewhat dangerous and unstable as it may break reading response for the second time to actually send it. It also could be huge. If you still want to do this and properties cannot handle it or not convenient, perhaps you may send the response in its own trace telemetry rather than in properties. trace will be correlated to the dependency. It will be a bit unusual to send telemetry from the initializer, but I don't foresee any issues with it.