Missing data from OpenTelemetry output

[szatixx]

Hi!

I am trying to integrate Semantic Kernel with an ASP.NET Core app, and trace the app with the built in OpenTelemetry output. I was fallowing the official docs, but it only provides fraction of the data it should.

My output is like this:

Activity.TraceId: d6cdfcfe9b91389f78462af195684085
Activity.SpanId: cbc9b34229dfeb60
Activity.TraceFlags: Recorded
Activity.ParentSpanId: 0d93bc4473128707
Activity.DisplayName: invoke_agent review_agent
Activity.Kind: Internal
Activity.StartTime: 2025-04-16T20:58:03.4577425Z
Activity.Duration: 00:00:00.6668680
Activity.Tags:
gen_ai.operation.name: invoke_agent
gen_ai.agent.id: f888c700-c769-4f53-b958-0b1e14bb7da0
gen_ai.agent.name: review_agent
Instrumentation scope (ActivitySource):
Name: Microsoft.SemanticKernel.Diagnostics
Resource associated with Activity:
telemetry.sdk.name: opentelemetry
telemetry.sdk.language: dotnet
telemetry.sdk.version: 1.11.2
service.name: unknown_service:JokeTeller

All the powerfull data is missing, like the prompt, the token count etc.

Is there a settings to add them?

I am using .NET 9 and Microsoft.SemanticKernel 1.46.0 and Microsoft.SemanticKernel.Connectors.AzureAIInference 1.46.0-beta

Thanks
Attila

[sophialagerkranspandey]

Adding @TaoChenOSU

[TaoChenOSU]

Hi @szatixx!

Thank you for raising the question!

Did you set the following switch to true: https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability/telemetry-with-console#add-telemetry?

// Enable model diagnostics with sensitive data.
AppContext.SetSwitch("Microsoft.SemanticKernel.Experimental.GenAI.EnableOTelDiagnosticsSensitive", true);

[szatixx]

Hi @TaoChenOSU!

Yes I did. Here is my full source: https://gist.github.com/szatixx/550aa6696bd18a684f5cf19c12e7e6bc

This is newer version than I was using in my original post, but there are still missing data. Now I have successfully integrated the code with Application Insights, and the newer version of SK and Azure Inference are sending more data, so there was some improvements.

I am attaching some images of my Application Insights traces. If I am using Azure OpenAI, it looks good in the App Insights, but not in Azure AI Foundry.
With Azure Inference API its a different story, it sends less traces, and doesn't shows up in AI Foundry at all.

My company is developing a system using 8-10 agents to solve a task together. We need the AI Foundry style monitoring feature to efficiently truck and debug our production system. So while it is not fully working its a no go for us.

Also I was also surprised that the name of the agent even in the more or less working Azure OpenAI traces are missing. I see the type of the agent message (eg. system, user) and the message itself, but not the name of the agent. Since we are using many cooperating agents it is hard to "replay" the chat without knowing the name of the sender. Do you have plans to include the agent name in the telemetry?

Thanks
Attila

Inference API missing data

Azure Ai Foundry error

SK telemetry

[TaoChenOSU]

Hi @TaoChenOSU!

Yes I did. Here is my full source: https://gist.github.com/szatixx/550aa6696bd18a684f5cf19c12e7e6bc

This is newer version than I was using in my original post, but there are still missing data. Now I have successfully integrated the code with Application Insights, and the newer version of SK and Azure Inference are sending more data, so there was some improvements.

I am attaching some images of my Application Insights traces. If I am using Azure OpenAI, it looks good in the App Insights, but not in Azure AI Foundry. With Azure Inference API its a different story, it sends less traces, and doesn't shows up in AI Foundry at all.

My company is developing a system using 8-10 agents to solve a task together. We need the AI Foundry style monitoring feature to efficiently truck and debug our production system. So while it is not fully working its a no go for us.

Also I was also surprised that the name of the agent even in the more or less working Azure OpenAI traces are missing. I see the type of the agent message (eg. system, user) and the message itself, but not the name of the agent. Since we are using many cooperating agents it is hard to "replay" the chat without knowing the name of the sender. Do you have plans to include the agent name in the telemetry?

Thanks Attila

Inference API missing data

Azure Ai Foundry error

SK telemetry

Hi!

What it is showing in your screenshot are the OTel GenAI events (in Application Insights, they are categorized as traces). The agent activities will be captured as spans (or activities in .Net and dependencies on Application Insights`). If you go to the dependency table on Application Insights, you should see the agents.

The error on AI Foundry portal has been fixed. Please update to the latest version of SK to give it a try.

[szatixx]

Thanks @TaoChenOSU, it looks good.

[justinbarias]

Thanks @TaoChenOSU, it looks good.

Hi there. Im facing the issue on my ChatCompletionAgent.

The completion calls (from the agent) aren't including token usage metrics.

But when i do a simple kernel.InvokePromptAsync(), it's there.

Did you find the solution to that?

[szatixx]

@justinbarias I havent experienced this issue.

[ChrisL-GU]

@justinbarias that was my exact experience. I was about to write and say I never resolved the issue, but I just checked Application Insights and I do see these metrics there now. Ex:

customMetrics
| where name == "semantic_kernel.connectors.openai.tokens.total"
| project value, valueCount, valueMax, valueMin, valueSum

value,valueCount,valueMax,valueMin,valueSum
922,1,922,922,922
941,1,941,941,941
742,1,742,742,742
741,1,741,741,741
1621,1,1621,1621,1621
741,1,741,741,741
1059,1,1059,1059,1059
1701,1,1701,1701,1701

I don't how reliably they are being recorded at this point.

Here's the relevant pieces of code I'm currently using:

AppContext.SetSwitch("Microsoft.SemanticKernel.Experimental.GenAI.EnableOTelDiagnosticsSensitive", true);

_ = services
    .AddOpenTelemetry()
    .UseAzureMonitor(options =>
    {
        options.EnableLiveMetrics = false;
        options.ConnectionString = appInsightsConnectionString;
    })
    .WithTracing(builder =>
        builder
            .AddSource(Telemetry.SourceName)
            .AddSource("Microsoft.SemanticKernel*")
            .SetResourceBuilder(Telemetry.ResourceBuilder)
    )
    .WithMetrics(builder => builder
        .AddMeter("Microsoft.SemanticKernel*")
        .SetResourceBuilder(Telemetry.ResourceBuilder));

_ = services.
    .AddScoped<Kernel>()
    .AddAzureOpenAIChatCompletion(
        serviceId: serviceId,
        deploymentName: deploymentName,
        endpoint: modelEndpoint,
        apiKey: modelApiKey)

chatCompletionService = kernel.GetRequiredService<IChatCompletionService>(AppSettings.AssignmentDeploymentName);
var response = chatCompletionService.GetStreamingChatMessageContentsAsync(chatHistory);

[ChrisL-GU]

I've been struggling with this myself. Following the example provided in the answer above works. (@TaoChenOSU! the public facing URL is: https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability/telemetry-with-console#add-telemetry)

But when trying to replicate it using DI in a web app it seems it's more particular on how it is configured. Below are my code snippets. I do get traces, but I'm not seeing any metrics. I've spent hours trying to get it to work. I don't know what I'm missing.

AppContext.SetSwitch("Microsoft.SemanticKernel.Experimental.GenAI.EnableOTelDiagnosticsSensitive", true);

_ = services
    .AddOpenTelemetry()
    .UseAzureMonitor(options =>
    {
        options.ConnectionString = appInsightsConnectionString;
    })
    .WithTracing(builder =>
        builder
            .AddSource(Telemetry.SourceName)
            .AddSource("Microsoft.SemanticKernel*")
            .SetResourceBuilder(Telemetry.ResourceBuilder)
            .AddConsoleExporter()
    )
    .WithMetrics(builder => builder
        .AddMeter("Microsoft.SemanticKernel*")
        .SetResourceBuilder(Telemetry.ResourceBuilder)
        .AddConsoleExporter());
        
        services
            .AddKernel()
            .AddAzureOpenAIChatCompletion(
                deploymentName,
                endpoint,
                apiKey);

[Inject] public required IChatCompletionService ChatCompletionService { get; set; }

...

var response = ChatCompletionService?.GetStreamingChatMessageContentsAsync(chatHistory)

[TaoChenOSU]

Hi!

You may need to run your chat completion service multiple times to see any metrics. They are measurements overtime.