OpenTelemetry feature

[RohitRanjanMS]

Introducing OpenTelemetry support for the Host enables users to adopt a method supported by OpenTelemetry for exporting telemetry data. This addition accommodates both existing AI SDK and new OpenTelemetry instrumentations, offering users the option to activate OpenTelemetry by adjusting the telemetry mode.

Telemetry Mode Configuration:

By default, the Host process is integrated with Application Insights through the AI SDK, emitting telemetry for requests, metrics, and logs. While logs are produced using Illoger, traces and metrics do not utilize OpenTelemetry's recommended APIs, such as ActivitySource and Meter.

To shift to a mode where the Host outputs telemetry in alignment with OpenTelemetry's standards and APIs, users can modify the telemetry mode within the host.json file or through an environmental variable override as follows:

{
  "telemetryMode": "openTelemetry"
}

Export Options:
Azure Monitor Exporter: Users can export telemetry to Application Insights by adding the APPLICATIONINSIGHTS_CONNECTION_STRING, enabling the Azure Monitor exporters and LiveMetrics.

OTLP Exporter: Facilitates telemetry export to any OTLP-compatible endpoint. Configuration is possible by setting the OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS for OTLP endpoints.

This implementation is compatible with all standard OpenTelemetry environment variables, allowing for the configuration of log and trace providers. As the OpenTelemetry Configuration specification becomes stable, there are plans to support more detailed configuration options, enabling users to fine-tune their telemetry behavior.

Issue describing the changes in this PR

resolves #9273

I will update the documentation later.

Pull request checklist

• My changes do not require documentation changes
  • Otherwise: Documentation issue linked to PR
• My changes should not be added to the release notes for the next release
  • Otherwise: I've added my notes to release_notes.md
• My changes do not need to be backported to a previous version
  • Otherwise: Backport tracked by issue/PR #issue_or_pr
• My changes do not require diagnostic events changes
  • Otherwise: I have added/updated all related diagnostic events and their documentation (Documentation issue linked to PR)
• I have added all required tests (Unit tests, E2E tests)

[jviau]

@RohitRanjanMS can you give a description of this PR to help guide the review?

1. What new config / settings are you introducing? What behavior do they influence?
2. What features does this PR bring in?
3. As a customer, how do I leverage the feature?

And some specific questions regarding OTel vs AppInsights SDK

1. Are we continuing to use the AppInsights SDK in the host alongside OTel?
2. If so, what is preventing us from completely replacing AppInsights SDK with OTel? Is it telemetry shape? Or some feature gap?

[lsl-2020]

Hi Rohit, a quick question here: after OTel mode is enabled in host, how could we make use of it in the worker side in isolated-worker mode? Where could the worker get the parent w3c trace context from the host in durable tasks? (Is it from Microsoft.Azure.Functions.Worker.Grpc.Messages.InvocationRequest.TraceContext?)
I just want to confirm if these will be covered in follow-up PRs in related repos such as durabletask or azure-functions-durable-extensions or azure-functions-dotnet-worker.

[RohitRanjanMS]

Hi Rohit, a quick question here: after OTel mode is enabled in host, how could we make use of it in the worker side in isolated-worker mode? Where could the worker get the parent w3c trace context from the host in durable tasks? (Is it from Microsoft.Azure.Functions.Worker.Grpc.Messages.InvocationRequest.TraceContext?) I just want to confirm if these will be covered in follow-up PRs in related repos such as durabletask or azure-functions-durable-extensions or azure-functions-dotnet-worker.

Hi @lsl-2020 , we will take care of the context propagation. Here's the PR for Isolated - Azure/azure-functions-dotnet-worker#2375

[RohitRanjanMS]

I'm closing this pull request because I need to delete this branch.
New PR- #9985

[lsl-2020]

Hi Rohit, a quick question here: after OTel mode is enabled in host, how could we make use of it in the worker side in isolated-worker mode? Where could the worker get the parent w3c trace context from the host in durable tasks? (Is it from Microsoft.Azure.Functions.Worker.Grpc.Messages.InvocationRequest.TraceContext?) I just want to confirm if these will be covered in follow-up PRs in related repos such as durabletask or azure-functions-durable-extensions or azure-functions-dotnet-worker.

Hi @lsl-2020 , we will take care of the context propagation. Here's the PR for Isolated - Azure/azure-functions-dotnet-worker#2375

Thank you @RohitRanjanMS for the good work! Looking forward to the release.