OpenTelemetry support

[RohitRanjanMS]

Introducing OpenTelemetry support for the Host enables users to adopt a method supported by OpenTelemetry for exporting telemetry data. This addition accommodates both existing AI SDK and new OpenTelemetry instrumentations, offering users the option to activate OpenTelemetry by adjusting the telemetry mode.

Telemetry Mode Configuration:

By default, the Host process is integrated with Application Insights through the AI SDK, emitting telemetry for requests, metrics, and logs. While logs are produced using Illoger, traces and metrics do not utilize OpenTelemetry's recommended APIs, such as ActivitySource and Meter.

To shift to a mode where the Host outputs telemetry in alignment with OpenTelemetry's standards and APIs, users can modify the telemetry mode within the host.json file or through an environmental variable override as follows:

{
  "telemetryMode": "openTelemetry"
}

Export Options:
Azure Monitor Exporter: Users can export telemetry to Application Insights by adding the APPLICATIONINSIGHTS_CONNECTION_STRING, enabling the Azure Monitor exporters and LiveMetrics.

OTLP Exporter: Facilitates telemetry export to any OTLP-compatible endpoint. Configuration is possible by setting the OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS for OTLP endpoints.

This implementation is compatible with all standard OpenTelemetry environment variables, allowing for the configuration of log and trace providers. As the OpenTelemetry Configuration specification becomes stable, there are plans to support more detailed configuration options, enabling users to fine-tune their telemetry behavior.

Issue describing the changes in this PR

resolves #9273

Dependencies changes

---

 Microsoft.Azure.WebJobs.Script.Tests.DependencyTests.Verify_DepsJsonChanges
 Source: DependencyTests.cs line 53
 Duration: 605 ms

Message: 
IMPORTANT: The dependencies in WebHost have changed and MUST be reviewed before proceeding. Please follow up with brettsam, fabiocav or mathewc for approval.

Previous file: C:\Users\Rohit Ranjan\Source\Repos\azure-functions-host\test\WebJobs.Script.Tests\bin\Debug\net8.0\Microsoft.Azure.WebJobs.Script.WebHost.deps.json
New file: C:\Users\Rohit Ranjan\Source\Repos\azure-functions-host\src\WebJobs.Script.WebHost\bin\debug\net8.0\Microsoft.Azure.WebJobs.Script.WebHost.deps.json

Changed:
- Azure.Core.dll: 1.35.0.0/1.3500.23.45706 -> 1.38.0.0/1.3800.24.12602

Removed:

Added:
- Azure.Monitor.OpenTelemetry.AspNetCore.dll: 1.2.0.0/1.200.24.16203
- Azure.Monitor.OpenTelemetry.Exporter.dll: 1.3.0.0/1.300.24.10803
- Azure.Monitor.OpenTelemetry.LiveMetrics.dll: 1.0.0.0/1.0.24.15803
- OpenTelemetry.Api.dll: 1.0.0.0/1.8.0.1229
- OpenTelemetry.Api.ProviderBuilderExtensions.dll: 1.0.0.0/1.8.0.1229
- OpenTelemetry.dll: 1.0.0.0/1.8.0.1229
- OpenTelemetry.Exporter.Console.dll: 1.0.0.0/1.6.0.1006
- OpenTelemetry.Exporter.OpenTelemetryProtocol.dll: 1.0.0.0/1.8.0.1229
- OpenTelemetry.Extensions.Hosting.dll: 1.0.0.0/1.8.0.1229
- OpenTelemetry.Instrumentation.AspNetCore.dll: 1.0.0.0/1.7.0.1114
- OpenTelemetry.Instrumentation.Http.dll: 1.0.0.0/1.7.0.1114
- OpenTelemetry.PersistentStorage.Abstractions.dll: 1.0.0.0/1.0.0.0
- OpenTelemetry.PersistentStorage.FileSystem.dll: 1.0.0.0/1.0.0.0
- System.ClientModel.dll: 1.0.0.0/1.0.24.5302

Expected: True
Actual: False

Stack Trace: 
DependencyTests.Verify_DepsJsonChanges() line 125
RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

I will update the documentation later.

Pull request checklist

• My changes do not require documentation changes
  • Otherwise: Documentation issue linked to PR
• My changes should not be added to the release notes for the next release
  • Otherwise: I've added my notes to release_notes.md
• My changes do not need to be backported to a previous version
  • Otherwise: Backport tracked by issue/PR #issue_or_pr
• My changes do not require diagnostic events changes
  • Otherwise: I have added/updated all related diagnostic events and their documentation (Documentation issue linked to PR)
• I have added all required tests (Unit tests, E2E tests)

[jviau]

Customer config approach for preview looks good to me. Primary areas of concern are:

1. Some telemetry semantics.
2. Conditioning the resource detector / azure-specific stuff to only when actually hosted by functions serverless model.
3. Performance impact of the telemetry processors in this PR. Lets do our best to avoid allocations as much as possible in these as they are hot-paths.

• OTel code may have examples

[jviau]

Are we not providing an environment variable way of enabling OTel? Or relying on default env variable config mapping?

[julealgon]

Where is this documented exactly? Is this PR the only place where there are setup instructions? They feel incredibly barebones to me.

Additionally, does this only support environment variable-based configuration? All of the OpenTelemetry libraries work with the IConfiguration model and that's how we setup up all of our services, by having settings such as OTEL_EXPORTER_OTLP_ENDPOINT in our shared Azure AppConfiguration instance that propagates it to dozens of separate projects.

The description on the PR seems to indicate our setup would not work, which is why I'd like for someone here to link to a more thorough documentation of OpenTelemetry + Azure Functions.

EDIT:
Found what seems to be the main documentation here:

• Use OpenTelemetry with Azure Functions