Distributed Context in ASP.NET and Open Telemetry

[tarekgh]

This issue is tracking the work that needs to be done for supporting the distributed context scenarios for ASP.NET and OpenTelemetry.

Here are the scenarios we need to support:

Distributed Context scenarios

These scenarios talking about ILogger and OpenTelemetry scenarios. OpenTelemetry has a goal to implement all these scenarios if you replace “ILogger” with “OpenTelemetry Logging API”. The goal of this document is to come up with scenarios that will not require OpenTelemetry to introduce new concepts and libraries. And out-of-the-box ASP.NET apps be more distributed apps friendly.

Out of the box ASP.NET application

• Distributed trace identifiers are automatically propagated through the app. This enables the correlation of telemetry across multiple components even when only a subset of these components has monitoring enabled.
  Note: This is already implemented via DiagnosticsSource.Activity.

• Log messages and exceptions are automatically associated with the distributed trace identifiers. This allows pinpointing the root cause of the problem faster by associating logs and exceptions with the downstream components that might have caused the problem or upstream - components providing incorrect data.
  Note: This is partially implemented - trace-id and span-id of a topmost request are added to the ILogger scope.

• Users may output distributed trace identifiers as part of an error page for the app user reference. This allows improving application supportability by providing a fast lookup reference that will simplify customer communication with support personnel.

Scoping of distributed logs

• The developer may add a name/value pairs to the scope of execution. This name/value pair will be associated with all logs captured in this scope
  Note: this is an existing ILogger feature

• The developer may configure some of the name/value pairs in scope to be distributed. These scope values will be transmitted over the wire and associated with the logs reported by downstream services. This feature enables scenarios like clients sending client app versions as a context property and all logs across many layers of a distributed trace will be attributed to it.

ASP.NET app with OpenTelemetry enabled

• When OpenTelemetry is enabled - all logs reported by ILogger must bye be attributed with the OpenTelemetry span identity. So telemetry reported by OpenTelemetry and ILogger can be cross-referenced.

• Spans reported by OpenTelemetry can use the scope’s name/value pairs configured by the customer for ILogger as dimensions for telemetry. This will ensure that OpenTelemetry benefits from app attribution made by the developer for better diagnostics experience. This includes both - local and distributed contexts.

Metrics and scopes

• Metrics aggregation in OpenTelemetry may use the scope’s name/value pairs as a dimension for aggregation based on configuration. This includes both - local and distributed contexts.

• EventCounters may use the scope’s name/value pairs as a dimension for aggregation. This includes both - local and distributed contexts.

Distributed “HTTP context”

• The developer must be able to set up business-critical properties that can be used across the entire distributed trace.

Tracked work items:

• Make Activity and Span the same. @tarekgh
• Add Ids and Context to all logs. @tarekgh
• Distributed Context API (distributed http context). Should be in a separate library. @tarekgh
  • string/string
  • Scope-able
  • Not attached to Activity

Ref:
OpenTelemetry .NET implementation
Improvements in .NET Core 3.0 for troubleshooting and monitoring distributed apps blog

CC @SergeyKanzhelev @pakrym

[tarekgh]

@sdmaclea did you close this one by mistake?

[sdmaclea]

The ID in the commit message was wrong. GitHub recently started using the comment message to close PRs... I saw it was wrong when I created the PR, and fixed the PR, but I didn't realize GH would close based on the commit message... My apologies...

[tarekgh]

Use new Activity to Replace OT Span: open-telemetry/opentelemetry-dotnet#660

[tarekgh]

Closing this one as all items are done except the last one which already tracked by another issue #31393.