Activities were getting closed with incorrect duration

[carlos-neto-maersk]

Context

While using @qhris (thank you 🙏 ) work before it was published, we started to notice a weird pattern for the metrics.
Our current setup has a .NET Framework ASP.NET application instrumented with OpenTelemetry dotnet client, and we needed to have the metrics tagged with route information, so that we are able to support grafana dashboards over it (we haven't found a way to do it over trace information). We are also using OTEL to feed the connected App Insights instance.

As soon as we packed and deployed the code, due to the high load the system is subjected to (10M+ requests/day) we noticed a saw tooth pattern on the reported graphs in AppInsights and Grafana.

As you can see on the above, we have several thousand requests being saved with a duration of ~1ms. We also have the following:


Sorry for the redacted services names.

Fixes #1431 .

Changes

I reverted ActivityHelper to 895a9c7.
Since this breaks the enrichment logic developed in #1407 (this was already address in #1425 ), I've added the duration computation logic into the HttpInMetricsListener (not the best place I know, but). I've tried to "import" most of the code from Activity, to have high precision.

Next steps

I would like to further improve the code and add unit tests, but to be clear my knowledge on this (ASP.NET lifecycle pipelines) is very limited to what Microsoft (still) has on their documentation pages, so I would request aid to understand the unit test case and to work on it.

For significant contributions please make sure you have completed the following items:

• Appropriate CHANGELOG.md updated for non-trivial changes
• Design discussion issue #
• Changes in public API reviewed

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: carlos-neto-maersk / name: Carlos Neto (d4b2048)

[vishweshbankwar]

#1431 (comment)

[qhris]

I believe moving aspNetActivity.Stop(); is a no go, and we need to find another way.

You can see why by setting options.ExportProcessorType = ExportProcessorType.Simple; (for OtlpExporterOptions).
Running it like this with the changes in this PR you'll see that a few span attributes will be missing:

• http.status_code
• http.route

They are missing because the span is exported before HttpInListener.OnStopActivity is run.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.