Bracket WAI middleware spans with detachContext #116
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I believe that stripping W3C Trace Context headers can mask this issue. Which makes me wonder if this is related to https://www.iankduncan.com/articles/2023-08-28-opentelemetry-gotchas-phantom-spans. |
|
Can you rebase this against |
velveteer
force-pushed
the
wai-detach-context
branch
from
cd3c0b5
to
714edfd
Compare
|
Not sure if you were notified but I rebased this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After updating to
hs-opentelemetry-instrumentation-wai-0.1.0.0, I noticed our Datadog APM traces were starting to group unrelated requests together. Notably all of these sporadic groups shared the same GHC thread ID.Further investigation led me to this commit: 34569fc. At first glance, removing this
bracketwould seem harmless, assuming each WAI request is served by a separate thread. But in factwarpuses some heuristics to enable the HTTP keep-alive optimization, which means a single thread can serve multiple requests. Without detaching the context from the thread after serving a request, the initial request context is re-used for subsequent requests, which means propagated OTel context will be inherited by those request spans.I tested this small patch in our production application and it appears to have resolved our APM woes.