Implement new POTel span processor #3223
Conversation
* create a new otel context `_SCOPES_KEY` that will hold a tuple of
`(curent_scope, isolation_scope)`
* the `current_scope` will always be forked (like on every span creation/context update in practice)
* note that this is on `attach`, so not on all copy-on-write context
object creation but only on apis such as
[`trace.use_span`](https://github.com/open-telemetry/opentelemetry-python/blob/ba22b165471bde2037620f2c850ab648a849fbc0/opentelemetry-api/src/opentelemetry/trace/__init__.py#L547)
or [`tracer.start_as_current_span`](https://github.com/open-telemetry/opentelemetry-python/blob/ba22b165471bde2037620f2c850ab648a849fbc0/opentelemetry-api/src/opentelemetry/trace/__init__.py#L329)
* basically every otel `context` fork corresponds to our `current_scope` fork
* the `isolation_scope` currently will not be forked
* these will later be updated, for instance when we update our top
level scope apis that fork isolation scope, that will also have a
corresponding change in this `attach` function
* create a new otel context `_SCOPES_KEY` that will hold a tuple of
`(curent_scope, isolation_scope)`
* the `current_scope` will always be forked (like on every span creation/context update in practice)
* note that this is on `attach`, so not on all copy-on-write context
object creation but only on apis such as
[`trace.use_span`](https://github.com/open-telemetry/opentelemetry-python/blob/ba22b165471bde2037620f2c850ab648a849fbc0/opentelemetry-api/src/opentelemetry/trace/__init__.py#L547)
or [`tracer.start_as_current_span`](https://github.com/open-telemetry/opentelemetry-python/blob/ba22b165471bde2037620f2c850ab648a849fbc0/opentelemetry-api/src/opentelemetry/trace/__init__.py#L329)
* basically every otel `context` fork corresponds to our `current_scope` fork
* the `isolation_scope` currently will not be forked
* these will later be updated, for instance when we update our top
level scope apis that fork isolation scope, that will also have a
corresponding change in this `attach` function
|
tested simple nested trace import sentry_sdk
from opentelemetry import trace
from time import sleep
sentry_sdk.init(
debug=True,
traces_sample_rate=1.0,
_experiments={"otel_powered_performance": True},
)
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("request"):
sleep(0.1)
with tracer.start_as_current_span("db"):
sleep(0.5)
with tracer.start_as_current_span("redis"):
sleep(0.2)
with tracer.start_as_current_span("http"):
sleep(1)
|
| # if have a root span ending, we build a transaction and send it | ||
| self._flush_root_span(span) |
There was a problem hiding this comment.
I'm thinking what happens if there's a child span that, for whatever reason (weird async cases? wonky instrumentation?), hasn't finished before the root span. Since we're not using on_start at all, there'll be virtually no record of the span at this point and the parent transaction will be sent. Once the child finishes, we'll run on_end with the now orphaned span and it'll be saved in _children_spans, but never sent, since the parent transaction is already gone. So we might have a potential leak there.
Do we need some sort of cleanup of orphaned spans? Should we wait a bit before flushing the transaction to account for child spans possibly ending very close to the parent span, but a bit late? IIRC I've noticed JS also having a small sleep in place.
TBH not sure how much of a real world problem late child spans are, but I can imagine they happen.
There was a problem hiding this comment.
Another option: Also use on_start to create some record of the span and then in on_end, if a transaction comes and we detect that it has unfinished child spans, wait for a grace period to give them time to finish?
There was a problem hiding this comment.
some heuristic cleanup logic will be done in a follow up PR yes, I haven't thought through exactly how we'll do it but JS just has a cutoff logic of 5 minutes
| # we construct the event from scratch here | ||
| # and not use the current Transaction class for easier refactoring |
There was a problem hiding this comment.
This is cool, haven't thought of completely bypassing this. I'll have to think about the implications for the granular instrumenter if we isolate OTel and our instrumentation like this.
There was a problem hiding this comment.
so for now I'm just creating a raw dict, but I'm thinking eventually of a TypedDict for Span too and the current Span/Transaction classes will basically be completely removed.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## neel/potel/initial-scope-management #3223 +/- ##
=======================================================================
- Coverage 79.27% 78.97% -0.30%
=======================================================================
Files 134 135 +1
Lines 14255 14335 +80
Branches 2990 3009 +19
=======================================================================
+ Hits 11300 11321 +21
- Misses 2107 2170 +63
+ Partials 848 844 -4
|
|
@sentrivana moved op/description/status logic to utils, but status mapping will be done separately. Maybe @szokeasaurusrex can look into making this logic as similar to JS as possible, but for now this is good enough for this PR. https://github.com/getsentry/sentry-javascript/blob/master/packages/opentelemetry/src/utils/mapStatus.ts |
sl0thentr0py
force-pushed
the
neel/potel/span-processor
branch
from
a626201
to
74f1589
Compare
sl0thentr0py
force-pushed
the
neel/potel/span-processor
branch
from
74f1589
to
2c29711
Compare
on_endinstead of bothon_start/on_endas beforespan_id -> childrenTransaction/Spanclasses at all to build the event because when we add our APIs later, we'll need to rip these out and we also avoid having to deal with theinstrumenterproblemfinished_spansevery time a root span ends and I think this uses more cpu than what I didis_sentry_spanlogic to utils