Memory leak & CPU burn in sentry-opentelemetry integration

[volodkindv]

How do you use Sentry?

Self-hosted/on-premise

Version

2.42.1

Steps to Reproduce

Context: two services using sentry-sdk and opentelemetry. One is async, based on Faststream with Kafka. Second - backend on synchronous Django.

Both of them are degrading after several hours/minutes of work. Symptoms are slow memory leak and high CPU load.

Reason is in SentrySpanProcessor, made as singletone with decorator in init() method.
Minimal example to reproduce:

import types
import gc
import sentry_sdk
from sentry_sdk.integrations.opentelemetry.propagator import SentryPropagator
from sentry_sdk.integrations.opentelemetry.integration import OpenTelemetryIntegration
from opentelemetry import trace
import sentry_sdk.scope
from opentelemetry.trace.propagation import set_span_in_context


OpenTelemetryIntegration.setup_once()

tracer = trace.get_tracer(__name__)
propagator = SentryPropagator()


def get_global_processors_count():
    return len(sentry_sdk.scope.global_event_processors)


def emulate_sentry_propagator_inject():
    with tracer.start_as_current_span("bar") as new_span:
        context = set_span_in_context(new_span)
        propagator.inject("any_carrier", context)  # here created new global_processor


def main():
    sentry_sdk.init(dsn=None, traces_sample_rate=1.0, debug=False)

    for i in range(10):
        # for example, outgoing http request
        emulate_sentry_propagator_inject()

        gc.collect()
        funcs = sum(1 for obj in gc.get_objects() if isinstance(obj, types.FunctionType))
        cells = sum(1 for obj in gc.get_objects() if isinstance(obj, types.CellType))

        print(f"After {i}th call emulate_sentry_propagator_inject:")
        print(f"  global processors_count: {get_global_processors_count()}")
        print(f"  function objects: {funcs}, cells: {cells}\n")


if __name__ == "__main__":
    main()

Expected Result

Constant count of objects in GC and global_event_processors

Actual Result

Count of objects in GC and global_event_processors increased after every call of "propagator.inject()".

[linear-code]

PY-2417

[volodkindv]

Suggested fix:

class SentryPropagator(TextMapPropagator):

    def inject(self, carrier, context=None, setter=default_setter):
        ....
        span_id = trace.format_span_id(current_span_context.span_id)

        # span_map = SentrySpanProcessor().otel_span_map
        span_map = SentrySpanProcessor.otel_span_map  # we don't need to create new object here
        sentry_span = span_map.get(span_id, None)
        if not sentry_span:
            return

        ...

class SentrySpanProcessor(SpanProcessor):

    ...
    _initialized = False  # add this
    
    ...

    def __init__(self):
        if not self._initialized:  # register newe global_event_processor only once
            @add_global_event_processor
            def global_event_processor(event: Any, hint: Any) -> Any:
                return link_trace_context_to_error_event(event, self.otel_span_map)

            self._initialized = True