Instrument Django app with Sentry and OTel and check differences.

[antonpirker]

Instrument a Django application once with Sentry and once with Sentry using OpenTelementry for span creation. We want to compare the span waterfalls, to see the differences and what needs to be done to have feature parity.

Test setup:

Application: https://github.com/getsentry/demo-movie-search/tree/main/backend/django
A request to the /api/shows/{pk}/ API endpoint will do:

• Run a Django view with multiple middleware (and triggering Django signals)
• Run multiple database queries (including an N+1 query) to Postgres
• Start a Celery task in the background
• Run multiple queries to a Redis server

This resembles a common setup in production Django applications, that is used all over the planet.

Scenario 1) Sentry:

sentry_sdk.init(
    dsn=sentry_dsn,
    release=sentry_release,
    environment=sentry_environment,
    traces_sample_rate=sentry_traces_sample_rate,
    send_default_pii=sentry_default_pii,
    debug=sentry_debug,
)

This will load Sentries integrations (like Django, Celery, ...) by default and those Sentry integrations will create spans. Opentelementry is not used.

Scenario 2) Sentry using OpenTelementry:

sentry_sdk.init(
    dsn=sentry_dsn,
    release=sentry_release,
    environment=sentry_environment,
    traces_sample_rate=sentry_traces_sample_rate,
    send_default_pii=sentry_default_pii,
    debug=sentry_debug,
    _experiments={
        "otel_powered_performance": True
    },
)

This will disable all Sentry span creation and spans will only be created by the OpenTelemetry SDK.

[antonpirker]

Example traces for both scenarios:

1. Sentry:
   https://sentry-sdks.sentry.io/performance/trace/060236bb8c7c4964b5dc902457ed6878/?node=txn-915f3493bb3e4f66864523ca78ce21b6&project=5461230&query=http.method%3AGET&referrer=performance-transaction-summary&showTransactions=recent&source=performance_transaction_summary&statsPeriod=1h&timestamp=1719405647&transaction=%2Fapi%2Fshows%2F%7Bpk%7D%2F&unselectedSeries=p100%28%29&unselectedSeries=avg%28%29

2. Sentry using Otel:
   https://sentry-sdks.sentry.io/performance/trace/13998512b73fbf891922ecfdda42d544/?node=txn-b9a55d514601420aa11b1e9ddcb44c7e&project=5461230&query=http.method%3AGET&referrer=performance-transaction-summary&showTransactions=recent&source=performance_transaction_summary&statsPeriod=1h&timestamp=1719499933&transaction=GET+api%2Fshows%2F%28%3FP%3Cpk%3E%5B%5E%2F.%5D%2B%29%2F%24&unselectedSeries=p100%28%29&unselectedSeries=avg%28%29

[antonpirker]

Scenario 1) Sentry

Trace in Sentry.io

[antonpirker]

Scenario 2) Sentry using Opentelementry

Trace in Sentry.io

[antonpirker]

Things that I noticed:

• Otel does not create Spans for middleware, view rendering, or Django signals (that is the Missing Instrumentation block in Otel trace)
• One can see trigger_notifications (the Celery task) in otel spans (which is cool), but there is not much in it.

[antonpirker]

One thing of note: Database spans in OTel include a special structured comment that includes tracing and other information:

[antonpirker]

OTel database spans also include information about the HTTP server they where running in: