Skip to content

Fix DagRun._emit_dagrun_span crash on None/empty context_carrier#64087

Merged
vatsrahul1001 merged 1 commit intoapache:mainfrom
yulit0738:fix/dagrun-none-context-carrier
Mar 23, 2026
Merged

Fix DagRun._emit_dagrun_span crash on None/empty context_carrier#64087
vatsrahul1001 merged 1 commit intoapache:mainfrom
yulit0738:fix/dagrun-none-context-carrier

Conversation

@yulit0738
Copy link
Contributor

Summary

Fixes a crash in DagRun._emit_dagrun_span() when context_carrier is None or empty {}.

  • Symptom: AttributeError: 'NoneType' object has no attribute 'get' in TraceContextTextMapPropagator().extract()
  • Cause: DagRuns created before OTel tracing was enabled have context_carrier = NULL in the database. When these DagRuns complete, _emit_dagrun_span() passes None directly to extract().
  • Fix: Early return when context_carrier is falsy, consistent with the existing guard in task_runner.py:148.

Reproduction Path

  1. Have existing DagRuns in the database (created before OTel was enabled → context_carrier = NULL)
  2. Enable OTel tracing
  3. Those DagRuns complete → update_state()_emit_dagrun_span() → crash

Note: This supersedes #61655 which targeted OtelTrace.extract() — that class was since removed in #63452.

Testing

  • Added parametrized unit test covering both None and {} carrier values
  • Verified no span is emitted when carrier is missing (graceful skip)

Was generative AI tooling used to co-author this PR?

  • Yes (please specify the tool below)
    Generated-by: Claude (Anthropic) following the guidelines

@uranusjr
Copy link
Member

TraceContextTextMapPropagator().extract() already has logic to deal with None and empty dict. I am under the impression these are expected values that should go through the same code path, not silently excluded early.

@yulit0738
Copy link
Contributor Author

TraceContextTextMapPropagator().extract() already has logic to deal with None and empty dict. I am under the impression these are expected values that should go through the same code path, not silently excluded early.

@uranusjr verified against the latest opentelemetry-python main — TraceContextTextMapPropagator.extract() has no None guard for carrier. DefaultGetter.get() calls
carrier.get(key, None) directly, which raises AttributeError on None.

The context_carrier column is nullable=True. After upgrading Airflow to a version with OTel tracing, existing DagRuns in the DB have context_carrier = NULL. When those DagRuns complete,
the scheduler crashes in _emit_dagrun_span() — not a task failure, but a scheduler process crash.

task_runner.py:148 already uses the same pattern: if msg.ti.context_carrier else None.

I can narrow the guard to only None (not empty dict) if you prefer — {} does pass through extract() safely.

@uranusjr
Copy link
Member

Let’s pass self.context_carrier or {} to OTel then, that’s probably more idiomatic.

DagRuns created before OTel tracing was enabled have context_carrier=NULL
in the database. When these DagRuns complete, _emit_dagrun_span() passes
None to TraceContextTextMapPropagator().extract(), which crashes with
AttributeError: 'NoneType' object has no attribute 'get'.

Use `self.context_carrier or {}` so OTel receives a valid carrier and
emits a root span per the OTel spec, consistent with task_runner.py:148.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@yulit0738 yulit0738 force-pushed the fix/dagrun-none-context-carrier branch from bb9dd0f to 2ae69e0 Compare March 23, 2026 06:57
@yulit0738
Copy link
Contributor Author

@uranusjr
Updated — now using self.context_carrier or {} instead of the early return, so OTel receives a valid carrier and emits a root span per spec.

You were right that this should go through the same code path. The only issue was None specifically — DefaultGetter.get() calls carrier.get(key, None) which raises AttributeError on None. The or {} lets OTel handle it naturally.

@eladkal eladkal modified the milestones: Airflow 3.1.9, Airflow 3.2.0 Mar 23, 2026
@eladkal eladkal added the type:bug-fix Changelog: Bug Fixes label Mar 23, 2026
@vatsrahul1001 vatsrahul1001 merged commit 4eabf79 into apache:main Mar 23, 2026
81 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:bug-fix Changelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants