Fix cross-DAG xcom_pull with include_prior_dates=True returning None (#70173)

[Pranaykarvi]

closes #70173

What

Cross-DAG xcom_pull(..., include_prior_dates=True) returned None on Airflow 3.2.x / 3.3.0 (regression from 3.1.7). The upstream XCom is written correctly; only the cross-DAG read fails.

Root cause

XComModel.get_many derived the include_prior_dates cutoff with a correlated scalar subquery keyed on DagRun.run_id == run_id AND DagRun.dag_id == cls.dag_id. On a cross-DAG pull the request carries dag_id = <target/upstream> but run_id = <caller's own run>, so no matching DagRun exists (cls.dag_id is the upstream DAG). The subquery resolves to NULL, the <= predicate is NULL, every row is filtered out, and the SDK falls back to None. Same-DAG pulls worked because run_id and dag_id agree.

The cls.dag_id scoping was intentional — it disambiguates run_id collisions across DAGs — so the fix keeps the caller's DAG rather than dropping the scoping.

Fix

include_prior_dates semantically means "XComs on or before the pulling task instance's own run date" (matching 3.1/2.x). The caller is identified server-side by the TI token, so the Execution API resolves the cutoff from the caller's own DAG run and passes it to get_many via a new optional prior_dates_before. When it's set, the query bounds directly on it; when it's None, the original correlated-subquery path is kept as a fallback for direct model-level callers. No SDK, schema, or API-version change — include_prior_dates was already parsed on both filter params.

Behavioral note

Because the cutoff now comes from the caller's TI, a same-DAG xcom_pull(run_id=<earlier run>, include_prior_dates=True) is bounded by the caller's run date rather than that explicit run's date. This restores pre-3.2 semantics (the reference was always the pulling TI's date), but it's a subtle change in that edge combination and is called out here intentionally.

Tests

• test_xcom_get_many_from_prior_dates_cross_dag (model): cross-DAG pull returns the upstream value.
• test_xcom_get_include_prior_dates_cross_dag (route): end-to-end reproduction of xcom include_prior_dates=True returns "None" instead of xcom #70173 through get_xcom.
• Existing test_xcom_get_many_from_prior_dates and ..._scopes_run_id_to_dag unchanged and passing.

Both new tests fail before the fix (model: TypeError on prior_dates_before; route: 404/None, the exact reported symptom).

---

[Pranaykarvi]

@kaxil @amoghrajesh
Summary: cross-DAG xcom_pull(..., include_prior_dates=True) returned None because the cutoff subquery was keyed on cls.dag_id (the target DAG) paired with the caller's run_id, which never matches → NULL → all rows dropped. Fix resolves the cutoff from the pulling TI's own DAG run (via the TI token) and passes it to get_many as prior_dates_before; the old subquery stays as the fallback. No SDK/schema change.

One call-out: same-DAG xcom_pull(run_id=<earlier run>, include_prior_dates=True) is now bounded by the caller's run date rather than that run's — this restores pre-3.2 semantics, but flagging it as an intentional behavior change. Happy to add a test pinning it if wanted.

Locally: 94 passed incl. 2 new regression tests; both fail before the fix.