The devel-common stand-in experiment (#227): full record, both rounds

[nredd]

Permanent record of the #227 experiment, both rounds, so nothing lives only in issue-comment threads. Everything below is reproducible from the linked artifacts.

Question (#227): how much of upstream Airflow's own tests/unit suite survives when the tests_common/devel-common harness is swapped for this plugin?

Method, two rounds:

• Round 1 (2026-08-21, Breeze CI containers on a Linux VM): apache/airflow main @ c9f06bc42b (3.4.0.dev). Strategy 1 = hybrid (tests_common active, piab owns AIRFLOW_HOME + metadata DB via --without-db-init); strategy 2 = shim (airflow-core/tests/conftest.py pytest_plugins swapped to a marker-only module, every missing fixture errors and becomes the metric). Subsets: unit/models, unit/serialization, unit/timetables
• Round 2, the drift scan (2026-08-21/22, local macOS, uv venvs + PyPI wheels + upstream constraints files + source checkouts at release tags, one sub-agent per version): strategy 2 + per-version stock-harness baseline at 3.1.0, 3.1.8, 3.2.2, 3.3.1, and on the 2.x family (2.9.3, 2.11.2) where the shim replaces the monolithic tests/conftest.py instead

Artifacts:

• Shim, runner scripts, both rounds' results.csv and xz'd per-cell pytest logs: nredd/airflow-exp-227 @ exp/227-piab-devel-common (files/exp227/results/, commits 92873a2c04 + d9839511d6; standalone labeled repo, not a PR vehicle), including per-version reproduction patches -- apply drift-<version>.patch to the upstream release tag to rebuild any cell exactly
• Findings as issues: #237 missing-fixture surface, #238 dag_maker contract divergence, #239 create_dag_run kwarg double-pass bug, #240 missing backfill TableGroup bug, #241 env drift-check coexistence dx, #244 retained-AIRFLOW_HOME accumulation. Per-version drift data is commented on Upstream fixture surface: six names block 195 core-unit tests #237-clear_db and dag-fixture cleanup miss the backfill tables #240

Round-1 results (3.4.0.dev, 2131 tests across the three subsets):

cell	baseline	strategy 1	strategy 2
timetables sqlite serial	238 pass	238 pass, 2.6x faster	230 pass / 8 fail
serialization sqlite serial	502 pass / 1 fail	identical, 1.4x faster	486 pass / 16 fail / 1 err
models sqlite serial	1387 pass	identical, 1.15x faster	631 pass / 465 fail / 329 err
all3 sqlite -n auto	1773 pass / 340 fail / 37 err	ABORT (env scrub vs drift check, #241)	1255 pass / 651 fail / 258 err
all3 postgres serial	2123 pass / 1 fail	not run	not run

Round-1 headlines: strategy-1 serial is bit-for-bit identical to baseline and 10-40% faster; baseline xdist is broken on stock harness (shared sqlite, lock storms) while piab's per-worker DBs run the same cell without them; zero probe failures on uncertified 3.4.0.dev.

Drift scan, unit/models (the signal subset):

version	baseline	strategy 2	s2 pass share
2.9.3 / py3.10	1194 pass, 2 fail	751 pass, 355 fail, 92 err	63%
2.11.2 / py3.12	1282 pass, 3 fail	795 pass, 397 fail, 95 err	62%
3.1.0 / py3.10	938 pass, 1 fail	444 pass, 328 fail, 206 err	45%
3.1.8 / py3.11	968 pass, 2 fail	445 pass, 358 fail, 208 err	45%
3.2.2 / py3.12	1086 pass, 0 fail	493 pass, 393 fail, 233 err	44%
3.3.1 / py3.13	1336 pass, 0 fail	602 pass, 449 fail, 326 err	44%
3.4.0.dev (round 1)	1387 pass, 0 fail	631 pass, 465 fail, 329 err	45%

timetables strategy 2 is clean (and beats baseline) at 2.9.3/3.1.0/3.1.8; the 8 sync_dagbag_to_db fails appear at 3.2 and hold flat. serialization s2 fails are flat at 15-27 everywhere, dominated by stock-harness env expectations baked into expected values (fileloc paths, operator extra links, allowed_deserialization_classes).

Failure-class drift, s2 unit/models:

class	2.9.3	2.11.2	3.1.0	3.1.8	3.2.2	3.3.1	3.4.dev
DAG.__init__ session kwarg (#238)	136	164	121	134	160	187	199
create_dagrun kwarg double-pass (#239)	45	39	46	47	48	65	66
run-id lookups assert None is not None (#238)	50	50	49	49	49	49	49
'DAG' has no attribute 'create_dagrun' (#238)	-	-	30	30	0	0	0
sync_dagbag_to_db missing (#238)	-	-	0	0	20	20	20
infer_manual_data_interval (#238)	-	-	0	0	23	24	24
create_task_instance fixture (#237)	87	89	65	65	54	81	77
testing_dag_bundle fixture (#237)	-	-	63	63	72	81	88
backfill FK cascade (#240)	0	0	32	32	47	83	83

Conclusions:

• The compat gap is stable, not growing: 44-45% s2 pass share across the whole 3.x line into 3.4.0.dev. Upstream grows its test count; the divergence classes rotate per minor (3.1 wants DAG.create_dagrun, 3.2+ wants sync_dagbag_to_db/infer_manual_data_interval) but the proportion holds. 2.x scores better (62-63%) on its smaller fixture surface
• piab bootstrapped cleanly on all seven versions -- certified 2.x, certified 3.x, and PROBED 3.4.0.dev. Zero bootstrap failures anywhere
• Two genuine piab bugs shook out and are range-confirmed: create_dag_run double-passes session when the caller supplies one #239 (with a 2.x execution_date variant) and clear_db and dag-fixture cleanup miss the backfill tables #240 (32 -> 83 errors across certified 3.x)
• The two highest-value changes are dag_maker contract divergence from upstream tests_common #238's session-kwarg routing plus a create_task_instance-shaped helper (Upstream fixture surface: six names block 195 core-unit tests #237): together ~250-320 tests per version, on both families
• piab's xdist story (per-worker DBs) is structurally better than the stock harness's shared-sqlite serial-per-test-type model

Caveats: drift scan ran on macOS (one spawn-vs-fork hang at 2.11.2, root-caused to stock-harness env dependence); constraints files pin no test-harness deps so the pytest stack resolved unconstrained except 2.9.3 (baseline pytest 7.4.4 vs s2 8.4.2 -- piab floors at 8); a handful of local-env baseline failures (optional providers, pandas/pyarrow, time-machine 3.4) appear identically in both columns. No PRs to apache/airflow anywhere, by design.

[nredd]

Round 3 (2026-08-23, piab 0.11.0). Posted to #227; archived here for the permanent record.

Round 3: same drift scan, piab 0.11.0 (all six issue fixes from rounds 1-2 shipped). Same venvs, worktrees, and shim patches as round 2, so the plugin version is the only variable on the six release legs. The 3.4.0.dev leg moved from Breeze/Linux to a local macOS source build at the same commit c9f06bc42b (playground is gone), so its cross-round comparison is environment-confounded; its baseline still came out near-identical (1383 vs 1387 pass).

unit/models, baseline vs strategy 2, round-2 s2 alongside:

version	baseline r3	s2 r2 (0.10.0)	s2 r3 (0.11.0)	s2 pass share r2 -> r3
2.9.3 / py3.10	1194 pass, 2 fail	751p / 355f / 92e	849p / 348f / 2e	63% -> 71%
2.11.2 / py3.12	1282 pass, 3 fail	795p / 397f / 95e	876p / 409f / 3e	62% -> 68%
3.1.0 / py3.10	938 pass, 1 fail	444p / 328f / 206e	567p / 335f / 80e	45% -> 58%
3.1.8 / py3.11	968 pass, 2 fail	445p / 358f / 208e	562p / 371f / 83e	45% -> 55%
3.2.2 / py3.12	1086 pass, 0 fail	493p / 393f / 233e	630p / 387f / 108e	44% -> 56%
3.3.1 / py3.13	1336 pass, 0 fail	602p / 449f / 326e	792p / 422f / 173e	44% -> 57%
3.4.0.dev (local src)	1383 pass, 0 fail	631p / 465f / 329e (Breeze)	836p / 426f / 173e	45% -> 58%

Failure-class drift, s2 unit/models, r2 -> r3:

class	2.9.3	2.11.2	3.1.0	3.1.8	3.2.2	3.3.1	3.4.dev
DAG.__init__ session kwarg (#238)	136 -> 0	164 -> 0	121 -> 0	134 -> 0	160 -> 7	187 -> 0	199 -> 0
create_dagrun double-pass TypeError (#239)	45 -> 0	39 -> 0	46 -> 0	47 -> 0	48 -> 0	65 -> 0	66 -> 0
-> now piab's guided ValueError	54	58	44	48	56	67	68
create_task_instance/create_dummy_dag fixture (#237)	87 -> 0	89 -> 0	65 -> 0	65 -> 0	54 -> 0	81 -> 0	77 -> 0
testing_dag_bundle fixture (#237)	-	-	63 -> 0	63 -> 0	72 -> 0	81 -> 0	88 -> 0
sync_dagbag_to_db missing (#238)	-	-	0	0	20 -> 0	20 -> 0	20 -> 0
run-id lookups assert None is not None	50 -> 50	50 -> 50	49 -> 49	49 -> 49	49 -> 49	49 -> 49	49 -> 49
infer_manual_data_interval	-	-	0	0	23 -> 26	24 -> 27	24 -> 27
'DAG' has no attribute 'create_dagrun'	-	-	30 -> 37	30 -> 38	0 -> 2	0 -> 2	0 -> 2
backfill FK cascade (#240)	0	0	32 -> 59	32 -> 59	47 -> 75	83 -> 83	83 -> 85

Reading:

• 0.11.0 delivers exactly what it claims. Every targeted class is at zero across the whole matrix: session kwarg routing, both Upstream fixture surface: six names block 195 core-unit tests #237 fixture names, sync_dagbag_to_db, and the double-pass TypeError is now piab's guided ValueError naming the fix (use dag_maker.session). s2 pass share jumps ~12 points on every 3.x version and ~7 on 2.x, and the ERROR columns collapse (326 -> 173 at 3.3.1, 92 -> 2 at 2.9.3)
• The gap moved DOWN the stack, it did not vanish. Tests that died at the fixture layer now run into the next divergence, which is why failed barely moved while passed grew: the backfill FK class grew at 3.1/3.2 (more backfill tests reach teardown, Backfill FK cascade survives #240: test-created backfill_dag_run rows still break per-Dag cleanup #258), and the biggest residual is run/TI parity -- piab-created DagRuns carry no TaskInstances and run-id lookups miss, a flat ~50 + ~25 + ~100 NoneType cascade on both families (dag_maker run parity: created DagRuns carry no TaskInstances and run lookups miss #259)
• Timetables is basically solved: round-2's 8 sync_dagbag_to_db fails at 3.2+ are now 5 passes + 3 guided ValueErrors; 3.1.x and 2.9.3 stay fully green (still better than the stock harness's own workday fail, which turned out to be calendar-date-dependent and passes for everyone today)
• One genuinely new bug class: deterministic sqlite database is locked in piab teardown at 3.3.1 (44 errors). We suspected parallel-leg machine load, reran 3.3.1 s2 models on a quiet box: byte-identical failure set, 131 locked mentions both runs. Not load. Deterministic sqlite database-is-locked during fixture cleanup on Airflow 3.3+ #263
• New issues from the residuals: Backfill FK cascade survives #240: test-created backfill_dag_run rows still break per-Dag cleanup #258 (backfill FK ordering in cleanup_dag), dag_maker run parity: created DagRuns carry no TaskInstances and run lookups miss #259 (DagRun/TI parity, the big one), Upstream fixture surface, wave 2: testing_team and mock_supervisor_comms #260 (testing_team + mock_supervisor_comms, the fixture wave 2 -- testing_team doubles to 41 at 3.3+), dag_maker handle gaps, wave 2: infer_manual_data_interval, run_ti(session=), and sdk-vs-scheduler DAG API #261 (infer_manual_data_interval, run_ti(session=), sdk-vs-scheduler DAG API handles), Dag metadata already exists: dag_id reuse across tests trips the piab registry #262 (dag_id-reuse registry collision), Deterministic sqlite database-is-locked during fixture cleanup on Airflow 3.3+ #263 (locked DB)
• 0.11.0 behavior visible in the logs: retention cap works (retention policy: failed; 2 other retained roots kept everywhere, no unbounded accumulation this round -- Retained failed-run AIRFLOW_HOME roots accumulate without bound #244 confirmed), zero SmokeColocationWarning (correct: no dag_bag consumers here)

Caveats, honestly: baselines needed AIRFLOW_HOME pinned to the round-2 per-leg homes -- with it unset, concurrent legs collide on the shared ~/airflow/airflow.db (baseline numbers published here all match round 2 exactly after the reruns, except two date-dependent workday-timetable fails that legitimately pass today). 2.9.3 keeps the round-2 pytest asymmetry (baseline 7.4.4, s2 8.4.2). The dev leg ran pytest 9.1.1 both phases with a minimal local provider set. s2 wall times are not comparable across legs (up to 6 concurrent runs on one machine).

Full logs + results-round3.csv committed on the fork (results dir, logs-round3/), reproduction patches unchanged from round 2. Still no PRs anywhere. 😈

[nredd]

Wave-2 fix record (2026-08-24, post round 3, all merged). Posted to #227; archived here.

Wave 2 is done: every issue the round-3 scan produced is now fixed and merged (plus two more found while fixing). The full arc, for the record:

Round-3 residuals -> fixes, all on main today:

issue	fix	merged
#258 backfill FK survives per-Dag cleanup	widen _cleanup_dag to backfill-referencing rows	#264 687c3c4 + polish #269 820d896
#263 deterministic database is locked	roll back the borrowed session before cleanup opens its own	#265 268f91c
#261 handle gaps wave 2	dag_maker.timetable, run_ti(session=), 2.x execution_date alias; documented wontfix set + ADR 0002 amendment	#270 2965370
#262 dag_id-reuse collision	per-process persisted-id history: serial reuse replaces, cross-worker still raises	#271 9639c1e
#259 run/TI parity	two PRs: identity-map refresh fix #272 e84212b, then upstream run defaults #274 ff4c72e + ADR 0003
#266 dag_version FK (found during #262)	dag_id-scoped run sweep before the version delete	#273 11631ba
#275 flaky backfill regression test (found by the merge train itself)	scope assertions to the test's own rows	#276 4474583
#260 fixture wave 2 (testing_team, mock_supervisor_comms)	SHELVED -- pure upstream parity, no end-user story, surface still moving upstream; plan archived on the issue

Direction / user-facing contract changes that came out of this wave:

• ADR 0003 supersedes 0002's run-id clause: bare create_dagrun() now defaults run_id="test", logical_date to the Dag's start_date, and dag_maker injects upstream's start_date ladder (kwarg > default_args > module DEFAULT_DATE > 2016 epoch, explicit None = opt-out). The hashed run-id "xdist mitigation" was hollow -- collision safety always came from the derived dag_id, and the ADR records that
• Re-registering a dag_id piab itself persisted earlier in the same process now REPLACES the leftover metadata instead of raising; cross-worker collisions still raise and xdist_group stays the documented mitigation
• dag_maker(session=...) teardown now rolls the borrowed session back before cleanup -- the Deterministic sqlite database-is-locked during fixture cleanup on Airflow 3.3+ #263 lock was a teardown-order deadlock with the caller's own uncommitted flush, proven by a 31s-vs-0.01s repro
• Per-Dag cleanup is now scoped by Dag identity, not fixture ownership: runs Airflow created behind piab's back (dag.test(), backfill machinery) are swept FK-safely
• New handles: dag_maker.timetable (kills the infer_manual_data_interval class), run_ti(session=), create_task_instance(execution_date=) with upstream's deprecation semantics. DAG.clear/partial_subset/_time_restriction-on-the-yield are documented wontfix with the serialized_dag escape hatch

Bonus find: the #274 CI exposed an upstream Airflow 3.2.0-only regression -- TaskInstance.get_task_instance dropped dag_id from its filter_by (present in 3.1.x, 3.2.1+, absent only in 3.2.0), which piab's old hashed run ids had masked and the new shared run_id="test" default surfaced as MultipleResultsFound. piab now queries the TI by full identity on every release, with a red-then-green repro on a purpose-built 3.2.0/py3.13 env.

Process notes, FWIW: all six fixes ran as parallel plan-first agents (plans human-approved before implementation), shipped through ship-issue + code-review, and merged via a serialized merge train under the strict-checks ruleset -- one live CI run at a time, stale runs canceled, flakes rerun-failed, conflicts rebased just-in-time. The train's procedure is now a skill in this repo (.claude/skills/get-it-merged, #268 2a3109524). Adversarial review earned its keep twice: it caught the cross-worker purge hole in #262's replace rule and the expire_all hole in #272's refresh pin before either shipped.

Data: round-3 results + logs remain at nredd/airflow-exp-227@2330c1da59 (results-round3.csv, logs-round3/, reproduction patches). The natural next step is a round-4 rerun of the drift scan against the next release to measure how many of the 44-45%->55-58% residual actually flipped -- the #259/#261/#262 fixes target the largest remaining classes (~300+/version). 😈