Purge partitioned_asset_key_log and restore the db clean coverage check - #70924
Closed
steveahnahn wants to merge 2 commits into
Closed
Purge partitioned_asset_key_log and restore the db clean coverage check#70924steveahnahn wants to merge 2 commits into
steveahnahn wants to merge 2 commits into
Conversation
steveahnahn
force-pushed
the
fix-db-clean-coverage-guard
branch
from
August 1, 2026 18:26
7ef0dd3 to
53b147c
Compare
1 task
The table was renamed from callback_request in Airflow 3.2.0 and the cleanup configuration was not updated with it. A configured table that does not exist is skipped with only a warning, so these rows were never deleted and the table grew without bound, with no way for an operator to purge it. Only callbacks that can no longer run are removed. A callback still awaiting execution owns its deadline row through an ON DELETE CASCADE foreign key, so deleting one would silently drop a deadline that has not fired yet. The sla_miss entry is dropped as well, since that table no longer exists in Airflow 3.
The check that is meant to catch a metadata table being left out of airflow db clean looked for the models package at a path relative to the test file. That path stopped resolving when the sources moved under airflow-core, so the check examined no models at all and its assertions held trivially for an empty set. It has not been able to report a missing table since, which is how the callback table went unpurged for several releases. Walking the package's own search path instead keeps the check working wherever the sources live, and asserting that models were found stops it from passing on an empty set again. With the check restored, partitioned_asset_key_log is the one table it reports that genuinely has no way to be purged: it carries no foreign key, so its rows are left behind when the partition Dag run they describe is cascade-deleted with its dag_run. The rest hang off a cleaned table by an ON DELETE CASCADE foreign key and are recorded as such.
steveahnahn
force-pushed
the
fix-db-clean-coverage-guard
branch
from
August 1, 2026 19:11
53b147c to
c6ce175
Compare
Contributor
Author
|
Folded into #70923 — the restored coverage check cannot pass without the callback fix there, and the combined change is small enough to review as one. Drafted-by: Claude Code (Fable 5) (no human review before posting) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Builds on #70923 and contains its commit, so the diff here will shrink to a single commit once that merges. The coverage check restored below cannot pass without it: with the check working,
callbackis reported as uncovered until #70923 registers it.test_no_models_missingexists to catch a metadata table being left out ofairflow db clean. It has not been able to do that for some time.It located the models package by walking up from the test file,
Path(__file__).parents[2] / "airflow/models". That resolves toairflow-core/tests/airflow/models, which stopped existing when the sources moved underairflow-core/src.pkgutil.iter_moduleson a missing directory yields nothing, soall_modelswas always empty and both assertions held trivially:Confirmed against the code prior to #70923: the check passed while
callbackwas in neither the cleanup config nor the exclusion list. That is how the table went unpurged for several releases.What this changes
airflow.models.__path__instead of reconstructing the path from the test's location, so the check keeps working wherever the sources live. It now discovers 52 models, against 0 before.partitioned_asset_key_logfor cleanup. It is the one table the restored check reports that genuinely has no way to be purged.Why partitioned_asset_key_log needs purging
It carries no foreign key at all.
asset_partition_dag_runis removed byON DELETE CASCADEwhen itsdag_runis cleaned, but the key-log rows describing that partition run are simply left behind. The only existing delete is inSchedulerJobRunner._create_dagruns_for_partitioned_asset_dags, which removes rows for stale partition runs when a rollup definition changes. That is not a retention mechanism, so the table grows without bound.The other tables, and why they are safe
Delete rules read from the live schema after
airflow db migrate, rather than from the ORM metadata, since migrations are what the deployed database actually has:asset_partition_dag_runON DELETE CASCADEfromdag_runasset_watcherON DELETE CASCADEfromtriggerdag_favoriteON DELETE CASCADEfromdaghitl_detailON DELETE CASCADEfromtask_instancehitl_detail_historyON DELETE CASCADEfromtask_instance_historytask_inlet_asset_referenceON DELETE CASCADEfromdagdeadline_alertON DELETE CASCADEfromserialized_dag, which followsdagasset_state_storeON DELETE CASCADEfromasset, itself deliberately left aloneteamVerification
callbackfrom the cleanup config makes the check flag it, so it now catches the defect that motivated Fix airflow db clean never purging the callback table #70923.partitioned_asset_key_logcleanup verified against a real database: an old row is purged and a recent one kept.test_db_cleanup.pysuite passes.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines