Skip to content

Improve deadline callback migration performance for large datasets#63925

Closed
ephraimbuddy wants to merge 2 commits intoapache:mainfrom
astronomer:fix-migration-0094
Closed

Improve deadline callback migration performance for large datasets#63925
ephraimbuddy wants to merge 2 commits intoapache:mainfrom
astronomer:fix-migration-0094

Conversation

@ephraimbuddy
Copy link
Contributor

Speed up migration 0094 (deadline inline callback → foreign key) for installations with millions of deadline rows. Uses a temporary batch table for deterministic batching and a minimal, migration-only serde instead of full deserialization, avoiding Task SDK and user code.

Validated with 10.7M rows migrating in ~24 minutes.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    composer 1.5 on cursor

@boring-cyborg boring-cyborg bot added area:db-migrations PRs with DB migration area:deadline-alerts AIP-86 (former AIP-57) labels Mar 19, 2026
Speed up migration 0094 (deadline inline callback → foreign key) for
installations with millions of deadline rows. Uses a temporary batch
table for deterministic batching and a minimal, migration-only serde
instead of full deserialization, avoiding Task SDK and user code.

Validated with 10.7M rows migrating in ~24 minutes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:db-migrations PRs with DB migration area:deadline-alerts AIP-86 (former AIP-57)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant