Skip to content

Refactor 0102_3_2_0_make_external_executor_id_text migration to support batch processing downgrade#63910

Open
jason810496 wants to merge 3 commits intoapache:mainfrom
jason810496:perf/migration/make-external-executor-id-text-batch
Open

Refactor 0102_3_2_0_make_external_executor_id_text migration to support batch processing downgrade#63910
jason810496 wants to merge 3 commits intoapache:mainfrom
jason810496:perf/migration/make-external-executor-id-text-batch

Conversation

@jason810496
Copy link
Member

Why

As pointed out in #63625 (review), the original migration (and also the PR [migration-0102] Change logic migration data of postgresql) will hold the ACCESS EXCLUSIVE lock that blocks ALL the concurrent READ and WRITE, For TaskInstance and TaskInstanceHistory tables at production scale, even 4 minutes of total blocking is unacceptable. (e.g. downgrade from 3.2.0 to 3.1.8 due to other feature's error but encounter 8mins downtime)

What

Instead we:

  1. ADD external_executor_id_new COLUMN (instant, ~ms level, brief ACCESS EXCLUSIVE)
  2. Backfill in batches
    • it might take longer than a naive ALTER TABLE task_instance ALTER COLUMN ... TYPE VARCHAR(250) DDL
    • but it's only a ROW EXCLUSIVE lock
    • so with this approach, it still allow all concurrent reads/writes for the rows that are not in the current batch.
  3. DROP external_executor_id + RENAME external_executor_id_new to external_executor_id in same TX (instant, ~ms level, brief ACCESS EXCLUSIVE)

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below) Discussing the locking behavior with Claude, Gemini

Copy link
Member Author

@jason810496 jason810496 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There're some SQL syntax level failure, I will fix the CI later on.

@uranusjr
Copy link
Member

I think this is on the right track (using batch and engine.begin())

@jason810496 jason810496 self-assigned this Mar 20, 2026
@jason810496 jason810496 force-pushed the perf/migration/make-external-executor-id-text-batch branch from a99c0fe to 1884e55 Compare March 20, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:db-migrations PRs with DB migration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migration 0102 downgrade holds exclusive lock on task_instance for extended period during TEXT→VARCHAR(250) rewrite

2 participants