fix(core): guarantee task visibility extension in start_tasks() - #667
Open
jumski wants to merge 2 commits into
Open
fix(core): guarantee task visibility extension in start_tasks()#667jumski wants to merge 2 commits into
jumski wants to merge 2 commits into
Conversation
start_tasks() invoked pgflow.set_vt_batch() inside an unreferenced SELECT CTE, so PostgreSQL could skip the call and leave a claimed task with only the short initial PGMQ read visibility (#656). The final statement now consumes a guard over the renamed visibility_reset CTE: execution is structurally required, and when set_vt_batch() updates fewer messages than were claimed the statement fails ("start_tasks(): visibility updated N of M claimed messages"), rolling back the task transitions and attempt increments and returning nothing. Effective delay stays coalesce(step.opt_timeout, flow.opt_timeout) + 2, matching the #664 stalled-recovery threshold. A repeatedly visible message whose task is already started claims nothing and consumes no attempt (benign duplicate). Consolidates the three unreleased Core migrations on main (terminalize_skipped_tasks #638/#649, failed_run_terminalization #645/#663, effective_step_timeout #621/#664) plus the dev temp migration into one Atlas-generated task_lifecycle_hardening migration against the 0.15.0 baseline. Both migration-only data repairs are preserved in order: active tasks under skipped steps become skipped before remaining active tasks under failed runs become cancelled. A 0.15.0 upgrade fixture (supabase/upgrade_fixture + scripts/run-upgrade-fixture) applies the consolidated migration and asserts both repairs plus runtime visibility, wired into the core test:pgtap target. Adds pgTAP coverage for all seven issue #656 test bullets: effective-timeout visibility (shorter/longer/null fallback), rollback on total and partial visibility failure, benign duplicate batches, and mixed-batch atomicity.
🦋 Changeset detectedLatest commit: 6ab5d5b The changes in this PR will be included in the next version bump. This PR includes changesets to release 5 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
View your CI Pipeline Execution ↗ for commit 6ab5d5b
💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗ ☁️ Nx Cloud last updated this comment at |
Use clock_timestamp() in visibility-delay pgTAP assertions because set_vt_batch() uses wall-clock time while now() stays fixed at transaction start. Recheck attempts_count after a repeatedly visible started message so benign duplicates remain pinned end to end. Remove the unsupported sqruff --force option from the core fix-sql target. sqruff 0.39.0 rejects that flag, which blocked the schema-first migration workflow required for issue #656.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
start_tasks()invokedpgflow.set_vt_batch()inside an unreferencedSELECTCTE, so PostgreSQL could skip the call and leave a claimed task with only the short initial PGMQ read visibility. A message then became visible while itsstep_tasksrow stayedstarted, and workers repeatedly read it as a task that never ran.visibility_resetCTE, making the visibility update structurally required. Whenset_vt_batch()updates fewer messages than were claimed, the statement fails withstart_tasks(): visibility updated N of M claimed messages, rolling back the task transitions and attempt increments and returning nothing.coalesce(step.opt_timeout, flow.opt_timeout) + 2, matching the fix(core): use effective step timeout for stalled tasks #664 stalled-recovery threshold. A repeatedly visible message whose task is alreadystartedclaims nothing and consumes no attempt — a benign duplicate, per Add private per-step queues with typed step workers #651/Add explicit shared queues with multi-flow dispatch #652 direction.main(terminalize_skipped_tasksSkip paths archive sibling messages but never terminalize sibling step_tasks rows, leaving status='started' tasks on completed runs #638/Skip paths archive sibling messages but never terminalize sibling step_tasks rows, leaving status='started' tasks on completed runs #649,failed_run_terminalizationFailed runs leave unfinished sibling tasks queued or started instead of cancelled #645/fix(core): terminalize unfinished tasks when runs fail #663,effective_step_timeoutrequeue_stalled_tasks() uses flow timeout instead of effective step timeout #621/fix(core): use effective step timeout for stalled tasks #664) plus the development temp migration into one Atlas-generatedtask_lifecycle_hardeningmigration against the0.15.0baseline (Release 0.15.1 — task lifecycle hardening #666). Both migration-only data repairs are preserved in order: active tasks under skipped steps becomeskippedbefore remaining active tasks under failed runs becomecancelled.0.15.0upgrade fixture (supabase/upgrade_fixture/+scripts/run-upgrade-fixture): starts from the0.15.0baseline, applies the consolidated migration, and asserts both historical repairs plus final runtime visibility behavior. Wired into thecoretest:pgtapnx target so CI executes it.clock_timestamp()to matchset_vt_batch(), and repeated duplicates recheck that the attempt count stays unchanged.--forceoption so the requiredpnpm nx fix-sql coremigration step runs normally.@pgflow/corepatch changeset.Checks
pnpm nx verify-migrations core— pass (incl. schemas-synced)pnpm nx gen-types core+verify-gen-types core --skip-nx-cache— passpnpm nx fix-sql core— pass (37 files, nothing to fix)pnpm nx test:pgtap core --skip-nx-cache— pass (288 files, 1352 tests; focused start_tasks suites 26/26;upgrade fixture: PASS)pnpm nx affected --target=prepush --base=origin/main --head=HEAD— 28/28 successfulatlas migrate validate— pass;atlas.sumre-hashed after the manual repair editsCloses #656