Retry transient read failures while polling the factory wrapper - #1133
Merged
Conversation
added 3 commits
September 11, 2026 17:34
The loop reads the wrapper's output through one sandbox exec call with a 25 second timeout. Under a saturated container one poll exceeded it, the error propagated, the loop parked the item and killed a run that had passed Verify and Review after 3 hours 48 minutes (QM-1 attempt 7). runFactoryProcess now retries up to six consecutive read failures with a two second pause. A vanished process session is still fatal at once.
The leak-check test scripted one throwing read, so with retries it slept 24 seconds and rejected with the fake's own guard error instead of the scripted one. It now scripts a persistent failure with no pause. A new test pins that a successful read resets the streak.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
runFactoryProcesspolls the wrapper throughsandbox.readProcess, which the exec-based process session implements as oneruncall with awaitMs + 20stimeout. On the local backend that is anAbortSignal.timeouton a single HTTP call. During QM-1 attempt 7 the container was saturated by the full test suite in the second Review pass, one poll exceeded the timeout, the error propagated out of the work stage, the loop parked the item withThe operation was aborted due to timeout, and sent TERM to a wrapper that had already passed Verify and Review after 3 hours 48 minutes.What
FACTORY_READ_RETRIES(6) consecutive read failures with aFACTORY_READ_RETRY_MS(2 s) pause. A success resets the count.no such process session) is still fatal on the first read, through the existingprocessIsGonepredicate.readRetryMsjoinsreadWaitMsandtermGraceMsas a test-time override.Tests
FACTORY_READ_RETRIES + 1throws withreadRetryMs: 0, and pin the same TERM-once and teardown-once behavior.FACTORY_READ_RETRIES = 0the transient-failure test fails; reverted.node --teston the process-work and effects files: 77 pass.tsc, eslint, prettier green.