fix(prover): do not persist aborted proving jobs so they can be re-proven after a restart#24558
Draft
AztecBot wants to merge 2 commits into
Draft
fix(prover): do not persist aborted proving jobs so they can be re-proven after a restart#24558AztecBot wants to merge 2 commits into
AztecBot wants to merge 2 commits into
Conversation
alexghr
approved these changes
Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When the prover broker cancels/aborts a proving job (
cancelProvingJob→#reportProvingJobError(..., aborted=true)), it wrote the abort to the database as a settled{ status: 'rejected', reason: 'Aborted' }result. On the next broker start,ProvingBroker.start()restores every persisted job and, if it has a result, replays that result intoresultsCacheinstead of re-enqueuing the job. So a proving job's id (a deterministic hash of its inputs) that had been aborted came back as a permanentAbortedrejection: any later enqueue of the same job hit thejobsCache.has(id)branch and returned the cached aborted status —Cached proving job … Not enqueuing again— and the facade failed the job withAborted.Net effect: once a proof was aborted, it stayed aborted across restarts, so the epoch could never be proven. A deploy restarts both the prover node and the broker, and this is what turned a planned mid-epoch redeploy into a testnet epoch prune.
Fix
An abort is a cancellation, not a terminal result, so the broker no longer persists it:
#reportProvingJobErrorskips thedatabase.setProvingJobErrorwrite whenabortedis true. The in-memory rejection is still recorded (so any current waiter is notified withAborted), but the database keeps the job as pending. Becausestart()re-enqueues every persisted job that has no stored result, the aborted job is retried after the restart instead of coming back as a poisonedAbortedrejection.This is deliberately the whole change — an in-memory "these were aborted" set would be wiped by the very restart we need to survive, so the durable fix lives entirely in what does (not) get written to the database. Genuine proof failures and timeouts are still persisted and terminal, unchanged.
Test
Adds a unit test in
proving_broker.test.ts(run against both the in-memory and persisted DB variants): enqueue a job, cancel it, restart the broker, and assert the job isin-queueagain and can be picked up and completed. It fails on the current code (the restarted broker restores the job asrejected) and passes with this change.Note: the test was not run locally in this session — executing the prover-client suite requires a full noir/wasm bootstrap that wasn't available here — so it is relied on to run in CI.