feat(sandbox): support proposer pipelining in local network#23277
Merged
Conversation
Turns on SEQ_ENABLE_PROPOSER_PIPELINING=true in every docker-compose.yml that boots `aztec start --local-network` for testing (end-to-end compose suite, docs/examples runner, playground). This routes the compose-based sandbox tests through the pipelined block production path so we catch sandbox/pipelining regressions in CI.
Under proposer pipelining the publisher for slot N is built during wall-clock slot N-1 and parks waiting for L1 to advance past the build slot before its background submission task can run `enqueueCheckpointForSubmission` (the parent-L1-sync wait inside `waitForValidParentCheckpointOnL1`). The AnvilTestWatcher had no trigger that fired in this window: the "slot filled" branch needed the not-yet-published checkpoint, the pending-tx debounce required txs in the mempool (already drained after broadcast), and the wall-clock fallback / `syncDateProviderToL1IfBehind` paths only fired *after* wall clock had crossed the next slot — i.e. the slow path itself. The result was that each L2 slot under pipelining took a full real-time slot (~72 s) before the L1 multicall fired. This change hooks the sequencer's `block-proposed` event in the local-network setup and forwards the target slot to the watcher via a new `setProposedTargetSlot`. The watcher then warps L1 (and, via `cheatcodes.warp`, the injected date provider) to that slot's timestamp ahead of the publisher's `sendRequestsAt` sleep. The multicall lands inside the target slot, the existing "slot filled" branch advances to the next slot, and the cascade keeps going at the speed of the build path. Sandbox boot with pipelining drops from ~5 minutes back to ~30 s. `block-proposed` is used rather than the cleaner `state-changed → PUBLISHING_CHECKPOINT` because the latter only fires *after* `waitForValidParentCheckpointOnL1` unblocks (which itself is what we are trying to break); see comment in local-network.ts for the full chain. The watcher only ratchets the stored slot up, and `warpToTimestamp` already rejects warps that would move L1 backwards, so duplicate or stale `block-proposed` events are no-ops. The new code path is gated on `isLocalNetwork`, so non-pipelined sandbox behavior and tests that disable the watcher are unaffected.
spalladino
added
the
ci-no-fail-fast
Routes the three aztec-up smoke tests that boot `aztec start --local-network` (basic_install, amm_flow, bridge_and_claim) through the pipelined sequencer path. These tests run on PR CI via the aztec-up/bootstrap.sh test suite, which installs the working-tree's `aztec` package via a local Verdaccio registry — so the new AnvilTestWatcher pipelining hook from the previous commit is available to them. The acceptance test under aztec-cli-acceptance-test/ is not migrated because it runs only post-release against the published nightly, which doesn't carry the watcher fix yet.
Contributor
Author
Pipelining auditVerified the sandbox tests in this PR's scope ran on CI with
Both runs are SUCCESS. All compose-routed sandbox tests under Direct config-dump evidence (
|
| Test | Ran | Pipelining=true in node config | Log |
|---|---|---|---|
aztec-up/test/amm_flow.sh |
yes (325s, PASSED) | yes | http://ci.aztec-labs.com/af2b68e7a1d884e1 |
aztec-up/test/bridge_and_claim.sh |
yes (248s, PASSED) | yes | http://ci.aztec-labs.com/418776554c4dce4c |
aztec-up/test/basic_install.sh |
yes (283s, PASSED) | n/a — script sets LOG_LEVEL=silent, but exports SEQ_ENABLE_PROPOSER_PIPELINING=true (see basic_install.sh L11-L13) |
http://ci.aztec-labs.com/47cd2a28761b8cd6 |
playground chromium |
yes (1 passed, 7.9s) | yes (line 33: "l1ChainId":31337,"enableProposerPipelining":true) |
http://ci.aztec-labs.com/928d273ed3d0849f |
playground firefox |
yes (1 passed, 21.0s) | yes (same config dump) | http://ci.aztec-labs.com/fcb032ebb35eac96 |
docs/examples runner (all 7 examples) |
yes (all PASS: aztecjs_connection, aztecjs_getting_started, aztecjs_advanced, aztecjs_authwit, aztecjs_testing, example_swap, recursive_verification) |
local-network stdout is not multiplexed into the docs-examples container log, but the compose env sets SEQ_ENABLE_PROPOSER_PIPELINING: 'true' (docs/examples/ts/docker-compose.yml L31) |
http://ci.aztec-labs.com/8b68c8bc3be31e54 |
Sample config-line for amm_flow (line 89 of its log):
INFO: ethereum:deploy_aztec_l1_contracts Deploying L1 contracts with config: {... "l1ChainId":31337,"enableProposerPipelining":true,"archiverPollingIntervalMS":500, ...}
Compose-routed e2e/cli-wallet tests (yarn-project/end-to-end/scripts/docker-compose.yml)
run_compose_test only streams the end-to-end container stdout, so the local-network node's config dump isn't surfaced in these test logs. Pipelining is set via the compose env at docker-compose.yml L31 (SEQ_ENABLE_PROPOSER_PIPELINING: 'true'); all tests below are cached green from a run on commit 323a501a (the commit that carries the compose env change + watcher fix):
| Test | Ran | Result | Log |
|---|---|---|---|
src/composed/e2e_cheat_codes.test.ts |
yes | 6/6 PASS | http://ci.aztec-labs.com/baf7e7b8f90a29f0 |
src/composed/e2e_local_network_example.test.ts |
yes | PASS | http://ci.aztec-labs.com/5e79c18b50c60b38 |
src/composed/e2e_token_bridge_tutorial_test.test.ts |
yes | 1/1 PASS | http://ci.aztec-labs.com/abaa477cdb607759 |
src/guides/up_quick_start.test.ts |
yes | 1/1 PASS | http://ci.aztec-labs.com/e1125c678f3e5b1b |
cli-wallet/test/flows/basic.sh |
yes | exit 0 | http://ci.aztec-labs.com/fba0aab5fe57d92a |
cli-wallet/test/flows/capture_authwits.sh |
yes | exit 0 | http://ci.aztec-labs.com/c12d62acfe072d8d |
cli-wallet/test/flows/create_account_pay_native.sh |
yes | exit 0 | http://ci.aztec-labs.com/5cfacc59657f277e |
cli-wallet/test/flows/no_alias.sh |
yes | exit 0 | http://ci.aztec-labs.com/ff7767636172afda |
cli-wallet/test/flows/private_transfer.sh |
yes | exit 0 | http://ci.aztec-labs.com/5ce2ec286391d4ab |
cli-wallet/test/flows/profile.sh |
yes | exit 0 | http://ci.aztec-labs.com/596834026cea9df8 |
cli-wallet/test/flows/public_authwit_transfer.sh |
yes | exit 0 | http://ci.aztec-labs.com/9c05d97fda12051c |
cli-wallet/test/flows/shield_and_transfer.sh |
yes | exit 0 | http://ci.aztec-labs.com/c59e6a2cb8003643 |
cli-wallet/test/flows/sponsored_create_account_and_mint.sh |
yes | exit 0 | http://ci.aztec-labs.com/52313069797f12c9 |
cli-wallet/test/flows/tx_management.sh |
yes | exit 0 | http://ci.aztec-labs.com/bbf5788486697094 |
Watcher fix observed firing in CI
The new AnvilTestWatcher log line Warped L1 to target slot N for pipelined publish was observed in CI for every test whose log surfaces local-network stdout. A few examples:
aztec-up amm_flow.sh(15 occurrences, slots 3,6,9,12,15,18,21,24,27,30,33,36,39,42,45): http://ci.aztec-labs.com/af2b68e7a1d884e1[16:01:14.300] INFO: aztecjs:utils:watcher Warped L1 to target slot 30 for pipelined publish
aztec-up bridge_and_claim.sh(9 occurrences): http://ci.aztec-labs.com/418776554c4dce4c[16:00:54.144] INFO: aztecjs:utils:watcher Warped L1 to target slot 21 for pipelined publish
playgroundchromium (≥5 occurrences): http://ci.aztec-labs.com/928d273ed3d0849faztec-1 | INFO: aztecjs:utils:watcher Warped L1 to target slot 2 for pipelined publish
playgroundfirefox (≥4 occurrences): http://ci.aztec-labs.com/fcb032ebb35eac96
(basic_install.sh runs at LOG_LEVEL=silent, and the e2e run_compose_test flow doesn't multiplex local-network stdout, so the line is invisible in those logs but pipelining is still ON per the compose env.)
Anything flagged
Nothing to flag.
.test_patterns.ymlhasskip: trueonly forsrc/composed/uniswap_trade_on_l1_from_l2.test.ts(already out of scope per the SANDBOX.md exclusion list — "bricked in new version of anvil").- All other compose/aztec-up entries in
.test_patterns.ymlcarryerror_regex(flaky-on-match) withoutskip, so they ran normally. - The test-engine summary confirms
No flaked tests foundon bothci/x-fastandci/x-full. - The 3 aztec-up tests we deliberately skipped from this audit (
counter_contract,default_scaffold,no_shadow_user_bins) do not startaztec start --local-network, so they have no sandbox to pipeline.
alexghr
approved these changes
May 14, 2026
This was referenced May 15, 2026
AztecBot
added a commit
that referenced
this pull request
May 15, 2026
… composes Both docs/examples/ts/docker-compose.yml and playground/docker-compose.yml ran with SEQ_ENABLE_PROPOSER_PIPELINING=true (added in #23277), but the sandbox is not yet configured to absorb pipelining's side effects: - example_swap stalls on `wait for proven block N` because the proven tip stops advancing in an idle pipelined sandbox (the original PR #23253 dequeue, http://ci.aztec-labs.com/b08ac48286302949). - aztecjs_advanced fails on `Cannot get L1 to L2 messages for checkpoint N: inbox tree in progress is N, messages not yet sealed` because under pipelining `AztecNodeService.simulatePublicCalls` reads L1->L2 messages from an in-progress checkpoint (http://ci.aztec-labs.com/419c4513023a1799). This is the same `simulator + inboxLag` mismatch already TODO'd in e2e_bot.test.ts and several e2e_fees tests. Disable the flag in the two sandbox composes to unblock the spartan merge train; aztec-up scripts (basic_install / bridge_and_claim / amm_flow) keep the flag and continue exercising pipelining in CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The local network sandbox (
aztec start --local-network) historically ran without proposer pipelining, so the compose-routed e2e suite (src/composed/*,src/guides/*, cli-wallet flows, docs examples, playground) never exercised the pipelined sequencer path. Turning pipelining on revealed that each L2 slot took a full real-time slot (~72 s) before the L1 multicall fired, blowing up sandbox boot from ~30 s to ~5 min, because the existingAnvilTestWatchertriggers don't fire in the pipelined-publish window.Approach
First commit flips
SEQ_ENABLE_PROPOSER_PIPELINING=trueon the three sandbox-test compose envs so every compose-routed test runs through the pipelined path. Second commit teachesAnvilTestWatcherabout the proposer's target slot by hooking the sequencer'sblock-proposedevent increateLocalNetwork; when the proposer has built a block destined for a slot beyond L1, the watcher warps L1 (and, viacheatcodes.warp, the injected date provider) forward, waking the pipelined publisher'ssendRequestsAtsleep and the upstreamwaitForValidParentCheckpointOnL1wait.block-proposedis used rather than the cleanerstate-changed → PUBLISHING_CHECKPOINTbecause the latter only fires afterwaitForValidParentCheckpointOnL1unblocks — which is what we are trying to break — so it would be circular.Changes
SEQ_ENABLE_PROPOSER_PIPELINING=trueto thelocal-network/aztecservice env so every compose-routed sandbox test runs pipelined.AnvilTestWatcher): newsetProposedTargetSlotsetter and awarpTimeIfNeededbranch (gated onisLocalNetwork) that warps L1 to the target slot's timestamp when it's ahead of L1.createLocalNetwork): subscribe to the sequencer'sblock-proposedevent and forwardslotto the watcher.Verified locally: sandbox boot drops from ~5 min back to ~27 s under pipelining, and
e2e_local_network_example.test.ts(both tests) passes in ~33 s.