fix(e2e): resolve flaky proposer-not-found in duplicate_attestation test#24548
Closed
AztecBot wants to merge 296 commits into
Closed
fix(e2e): resolve flaky proposer-not-found in duplicate_attestation test#24548AztecBot wants to merge 296 commits into
AztecBot wants to merge 296 commits into
Conversation
Disables the testnet transaction bot until the v5 release. `spartan/environments/testnet.env` set `BOT_TRANSFERS_REPLICAS=0` (was `1`). `BOT_SWAPS_REPLICAS` was already `0`, and `BOT_CROSS_CHAIN_REPLICAS` is unset (defaults to `0` in `variables.tf`), so this takes testnet to 0 bots overall. Requested in #alerts-testnet — the bot was manually scaled to 0; this makes that the deployed default until v5 ships. --- *Created by [claudebox](https://claudebox.work/v2/sessions/fc2e7cd0077cd038) · group: `slackbot`*
…23884) ## What Adds a tiny workflow that, on every push to `next`/`v5-next`, dispatches the private mirror's (`AztecProtocol/aztec-packages-private`) `sync-upstream-<branch>.yml` via AztecBot — so the mirror is **notified** of public changes instead of **polling** every 15 minutes (companion PR there drops the cron). ## Safety This workflow **only dispatches** a workflow in the private repo. It does **not** checkout, read, or push any private content — there is no path for private commits to enter this public repo. Auth is the existing `AZTEC_BOT_GITHUB_TOKEN` secret. Minimal: one new file, one `gh workflow run` call.
Main changes: - Bind prover-supplied IPA `G_0` values in `IPA::batch_reduce_verify` with an independent batched SRS MSM check. - Harden `ChonkBatchVerifier` lifecycle, queue validation, duplicate request IDs, queue bounds, invalid VK/proof-size handling, callback exceptions, and shutdown cleanup. - Harden bbapi batch verifier FIFO result delivery and startup cleanup, including FIFO path validation, terminal FIFO failure handling, and pending request failure results for malformed proof fields. - Make `BatchChonkVerifier` reject pending TS promises on FIFO/decode failures, wait briefly for native stop to flush FIFO result frames, and clean up temp FIFO directories reliably. - Make Chonk verifier API wrong-VK byte lengths return `valid=false` before deserialization, including in wasm builds where catch blocks are compiled away. - Keep the pinned bb.js Chonk test from re-reading bounded wasm inputs after the native backend case has run.
Main changes: - Bind prover-supplied IPA `G_0` values in `IPA::batch_reduce_verify` with an independent batched SRS MSM check. - Harden `ChonkBatchVerifier` lifecycle, queue validation, duplicate request IDs, queue bounds, invalid VK/proof-size handling, callback exceptions, and shutdown cleanup. - Harden bbapi batch verifier FIFO result delivery and startup cleanup, including FIFO path validation, terminal FIFO failure handling, and pending request failure results for malformed proof fields. - Make `BatchChonkVerifier` reject pending TS promises on FIFO/decode failures, wait briefly for native stop to flush FIFO result frames, and clean up temp FIFO directories reliably. - Make Chonk verifier API wrong-VK byte lengths return `valid=false` before deserialization, including in wasm builds where catch blocks are compiled away. - Keep the pinned bb.js Chonk test from re-reading bounded wasm inputs after the native backend case has run.
…th Constantine recoder (#23691) Isolates the scalar-multiplication-**independent** ECC changes from the Pippenger refactor (#23297) into their own reviewable layer, stacked on top of the Constantine recoder PR (#23562). ## Stack `merge-train/barretenberg` ← #23562 (Constantine recoder) ← **this PR** ← #23297 (Pippenger) This PR depends on #23562 because it consumes the shared signed-Booth primitives in `ecc/groups/booth_recode.hpp`. It must merge after #23562. ## What's here - **element**: add `straus_msm`; rewrite `batch_mul_with_endomorphism` / `operator*` onto carry-less signed-Booth window slices, replacing the old `EndomorphismWnaf` lookup-table path. - **field**: short-circuit `split_into_endomorphism_scalars` for small `k`. - **wnaf**: drop the runtime `fixed_wnaf` overload — orphaned once `EndomorphismWnaf` (its only caller) was removed. - **dedup vs booth_recode.hpp**: element_impl no longer carries its own copy of `BoothSliceParams` / `compute_booth_slice_params`; the generic branchless `booth_packed_digit` reader now lives in `booth_recode.hpp` as a shared primitive. element_impl keeps only its element-specific GLV-endo window schedule + EC math. The Pippenger MSM path keeps its own perf-tuned reader. - **tests**: `straus_msm` correctness/edge cases and `batch_mul` K2 bit-width coverage. ## Testing `ecc_tests` builds and `StrausMsm*` + `BatchMul*` pass on the endomorphism curves (bn254, grumpkin); secp curves skip the endo tests by design.
Ports [AztecProtocol/barretenberg-claude#3643](AztecProtocol/barretenberg-claude#3643). --------- Co-authored-by: sergei iakovenko <105737703+iakovenkos@users.noreply.github.com> Co-authored-by: iakovenkos <sergey.s.yakovenko@gmail.com>
Barretenberg side of noir-lang/noir#12554 * adds support for the `msgpack-tagged` format * allows all `msgpack*` formats Tested similarly to #12841 with a `nargo` compiled from noir-lang/noir#12652 The bytecode size gains were measured in noir-lang/noir#12635
Updates the developer and operator docs from v4.3.0 to v4.3.1 (bug-fix release, both mainnet and testnet). There are no docs content changes between the two tags, so instead of a full re-cut the versioned snapshots are renamed and macro-resolved version strings bumped in place **Content additions:** - New v4.3.1 section in the operator changelog (`docs-operate/.../changelog/v4.3.md`, mirrored into the snapshot) - Added the v4.3.x entry to the operator changelog index, which was missing on `next` (it only existed on the release branch) - Replaced a hardcoded `v4.3.0` in `testing_contracts.md` with the `#include_aztec_version` macro so it tracks releases automatically
Updates the developer and operator docs from v4.3.0 to v4.3.1 (bug-fix release, both mainnet and testnet). There are no docs content changes between the two tags, so instead of a full re-cut the versioned snapshots are renamed and macro-resolved version strings bumped in place **Content additions:** - New v4.3.1 section in the operator changelog (`docs-operate/.../changelog/v4.3.md`, mirrored into the snapshot) - Added the v4.3.x entry to the operator changelog index, which was missing on `next` (it only existed on the release branch) - Replaced a hardcoded `v4.3.0` in `testing_contracts.md` with the `#include_aztec_version` macro so it tracks releases automatically
…ucture llms.txt (#23567) ## Why An agent pointed at docs.aztec.network hit 404s, gave up, fell back to stale training data, and missed `SingleUseClaim` (the right primitive). Causes: pages deleted without redirects, and `llms.txt` truncating its API index at 100 of ~1,300 entries (alphabetical, hiding everything past `m`). Three fixes + a CI guard so the deletion case can't recur silently. ## What changed **1. Orphan-URL detection** — `docs/scripts/check_orphaned_urls.sh` (new) - Flags any baseline URL that no longer resolves **and** has no `netlify.toml` redirect (inverse of `validate_redirect_targets.sh`). - Handles `*` splat / `:name` placeholders. `FAIL_ON_ORPHAN=1` = hard gate (default warn-only). **2. Restored redirects** — `docs/netlify.toml` - 301s for 5 previously-404 pages (4 deleted in #16788, + bare `/developers/getting_started`), e.g. `private_voting_contract` → `state_variables` (where `SingleUseClaim` lives). **3. llms.txt restructure** — `docs/scripts/append_api_docs_to_llms.js` - Spec-compliant root: single H1 + `>` summary, `llms-full.txt` link up top, droppable `## Optional` section ([awesome-aztec](https://github.com/AztecProtocol/awesome-aztec)). - Hub-and-spoke (Cloudflare/Next/Svelte pattern): root `## API Reference` is now 2 links, not a ~286-module dump (root API content 84 KB → <1 KB). - Full per-symbol index moves to scoped files an on-task agent fetches: `aztec-nr-api/mainnet/llms.txt` (~286 modules, structs/traits/fns inline — `SingleUseClaim` stays grep-able) and `typescript-api/mainnet/llms.txt`. Each is its own single-H1 doc. `llms-full.txt` unchanged. - Fail-loud sentinel: build aborts if the rustdoc index collapses below floors (≥50 modules, ≥100 item names; currently 286/662), so a future nargo HTML change can't silently gut the symbol index. - Build: bare `docusaurus` → `yarn docusaurus` in `package.json` + `clean.sh` (use repo-pinned binary, not global PATH). **4. CI guard** — `docs/bootstrap.sh` + `docs/snapshots/published-urls.txt` - `check_orphaned_urls` runs on every docs PR with `FAIL_ON_ORPHAN=1`. Catches accidental deletion-without-redirect (the incident class above). - Baseline is read from the **base branch** (`git show origin/$GITHUB_BASE_REF:…`), not the working tree, so a PR can't weaken its own gate by editing the snapshot. (This PR introduces the file, so it falls back to the working-tree snapshot for itself; tamper-resistance applies to every PR after merge.) - `refresh_url_snapshot.sh` only grows the protected set (registering newly-published pages); retiring a page needs just the redirect, not a snapshot edit. ## Test plan - [x] `validate:redirects` + `validate:api-ref-links` clean - [x] `check_orphaned_urls.sh`: 162 resolved, 0 orphaned; flags synthetic deletions - [x] Full `yarn build`: root single-H1 + `llms-full.txt` link + `## API Reference` (2 dynamic-version links) + `## Optional`; scoped indexes single-H1, `SingleUseClaim` present, 9 TS package links - [x] No-API build: only `## Optional`, single H1, no scoped files - [x] Sentinel fires (exit 1) when the rustdoc index is forced below the floor - [x] Base-branch baseline sourcing verified (falls back to working-tree snapshot for this introducing PR, as designed) - [x] `refresh_url_snapshot.sh` idempotent; Codex reviewed - [x] [Netlify preview](https://deploy-preview-23567--aztec-docs-dev.netlify.app): all 5 redirects 301 to targets that resolve 200; `/llms.txt`, `/llms-full.txt`, `/aztec-nr-api/mainnet/llms.txt`, `/typescript-api/mainnet/llms.txt` all serve 200; scoped module links resolve Refs: #16788, #17168, #23572 (scoped-index split, merged here).
## Summary Adds an auto-generator for `docs/docs-operate/operators/reference/cli-reference.md`. The file was previously hand-maintained and silently drifted between v4.2.x and v4.3.0 (`--pxe` flag removed, `--local-network.testAccounts` added, ~450 lines of help-text drift across other flags). The developer-facing CLI references (`aztec`, `aztec-wallet`, `aztec-up`) already auto-generate via `scan_cli.py` + `cli_docs_config.json`, but `aztec start` is a single flag-list dump rather than a recursive command tree, so a simpler dedicated script makes more sense than retrofitting the existing framework. ## What changed - **New script:** `scripts/cli_reference_generation/generate_operator_cli_ref.sh` runs `aztec start --help` and concatenates with the hand-curated preamble. Retries on truncated output (the dockerized CLI sometimes drops trailing stdout when captured via `$(...)` subshell — fixed by writing to a temp file). - **New preamble file:** `scripts/cli_reference_generation/operator_cli_preamble.md` holds the hand-curated frontmatter + intro. - **New yarn script:** `generate:operator-cli-reference` in `docs/package.json`. - **Batch integration:** `generate_all_cli_docs.sh` now also generates the operator ref (skipped when `OUTPUT_DIR` is set, since that's the per-version flow which doesn't touch the operator doc). - **README updated** with a note explaining the side-path. - **Regenerated** `docs-operate/operators/reference/cli-reference.md` via the new pipeline (982 lines). The diff vs `next` reflects what should have been a v4.3.0 hand-update but never happened. ## Why this isn't part of `cli_docs_config.json` The existing `scan_cli.py` infrastructure walks subcommand trees and parses `Commands:` / `Subcommands:` sections. `aztec start` has no subcommands — it's one command with ~950 flag descriptions. Folding it into the existing config would require either special-casing inside `scan_cli.py` or duplicating the preamble handling. A 70-line standalone script is simpler. ## Out of scope `scan_cli.py` exhibits the same dockerized-CLI stdout-truncation bug (visible in the dev `aztec_cli_reference.md` line 1937 where "Default" is truncated to "Defau"). Worth fixing in a follow-up — same root cause, different code path. ## Test plan - [x] `yarn generate:operator-cli-reference` produces 982-line output ending cleanly with the TXE section - [x] `./scripts/cli_reference_generation/generate_all_cli_docs.sh --force current` triggers the new step at the end of the batch - [x] Retry logic activates and recovers on flaky CLI runs - [ ] CI build runs cleanly
… add TXE tests (#23517) ## Summary Align the counter contract tutorial's example with the canonical `aztec init` counter template, add TXE tests, and run them in docs CI. - **Naming aligned with the template.** The docs example's initializer is renamed from `initialize`/`headstart` to `constructor`/`initial_value` to match the counter scaffold that `aztec init` generates (`yarn-project/aztec/scripts/templates/counter/`). The `increment(owner)` and `get_counter(owner)` signatures already match the template and are unchanged. This PR does not modify the CLI template itself; the docs example follows it. - **TXE tests.** Adds a `counter_contract_test` package (mirroring `logging_example_test`) with tests for the constructor and increment, matching the template's test layout. - **CI.** A new `test-contracts` step in `docs/examples/bootstrap.sh` starts a TXE and runs `nargo test` for both `counter_contract_test` and `logging_example_test`, so these tests run in docs CI rather than sitting unexecuted. - **Tutorial polish.** Sentence-case headings, a more accurate privacy description (the count is confidential; dropped the overstated "no one knows when you increment"), and prose updated for the `constructor` naming. ## What's not in scope - No change to the `aztec init` counter template; the docs example is aligned to it. - No change to the example's authorization model: `increment(owner)` matches the template. ## Test plan - CI's `docs/examples/bootstrap.sh compile` builds `counter_contract` against the `constructor` signature. - The new `docs/examples/bootstrap.sh test-contracts` step runs `nargo test` for `counter_contract_test` and `logging_example_test` against a TXE. This could not be run locally (stale in-repo nargo, and the step needs a built `yarn-project/txe`), so the first real validation is this PR's CI run. - Docusaurus dev build renders the updated tutorial without `#include_code` errors.
…#23887) Stacks on #23297 (base = `lde/zacs-pippenger`). Adds edge-case unit tests, fuzzers, and one out-of-bounds fix for the new round-parallel Pippenger / MSM, focused on the areas the refactor is most fragile in: **arena sizing** and **parameter-dependent dispatch**. Test-only except for a one-line OOB fix in an existing test. Validated natively **and under AddressSanitizer + `_GLIBCXX_DEBUG`**. ## Bug found & fixed **Out-of-bounds scalar read in `test_batch_multi_scalar_mul_large_dense`** (existing in #23297). It indexed `scalars[k*8192 + i]` up to 32767 against the 31013-element fixture. Release builds did silent UB (both result and reference read the same out-of-bounds garbage, so the assertion still passed — the bug was invisible); the debug/ASAN allocator aborts. Wrapped with `% num_points`, matching the sibling ragged test. ## Tests added **Arena (the documented "drift" bug class — sizer vs live allocator):** - `ArenaLayoutFitsAcrossDispatchSpace` — sweeps ~14k combos of thread count (incl. odd/non-pow2), N at every dispatch boundary, **`dedup=true` and `ext_glv=true`** (neither was ever checked against the sizer), and bit budgets, plus 1600 deterministic pseudo-random N. Any miss is a guaranteed Zone overflow. - `EffectiveNumBitsBandSmallScalars` — drives real MSMs in the native non-GLV mid-band (2^13<n<2^17) where the sizer's defensive bit-budget sweep is skipped, so an under-count surfaces as an OOB write rather than passing silently. **Dispatch / GLV / dedup:** - `DispatchFuzz` — randomized differential fuzzer over (N, start_index, thread count, scalar distribution, dedup) vs naive. - `GlvExtremeMagnitudeScalars` — magnitude-maximizing scalars (r−1, (r−1)/2, λ±1, the k2-negative-fix region, bit-{126,127,128,129,253} boundaries) through the real `use_glv=true` path; checks the 2-limb (≤128-bit) split halves recombine to k·P, catching any silent >128-bit truncation or top-window carry drop. - `DedupLargeClusterCarryAndCaps` — equal-scalar clusters of distinct points past `DEDUP_MAX_CHUNK_MEMBERS` and the staged cap, exercising the multi-chunk tree-reduce carry and the partial-consolidation fallback (vs naive and vs dedup-off). - `ExternalGlvDoubledDirect` — the `external_glv_doubled` aliasing branch directly (was only reached transitively via the batch driver), incl. N above the native GLV threshold. **Batch driver (the production commit-batch path):** - `BatchDriverSharedPathRagged` — `batch_multi_scalar_mul(handle_edge_cases=false)` is the **only** entry into the shared `pippenger_round_parallel_batched` path (GLV-group assignment, `external_glv_doubled` aliasing, single shared arena sized to the largest member). Every existing batch test uses the default `handle_edge_cases=true` and routes around it. Ragged batch with empty MSMs interspersed (the `n==0 → infinity` skips), fully-zero MSMs, a mixed GLV / non-GLV group split, total nonzero past the >4096 REBALANCE threshold, and mixed per-MSM dedup hints. **Large-N multi-threaded edge cases (Jacobian path):** - `HandleEdgeCases{PointAtInfinity,InversePairs,AllInfinity}` — infinity / P+(−P) bucket collisions at large N with forced 8 threads, hitting the multi-threaded Jacobian split + cross-thread reduction. `scalar_multiplication_safe_mode.test.cpp` covers these only at ≤60 pts (single-threaded). ## Verification Built `ecc_tests` on native (x86_64) — all new tests pass for BN254 and Grumpkin; full `*ScalarMultiplication*`+`*VariableWindow*` suite green. Then rebuilt under the `asan-fast` preset (ASAN + `_GLIBCXX_DEBUG`) and ran the entire suite (123 tests): clean after the OOB fix — the **real** Zone P/W/S allocator validated across the whole swept dispatch space, not just the test-side layout mirror. clang-format-20 clean. Large/GLV-threshold-dependent cases are `__wasm__`-skipped per the file's convention. --------- Co-authored-by: iakovenkos <sergey.s.yakovenko@gmail.com>
BEGIN_COMMIT_OVERRIDE fix(e2e): always init bb sync in wallet worker (#23855) fix(e2e): poll for high-value inclusion in n_tps bench (#23858) test: stabilize validator_nuke_and_suppression post-recovery assertion (#23852) chore: run next-net on internal IP addresses (#23862) fix: init bb sync before node rpc (#23864) fix: check epoch job overlap (#23481) fix: interrupt checkpoint job on sequencer stop (#23930) END_COMMIT_OVERRIDE
> [!IMPORTANT] > **Draft until the IAM policy is applied.** Depends on AztecProtocol/github-aws-oidc-factory#21 (grants the `pipeline-exec` role `ec2:CreateFleet`/`CreateLaunchTemplate`/`DeleteLaunchTemplate` + the describe/launch-template perms). Without it, provisioning hits `UnauthorizedOperation`. Marking ready once that policy is live. ## Problem Build instances almost never won their spot bid. Two root causes: 1. **Diversification never ran.** `aws_request_instance` looped `m6a → m7a → m7i`, but `aws_request_instance_type` fell back to on-demand *inside the first pool's call*, so m6a.32xlarge spot failed → m6a on-demand succeeded → returned. m7a/m7i spot was never tried (and `m7i.32xlarge` doesn't even exist). 2. **A single pool is the worst case.** AWS Spot Placement Score for `m6a.32xlarge` in us-east-2 is **1/10** in every AZ — one instance type, effectively one AZ, with a sub-on-demand per-cpu bid. ## Change Replace the legacy `request-spot-instances` call with a single **`aws ec2 create-fleet --type instant`** per request: - **`capacity-optimized-prioritized`** across an expanded pool set spanning **192/128/64-vCPU** sizes (48xl/32xl/16xl) and **all AZs** (subnet overrides in the SG's VPC). `Priority` prefers 128 → 192 → 64; 64 is still taken as spot before any on-demand. - **Memory-safe families only** (m + r, ≥4 GB/vCPU). Measured build peak is ~1.9 GB/vCPU (driven by the parallel build fan-out), so c-series (2 GB/vCPU) is excluded by default and gated behind `CI_SPOT_INCLUDE_C` pending build-parallelism throttling. - **Spot retry loop** within a `CI_SPOT_TIMEOUT` budget (default 60s, `CI_SPOT_POLL` interval), preserving the historical "try spot ~1 min, then on-demand" behaviour; each retry re-samples the whole diversified pool. - **On-demand fallback** (`lowest-price` create-fleet) only after the spot budget expires. - An **ephemeral launch template** carries the common config (AMI/SG/instance-profile/EBS + launch-time tags) and is deleted on exit. `--type instant` leaves no open spot request or persistent fleet, so teardown stays `iid`-based. `bootstrap_ec2` now requests `cores=192,128,64`. ## Files - `ci3/aws_request_instance` — rewritten as the override-list builder (size→priority, family pools, `AWS_INSTANCE`/`CI_SPOT_INCLUDE_C` handling). - `ci3/aws_request_instance_type` — create-fleet provisioning (LT, subnet lookup, spot retry loop, on-demand fallback, markers, LT cleanup); IP-wait/SSM/SSH tail unchanged. - `ci3/bootstrap_ec2` — `cores=192,128,64`. ## Validation Exercised live against us-east-2: - Builder emits correct tokens/priorities across amd64, arm64, `CI_SPOT_INCLUDE_C`, `AWS_INSTANCE`, and the uncurated `cpus=4` path. - create-fleet returned a **spot `m7i.48xlarge`** immediately (48 overrides, valid config). - Real script end-to-end: spot acquired via the retry loop, markers written, SSM `Online`, launch template deleted, clean exit. - On-demand fallback (`NO_SPOT=1`) acquires and marks `ondemand`. - Teardown: no leaked instances, launch templates, or fleets.
BEGIN_COMMIT_OVERRIDE chore: reduce parallelism for e2e check circuit (#24458) END_COMMIT_OVERRIDE
…duces #24471) (#24472) ## What Reproduces the changes in #24471 (originally opened from a fork) so it runs under the org's CI/merge tooling. Corrects the testnet **Sponsored FPC address** and the `v4.3.1` versioned getting-started page to the live `5.0.0-rc.2` testnet values. Two files, +5/−5 (identical to #24471 — resulting blob SHAs `e7521e40cef` and `56c6a7445df` match): - `docs/docs-developers/getting_started_on_testnet.md` — `SPONSORED_FPC_ADDRESS` `0x261366…7880` → `0x1969946536f0c09269e2c75e414eef4e21a76e763c5514125208db33d7d944d7` - `docs/developer_versioned_docs/version-v4.3.1/getting_started_on_testnet.md` — install/warning version `4.3.1` → `5.0.0-rc.2`, `NODE_URL` `https://rpc.testnet.aztec-labs.com` → `https://v5.testnet.rpc.aztec-labs.com`, `SPONSORED_FPC_ADDRESS` `0x08b888…f765b` → the same `0x1969…944d7` ## Validation Verified against the **live rc.2 testnet** (`https://v5.testnet.rpc.aztec-labs.com`, rollupVersion `2787991301`) via `node_getContract`: - `0x1969946536…944d7` → **deployed** (version 2, contractClassId `0x095a9366d37ca82e92326aa9dea8c917d9a9c859ebdf6a116299c5a9e5f7ef3c`) — the real Sponsored FPC ✅ - `0x261366…7880` → **null** (not deployed) — the wrong value previously on `next` - `0x08b888…f765b` → **null** (not deployed) — the older wrong value in the versioned page ## Notes - Supersedes fork PR #24471 — that one can be closed in favor of this. - The same wrong FPC (`0x2613…`) is also present on `v5-next`'s `getting_started_on_testnet.md`; a matching correction there is a separate small PR (not included here). --- *Created by [claudebox](https://claudebox.work/v2/sessions/59cd7d98b38b9d90) · group: `slackbot`*
Fix A-1360. Stacked on top of #24459
BEGIN_COMMIT_OVERRIDE fix: improve valkeys password handling (#24459) chore: pin pako (#24501) feat: allow separate valkeys passwords. (#24460) chore: loosen alert routing rules (#24499) chore: default metrics-deploy Slack alert mention to ClaudeBox (#24484) feat: speed up valkeys generation (#24467) fix(release-image): stamp stdlib/package.json with release version (#24410) END_COMMIT_OVERRIDE
Sets up the merge-train automation for a `v6-next` release line, mirroring the existing `spartan-v5` → `v5-next` pattern, so public v6 work can run on `merge-train/spartan-v6` targeting `v6-next` (forked off `v5-next`). ## Changes - **`merge-train-create-pr.yml`** — generalize base-branch detection from a hard-coded `*-v5` check to any `-v<N>` suffix (`-v5` → `v5-next`, `-v6` → `v6-next`; no change to v5 behavior). Add `merge-train/spartan-v6` to the `ci-full-no-test-cache` list. - **`merge-train-next-to-branches.yml`** — add `v6-next` to the push trigger and a branch for it that auto-pulls `v6-next` into `merge-train/spartan-v6`. - **`merge-train-stale-check.yml`** — add a daily `spartan-v6` stale-check job (`BASE_BRANCH: v6-next`, `#team-alpha`). ## Deliberately out of scope - **`private-port-next` label** stays scoped to `v5-next` bases, so `spartan-v6` PRs are **not** auto-forward-ported to `next`. `v6-next` is a staging line to be ported at release time, not per-PR. - **`notify-private-sync.yml`** is untouched — `v6-next` is public-first with no private mirror, so it must not dispatch a (non-existent) `sync-upstream-v6-next.yml`. - **Nightly release-tag / spartan-bench** are untouched — `v6-next` is not a release line yet. ## Required branch-creation sequence (needs branch-push rights — not doable via the bot) These workflows execute from the pushed branch's own copy, so the wiring must be present on `v6-next` before the train branch is created: 1. Merge this PR to `next` and let it propagate to `v5-next` (the merge-train workflows are kept byte-identical across `next`/`v5-next`), **or** merge `next` → `v5-next` once this lands. 2. Create `v6-next` off `v5-next` (now carrying the v6 wiring). 3. Create `merge-train/spartan-v6` off `v6-next`. `create-pr` will see the tip is already in `v6-next` and skip; the first real commit opens the train PR against `v6-next`. Creating `merge-train/spartan-v6` before step 2 completes would make `create-pr` fall back to base `next` and open a spurious full-diff PR — so keep the ordering. --- *Created by [claudebox](https://claudebox.work/v2/sessions/706da64a13de6c13) · group: `slackbot`*
…24433) ## Problem A single merge-train commit spawned **three** concurrent CI runs (observed on PR #24431 for commit `b7940093a5` — three `pull_request` runs, same SHA, same second), each launching its own EC2 box with the identical name. ## Root cause Merge-train PRs are intentionally exempt from CI cancellation so each *distinct* accumulated train commit gets tested. But the concurrency group keyed on `github.run_id`: ``` group: ci3-${{ (startsWith(github.event.pull_request.head.ref, 'merge-train/') && github.run_id) || ... }} ``` `github.run_id` is unique **per run**, so `cancel-in-progress` never collapsed anything — including duplicate `pull_request` events for the *same* commit. When a sub-PR merges into `merge-train/spartan`, the bot fires a burst at one SHA: a `synchronize` plus re-applying the `ci-no-squash` and `ci-full-no-test-cache` labels (the `labeled` type is enabled). Each became its own uncancellable run. Those runs then all built the same instance name (`aztec-packages_merge-train_spartan_amd64_x-full-no-test-cache`) and raced `bootstrap_ec2`'s same-name reap (a check-then-launch race), so all three boxes survived. ## Fix - **`ci3.yml`** — key the merge-train concurrency group on `github.event.pull_request.head.sha` instead of `github.run_id`. Same-commit duplicate events now collapse (`cancel-in-progress`); distinct commits still run concurrently. Non-merge-train PRs, `merge_group`, and `push` are untouched — the changed operand is only selected when the head ref is `merge-train/*` (for everything else `startsWith(...)` is false and the expression falls through to `merge_group.head_ref || ref_name` exactly as before). - **`bootstrap_ec2`** — append the commit sha to merge-train instance names, so the distinct commits that *do* run concurrently get their own boxes instead of colliding / reaping each other. `ci.sh`'s connect helpers match the Name tag by substring, so the sha-free base name from `aws_instance_name` still resolves them. ## Verification - Merge a sub-PR into `merge-train/spartan`; confirm the `synchronize` + two `labeled` events collapse to **one** in-progress `ci3` run for that SHA and exactly one instance is requested. - Non-regression: push two commits quickly to a normal PR; the older run is cancelled, only the latest proceeds (unchanged).
…epo's mode (#24523) ## Problem When a `merge-train/*` PR on **`aztec-packages-private`** is dequeued from the merge queue, `merge-queue-dequeue-notify.yml` posts to the team channel (e.g. `#honk-team` for `merge-train/barretenberg`) and kicks off a ClaudeBox session to investigate and open a fix PR. That kickoff (`.github/workflows/claudebox.yml`) hardcoded `CLAUDEBOX_MODE: public`. So the fix session lands in **public** mode, where it has no access to `aztec-packages-private` — it can't read the dequeued PR, can't fetch the private `merge-train/*` branch, and can't open the fix PR. The session correctly refuses and tells a human to re-run with `--private`, which defeats the point of the automated kickoff. Root cause: two things defeated claudebox-server's existing "private repo ⇒ private mode" safety net (`RepoRequiresPrivateMode`): 1. `mode: public` was sent explicitly, overriding the inference. 2. The `repo` sent to ClaudeBox was `github.repository` of the (public) workflow run, not the private repo the train actually lives on. ## Fix Thread the **target repo** through the kickoff so the session's mode matches where the train lives: - **`ci3/merge_train_failure_slack_notify`** — pass `--repo "$GITHUB_REPOSITORY"` on both the dequeue and CI-failure kickoffs (the repo the dequeue/failure fired in is the repo the train is on). - **`ci3/slack_notify_with_claudebox_kickoff`** — accept `--repo` / `--mode` and forward them as `-f repo=` / `-f mode=` workflow inputs. Existing callers (`release_canary_slack_notify`) are unaffected — the flags are optional. - **`.github/workflows/claudebox.yml`** — add `repo` and `mode` `workflow_dispatch` inputs. Mode is now selected in priority order: 1. an explicit `mode` input; 2. **private** when the target repo is a private mirror (`…-private`); 3. the existing default (`vars.CLAUDEBOX_MODE || 'public'`) for ordinary `/claudebox` work on `aztec-packages`. The kickoff still dispatches the **public** `aztec-packages` `claudebox.yml` (which holds `CLAUDEBOX_API_SECRET`); only the payload's `repo`/`mode` change, so a private-train dequeue now runs the fix session in **private** mode against `aztec-packages-private`. The `/claudebox` PR-comment path is unchanged: for public-repo comments the target repo is public, so it stays public. ## Testing These are GitHub Actions / bash config files with no unit-test harness in this repo, so no red/green unit test is feasible. Verified: - `bash -n` on both edited ci3 scripts. - YAML parse of `claudebox.yml`. - Traced the expression: `endsWith('AztecProtocol/aztec-packages-private', '-private')` → `private`; `endsWith('AztecProtocol/aztec-packages', '-private')` → falls through to `public`. Empty `inputs.mode`/`inputs.repo` (the `issue_comment` path) fall through to the public default. ## Notes These files mirror to `aztec-packages-private`, where the dequeue workflow actually runs, so the fix takes effect there once mirrored. --- *Created by [claudebox](https://claudebox.work/v2/sessions/99df29bc5e0ecc79) · group: `slackbot`*
Adds a GitHub Actions Workload Identity pool/provider restricted to the aztec-labs-eng/treasury-infra repository, a treasury-infra-ci service account it may impersonate, and a treasury-infra Docker repository the account can write to. Lets that repo's CI push its propose-watcher image keylessly; GKE nodes already read Artifact Registry project-wide. The pool/provider names (github / treasury-infra) are referenced by the GCP_WIF_PROVIDER variable already set on the treasury-infra repo.
Adds the IAM for `aztec-labs-eng/treasury-infra` GitHub Actions to push Docker images (the propose-watcher service, [treasury-infra#4](aztec-labs-eng/treasury-infra#4)) to Artifact Registry in `testnet-440309` — keyless via Workload Identity Federation, as requested by Alex instead of ad-hoc gcloud commands. **What it declares** (in `spartan/terraform/gke-cluster/`): - `iam.tf`: service account `treasury-infra-ci`, a `github` Workload Identity pool, an OIDC provider trusting GitHub-issued tokens **with an attribute condition pinning it to exactly the `aztec-labs-eng/treasury-infra` repository**, and the `workloadIdentityUser` binding. - `docker-registry.tf`: Docker repository `treasury-infra` (us-west1) and an `artifactregistry.writer` binding for the CI service account **scoped to that one repository** — nothing project-wide. **Notes for the applier:** - `terraform validate` passes; I don't have permissions to plan/apply against the live state (editor only). - A previously hand-created `treasury-infra` AR repo and `treasury-infra-ci` SA were deleted before this PR, so `terraform apply` creates everything fresh — no imports needed. Expected plan: **6 to add, 0 to change, 0 to destroy**. - GKE image pulls need nothing: `aztec-gke-nodes-sa` already has project-level `artifactregistry.reader` (managed in this same file). - The consuming workflow and repo variables are already in place on the treasury-infra side; once this applies, its `build-push` job works immediately.
…24539) Follow-up to the merged v6 merge-train wiring (#24516 / #24517). Generalizes the `private-port-next` auto-label in `merge-train-create-pr.yml` from `v5-next`-only to any `v<N>-next` base, so `merge-train/spartan-v6` PRs into `v6-next` are automatically forward-ported by the private-port-next driver — matching the agreed v6 plan. ``` - if [[ "$base_branch" == "v5-next" ]]; then + if [[ "$base_branch" =~ ^v[0-9]+-next$ ]]; then ``` No change to v5 behavior (`v5-next` still matches). Companion to #24538 (same change on `v5-next`), keeping the `next`/`v5-next` copies of this workflow identical. --- *Created by [claudebox](https://claudebox.work/v2/sessions/706da64a13de6c13) · group: `slackbot`*
…tation test advanceToEpochBeforeProposer checked 20 epochs for the target proposer. With a 2-slot epoch and warmupSlots=1 each attempt inspects a single slot, and with a 4-member committee the target is that slot's proposer with probability 1/4 (proposer index is keccak(epoch, slot, seed) mod committee size, independent per attempt). The miss probability over 20 attempts is (3/4)^20 ~= 0.32%, which showed up as a real CI flake. 50 attempts bound it at ~6e-7; each attempt is only a committee query plus an anvil warp.
AztecBot
added
ci-draft
spalladino
pushed a commit
that referenced
this pull request
Jul 6, 2026
…uplicate_proposal test (#24557) ## Problem `multi-node/slashing/duplicate_proposal › slashes validator who sends duplicate proposals` (`yarn-project/end-to-end/src/multi-node/slashing/equivocation_offenses.test.ts`) flaked in CI on the unrelated PR #24556 with: ``` Target proposer 0x90f79bf6eb2c4f870365e785982e1f101e93b906 not found in any slot after 20 epoch attempts ``` Evidence: http://ci.aztec-labs.com/eb0b4e658e28440c (the sibling `duplicate_attestation` case in the same file passed in the same run). ## Root cause Pure statistics — no logic bug. `advanceToEpochBeforeProposer` (`yarn-project/end-to-end/src/multi-node/slashing/setup.ts` on this branch) scans upcoming epochs for one where the target validator is the proposer, giving up after `maxAttempts = 20` epochs. The test calls it with the default. This suite uses a 2-slot epoch with `warmupSlots = 1`, so each epoch attempt inspects exactly **one** candidate slot. The proposer for a slot is `keccak256(abi.encode(epoch, slot, seed)) % committeeSize`, and with the suite's 4-member committee (`COMMITTEE_SIZE = NUM_VALIDATORS = 4` in `setup.ts`) each attempt is an independent 1/4 draw. Miss probability over 20 attempts: `(3/4)^20 ≈ 0.32%`, i.e. roughly 1 in 315 runs — matching the observed occasional flake. ## Fix Raise the default `maxAttempts` from 20 to 50, bounding the miss probability at `(3/4)^50 ≈ 5.7e-7` (~1 in 1.8M). Each attempt is only a committee query plus an anvil timestamp warp, so the expected attempt count stays ~4 and the extra headroom costs essentially nothing. The raised default also covers this helper's other callers with the same exposure (`broadcasted_invalid_block_proposal_slash`, `broadcasted_invalid_checkpoint_proposal_slash`), none of which override `maxAttempts`. ## Relationship to PR #24548 This is the same failure mode fixed by #24548 for the `duplicate_attestation` flake, but #24548 does **not** cover this call site: its patch touches `yarn-project/end-to-end/src/e2e_p2p/shared.ts`, a file that does not exist on `merge-train/spartan-v5` (on this branch the helper lives in `multi-node/slashing/setup.ts`, and there is no `e2e_p2p/` directory). #24548's head branch was cut from a `next`-layout tree while targeting `merge-train/spartan-v5`, which is why GitHub reports it as dirty with ~4k changed files — it needs a rebase onto the branch it targets, at which point its fix would land in this same file. This PR applies the fix directly to the `merge-train/spartan-v5` copy of the helper. No tracked issue exists for this flake. --- *Created by [claudebox](https://claudebox.work/v2/sessions/0959caa9f03f59fa) · group: `slackbot`*
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Superseded by #24557.
This PR was originally written against
next, fixingduplicate_attestation.test.tsviayarn-project/end-to-end/src/e2e_p2p/shared.ts. After retargeting the PR base tomerge-train/spartan-v5, we found that file doesn't exist on that branch at all (noe2e_p2p/directory), so this diff can't be rebased/cherry-picked onto it — the fix needs to live in a different file (yarn-project/end-to-end/src/multi-node/slashing/setup.ts) for this branch's layout.#24557 already fixes the identical root cause (proposer-search retry budget too low —
(3/4)^20 ≈ 0.32%miss rate) in the correct location formerge-train/spartan-v5. Closing this one as redundant/unmergeable rather than force-rebasing a diff that has no valid target on this branch.Created by claudebox · group:
slackbot