Skip to content

fix(ci): raise AVM check-circuit per-tx timeout to 180s#23742

Draft
AztecBot wants to merge 1 commit into
nextfrom
cb/avm-check-circuit-timeout-180s
Draft

fix(ci): raise AVM check-circuit per-tx timeout to 180s#23742
AztecBot wants to merge 1 commit into
nextfrom
cb/avm-check-circuit-timeout-180s

Conversation

@AztecBot
Copy link
Copy Markdown
Collaborator

What

The nightly/next AVM Circuit Inputs Collection and Check workflow's avm-check-circuit job has been failing since #23644 (chore: replace bb_cli avm_prove and avm_check_circuit). It first failed on run #1720 (commit 49f6ee5) and again on the scheduled run #1721 (389dc29).

Root cause

Of the ~785 dumped AVM circuit inputs checked per run, exactly one times out: the e2e_multiple_blobs tx 0x03713ef6…. Its check log shows:

  • Simulate + APP_LOGIC: ~2s
  • Generating trace: ~23s (the bulk)
  • Running check (with skippable) circuit over 700560 rows — killed at the 30s mark by timeout -v 30s, exit code 124

Every other tx finishes in 5–6s. The single timeout makes GNU parallel halt (--halt now,fail=1) and the job exits 124.

Each check runs isolated with CPUS=2 (docker --cpus=2), so the 700k-row e2e_multiple_blobs circuit needs ~45s of throttled CPU for trace generation plus the check — just over the existing TIMEOUT=30s. The refactor in #23644 pushed this largest tx across that tight boundary. The per-check budget, not the proving logic, is the problem; the code comment already anticipated this case ("transactions could need more CPU and MEM than we allocate by default … they might start timing out").

Fix

Raise the per-check TIMEOUT from 30s to 180s in avm_check_circuit_cmds (yarn-project/end-to-end/bootstrap.sh). This gives ~4x headroom over the observed ~45s need for the largest tx while staying under the 300s slow-job warning threshold in parallelize. Resource allocation (CPUS=2, MEM=8g) is unchanged to preserve the balanced 64-jobs × 2-CPU = 128-core model and avoid oversubscription; memory peaked at ~3.9 GiB against the 8 GiB limit, so it has ample headroom. The small/fast txs are unaffected.

CI run that surfaced this: https://github.com/AztecProtocol/aztec-packages/actions/runs/26674308080


Created by claudebox · group: slackbot

@AztecBot AztecBot added the claudebox Owned by claudebox. it can push to this PR. label May 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant