feat(ci): promote sanitized-fuzz from weekly to daily, scale rounds 5-7x (#109) - #129
Merged
Conversation
…-7x (#109) Smoke-run on #128 (100 ASan + 50 TSan rounds via workflow_dispatch) showed the workflow takes ~9 minutes end-to-end and measured ~1.4 s/round across both sanitizers — orchestration cost, not sanitizer overhead, dominates per-round time. That leaves significant headroom: the original 1500/500-round weekly cadence projects to ~50 minutes (1.4 s × 2000) of actual fuzz work, well under the 6h hosted-runner ceiling. Public-repo Actions minutes are unmetered, so the bottleneck is wall time per run, not budget. Promote to daily (12:00 UTC+8 = 04:00 UTC) and raise rounds to fill a ~3.5h fuzz envelope: | Pass | rounds (was → now) | wall ~ | notes | |------|--------------------|--------|---------------------------------| | ASan | 1500 → **5000** | ~120m | step 130m, inner timeout 125m | | TSan | 500 → **3500** | ~80m | step 100m, inner timeout 95m | Job timeout-minutes raised 200 → 240 to accept the larger fuzz envelope plus build (~3.5m for both sanitizer Debug builds) + apt + codegen (~3m) + setup (~1m) ≈ 205m, leaving ~35m runner-noise headroom. Per-round cost is empirically symmetric across ASan and TSan because the per-round work in differential-fuzz.py is dominated by spawning go / java / cpp child processes and serializing their JSON outputs; the cpp-side sanitizer slowdown is amortized to background. So the TSan-vs-ASan ratio is closer to 1:1 in this orchestration than the 3-10x raw cpp ratio would suggest. Treat this as the operating model; future cpp work that lifts cpp's per-round share (say, in-process fuzz) will need budgets re-derived. Rename `weekly-sanitized-fuzz.yml` → `daily-sanitized-fuzz.yml` (git mv) so cron / file name / workflow name stay coherent. Updated the workflow `name:`, concurrency group, all comments, all step summary labels, all issue titles, and the divergence/incomplete artifact filename. No semantic logic change beyond cadence + rounds + budgets. The bot-flagged cancellation-bypass small in #128 (job timeout canceling mid-fuzz so `!cancelled()` evaluates false on the incomplete-issue step) becomes more likely with the larger envelope but still has ~35m headroom on a typical run; observe-and-iterate remains the right call. #109 / #124 background unchanged: this workflow is the deep-diagnostic complement to nightly diff-fuzz, designed to surface sanitizer stacks at the moment a divergence occurs rather than after a Release-mode divergence triggers a reactive 40x TSan replay (which on #109 / #124 produced "No ThreadSanitizer report" because the rare window had already closed).
Contributor
🔍 PR 审查
cadence/rounds/timeout 调整合理，git mv 保留了历史，所有 "weekly" 引用已同步重命名。无阻塞问题；但 incomplete-pass issue 模板内残留了旧的 timeout/rounds 数值，且预算注释对构建耗时估计偏乐观。
|
…unds (#129) Address bot-reviewer concern: the "Open issue on incomplete pass" step's triage suggestion text held over from #128 (ASan 230m / TSan 85m / TSan 500 rounds), but this PR cut ASan inner to 125m, raised TSan inner to 95m, and raised TSan rounds to 3500. A future incomplete-pass issue filed by this workflow would have shown stale numbers and misled the "bump the limit or cut rounds" decision. Refresh the per-exit-code triage paragraph to reflect: - ASan inner timeout 125m, 5000 rounds (~120m projected, ~5m head) - TSan inner timeout 95m, 3500 rounds (~80m projected, ~15m head) - Add a one-liner: routine exit=124 → runner noise, bump timeout first; only cut rounds if recurrent (not the previous "may need budget verification" hedge that no longer applies after the smoke run). No behavior change — pure documentation accuracy fix in a step that fires only on degraded runs.
Contributor
🔍 PR 增量审查
本次增量（ 已交叉核对全文件，triage 文案与各处配置完全一致：
增量改动无阻塞问题，质量良好。 |
3 tasks
Liam0205
added a commit
that referenced
this pull request
Jun 19, 2026
Re-release attempt. v0.10.7 was tagged on 2026-06-17 (commit 5618f49) but its release.yml workflow_run trigger failed to fire (root cause fixed in #126), so PyPI / Maven Central never received v0.10.7 artifacts and there is no GitHub Release entry. Re-publishing v0.10.7 by force-moving the tag to a new commit is not viable: Go module proxy already cached pine-go/v0.10.7 → 5618f49 (see proxy.golang.org/.../@v/v0.10.7.info, served at 2026-06-17 15:47Z). Moving the tag would create an unfixable mismatch between the git tag SHA and the proxy-cached SHA, so any `go get @v0.10.7` keeps fetching 5618f49 regardless of what git says. Bump to 0.10.8 instead, which captures the post-#126 commits inline: bc8d6ac chore: bump 0.10.6 → 0.10.7 + #119 / #125 fixes 6471007 fix(ci): release.yml workflow_call gate (#123) ce4dbff fix(ci): incomplete-pass signal (#109) b934876 feat(ci): weekly sanitized fuzz (#109) ef24382 feat(ci): weekly → daily, scale rounds (#109) 619db9c fix(ci): refresh incomplete-pass triage doc (#129) bump-version.sh ran the full four-runtime test + cross-validate sweep before this commit; all 12 cross-validate sections PASS. Next: make tag-release will create v0.10.8 + pine-go/v0.10.8 and push, which now triggers release.yml directly via push:tags (no more workflow_run quirk) → invokes ci.yml as workflow_call gate → on green publishes to PyPI / Maven Central. v0.10.7 stays as a stale git tag pointing at 5618f49. Document it as "never published" if anyone notices the gap; not worth deleting since deletion does not unmake the proxy cache.
4 tasks
Liam0205
added a commit
that referenced
this pull request
Jun 21, 2026
…put (#132 #133) The 2026-06-20 and 2026-06-21 daily-sanitized-fuzz runs both tripped exit=124 on BOTH the ASan and the TSan pass — the workflow's diagnostic capability went to zero for two consecutive days. Both issues (#132, #133) auto-filed by the incomplete-pass detector are unanimous: ASan 5000-round and TSan 3500-round budgets are too aggressive for production-grade runner pressure. Per-round time was assumed at ~1.4 s/round based on the #128 smoke run (100 / 50 rounds in 9 minutes). Production runs measure ~1.5 s/round ASan and ~1.6 s/round TSan — close to the smoke-run estimate but not within the 5-10 % buffer the original budget allowed. Possible reasons: - Smoke runs do not accumulate sanitizer shadow-memory pressure that builds over thousands of rounds. - Runner noise compounds: 100 rounds across a quiet runner ≠ 5000 rounds across a runner that may swap, GC-pause, page-fault, etc. - ASan / UBSan with -O1 -g and TSan with shadow-of-shadow tracking both have non-linear cost growth as the heap and seen-event sets fill up. Cut rounds 40 % and re-derive the budgets from the measured rates, with a 20 % headroom inside each inner timeout (vs. the previous ~5-15 %, which proved insufficient): | | rounds (was → now) | rate | inner timeout (was → now) | headroom | |---|---|---|---|---| | ASan | 5000 → **3000** | 1.5 s/r | 125m → **90m** | ~15m of 90m (20 %) | | TSan | 3500 → **2000** | 1.6 s/r | 95m → **65m** | ~10m of 65m (~18 %) | Step `timeout-minutes` and the job-level `timeout-minutes` follow the same pattern: 130/100 → 95/70, 240 → 180. Total wall-time projection ~140m, leaving ~40m headroom against the 180m job ceiling for build + apt + codegen-parity overhead. Also refreshed: - Header rationale comment (now references the #132/#133 production measurements, not the smoke-run estimate) - Step-summary table headers (defaults `'5000'` / `'3500'` → `'3000'` / `'2000'`) - Incomplete-pass issue triage paragraph (`125m` → `90m`, `95m` → `65m`, round counts and rate sources reflect production) Coverage trade-off: 2000 + 3000 rounds is still ~5x the original weekly cadence (1500/500), and unlike weekly we run daily — so seven-day cumulative coverage is 21000 ASan + 14000 TSan rounds vs. the previous weekly 1500 + 500. Net coverage strictly improved over the original design, just trimmed below the over-aggressive #129 levels. Closes #132. Closes #133. Out of scope: #109 root-cause fix (TSan finally captured the race in #131; analysis posted on #109; fix design pending). Daily fuzz must run cleanly first or the next #109 race window will silently slip past again — which is exactly what this commit restores.
Merged
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #128.
Why scale up
The smoke run on #128 (100 ASan + 50 TSan via workflow_dispatch) measured the workflow at ~9 minutes end-to-end, with ~1.4 s/round across both sanitizers. Per-round time is dominated by the orchestration loop in
differential-fuzz.py(spawn go/java/cpp, serialize JSON outputs), not by sanitizer instrumentation overhead. That makes the original weekly 1500/500-round cadence ~50 minutes of fuzz work — far below what the runner can absorb.Public-repo Actions minutes are unmetered. The real cap is wall time per run. Promote to daily, fill the envelope.
Changes
timeout 125m)timeout 95m)weekly-sanitized-fuzz.ymldaily-sanitized-fuzz.yml(git mv, 85% similarity)Total projected wall time: ~205m, leaving ~35m headroom for runner-side noise within the 240m job ceiling.
Per-round symmetry between ASan and TSan
Smoke-run measured ASan and TSan at ~equal per-round time (~1.4 s) despite TSan's notional 5-10x raw cpp slowdown. Reason: each round in
differential-fuzz.pyspawns go/java/cpp child processes and serializes their outputs, so the cpp-side sanitizer cost is amortized into a smaller fraction of round time than headline numbers suggest.Treat this as the operating model. If a future change lifts cpp's per-round share (e.g. in-process fuzz harness), budgets need re-deriving.
#128 small (1) — cancellation bypass
Bot flagged that a job timeout could cancel mid-fuzz, making
!cancelled()false and bypassing the incomplete-issue step. With the larger envelope (240m vs 200m), this becomes more likely than before, but the ~35m headroom is still ample on a typical run. Observe-and-iterate remains the right call — first few daily runs will validate the actual wall-time distribution.Test plan
python3 -c 'import yaml; yaml.safe_load(...)')grep -i weekly→ 0 hits)timeout-minutes: 240