ci: debug solx-tester on macOS runners (main baseline for #524 mac validation) - #546
Draft
nebasuke wants to merge 4 commits into
Draft
ci: debug solx-tester on macOS runners (main baseline for #524 mac validation)#546nebasuke wants to merge 4 commits into
nebasuke wants to merge 4 commits into
Conversation
solx-tester has no macOS CI coverage anywhere; on PR #545 it was signal-killed within seconds on macos-15-large and ran for hours on macos-15-xlarge, but with PR #524 included, so mac breakage and #524 regressions were indistinguishable. This branch is plain main, so its results are the macOS baseline. The workflow mirrors test.yaml's build-llvm/build-solc parameters to hit the finished-install caches saved on main, invokes solx-tester directly so a signal death is named instead of collapsing to "subprocess failed without exit code", and samples hung processes before killing them. Crash reports and samples are uploaded as artifacts.
Three Intel-only steps chasing the solx-tester SIGSEGV (garbage pointer assembled from caller-address bytes; run 29364158123): a single-threaded corpus pass to test concurrency dependence, a malloc-scribble run whose 0x55/0xAA fill patterns distinguish use-after-free from uninitialized reads without a rebuild, and an ASAN-instrumented tester build that names the exact allocation/free/use sites. macos-15-xlarge is commented out of the matrix while iterating: its baseline is recorded and it is the most expensive runner.
Run 29366844647 answered the class questions: the SIGSEGV is deterministic (2s single-threaded), not heap corruption (identical 0x12-pattern registers under MallocScribble), and invisible to ASAN. What remains is naming the inlined dependency frame at Input::try_from_ethereum+3420, so run the single-threaded corpus under lldb --batch (backtrace with inline resolution, disassembly at pc, registers) with --verbose to name the crashing test, and upload the tester binary for offline symbolication of the recorded .ips offsets.
The lldb autopsy (run 29371120991) pinned the crash to an aligned SSE store to a structurally misaligned frame slot in Input::try_from_ethereum — codegen, not tester logic, matching the rust-lang/rust#159035 enum-niche miscompile family that is present in the pinned 1.96.1 and has an LLVM fix pending backport as 1.97.1. Build the tester with 1.95.0 (pre-regression) and nightly (fixed LLVM) and run the Ethereum corpus with each: if both are green while 1.96.1 crashes, the fix is a toolchain bump, not a code change.
This was referenced Jul 14, 2026
nebasuke
added a commit
that referenced
this pull request
Jul 16, 2026
Distills the mac debug harness (PR #546) and the Windows spike into a reusable composite action: build solx/solx-tester, smoke test, suite run under a parametrized watchdog with per-OS hang diagnostics (macOS: sample of tester + top-level solx parents + pooled workers, crash-report collection; Windows: tasklist snapshot), named signals, and stage outcomes aggregated at the end so diagnostics always run. The workflow runs the full suite on the free macos-15 and windows-2025 runners, with the two Windows prerequisites proven by the spike: drop the bundled static libstdc++.a (solx#550) and keep solx-solidity/test through disk cleanup. Trigger is temporarily push-to-branch for verification; Stage 3 switches it to a ci:cross-os label + workflow_dispatch + weekly cron on main.
nebasuke
added a commit
that referenced
this pull request
Jul 16, 2026
Distills the mac debug harness (PR #546) and the Windows spike into a reusable composite action: build solx/solx-tester, smoke test, suite run under a parametrized watchdog with per-OS hang diagnostics (macOS: sample of tester + top-level solx parents + pooled workers, crash-report collection; Windows: tasklist snapshot), named signals, and stage outcomes aggregated at the end so diagnostics always run. The workflow runs the full suite on the free macos-15 and windows-2025 runners, with the two Windows prerequisites proven by the spike: drop the bundled static libstdc++.a (solx#550) and keep solx-solidity/test through disk cleanup. Trigger is temporarily push-to-branch for verification; Stage 3 switches it to a ci:cross-os label + workflow_dispatch + weekly cron on main.
nebasuke
added a commit
that referenced
this pull request
Jul 21, 2026
Distills the mac debug harness (PR #546) and the Windows spike into a reusable composite action: build solx/solx-tester, smoke test, suite run under a parametrized watchdog with per-OS hang diagnostics (macOS: sample of tester + top-level solx parents + pooled workers, crash-report collection; Windows: tasklist snapshot), named signals, and stage outcomes aggregated at the end so diagnostics always run. The workflow runs the full suite on the free macos-15 and windows-2025 runners, with the two Windows prerequisites proven by the spike: drop the bundled static libstdc++.a (solx#550) and keep solx-solidity/test through disk cleanup. Trigger is temporarily push-to-branch for verification; Stage 3 switches it to a ci:cross-os label + workflow_dispatch + weekly cron on main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#524 (persistent worker subprocesses) needs macOS validation: worker-thread changes have a history of regressing badly on Mac while looking fine on Linux. #545 tried running the integration suite on large macOS runners on top of #524, and
solx-testerwas signal-killed within seconds onmacos-15-largeand ran for hours without finishing onmacos-15-xlarge— but that run can't distinguish "solx-tester has never worked on macOS" from "#524 regressed on macOS". Notably, on the Intel runner main's own binaries crashed identically in the baseline-benchmark step, so at least the Intel crash predates #524.This branch is plain
main, so whatever it does on macOS is the macOS baseline:What it does
One push-triggered workflow (
mac-tester-debug.yaml), matrix overmacos-15,macos-15-large,macos-15-xlarge:build-llvm/build-solcare called with test.yaml's exact macOS parameters, so both hit the finished-install caches saved onmain(v1-llvm-macOS-*,v1-solc-macOS-*). Only the Rust workspace compiles; an iteration is ~15–20 min.solx-testeris invoked directly instead of viasolx-dev test solx-tester— the wrapper collapses a signal death tosubprocess failed without exit code; here the step prints e.g.killed by signal 11 (SIGSEGV).strip = none+ line tables so crash reports andsampleoutput are readable; LLVM is RelWithDebInfo with assertions on.samples the tester and its--recursive-processworkers before killing them (for the xlarge-style hang)..ipscrash reports from~/Library/Logs/DiagnosticReportsand allsampleoutputs are printed and uploaded as a per-runner artifact.Uses the
free-disk-space-macosaction from #533 (just merged), which with warm artifact caches usually skips the cleanup entirely.