Skip to content

ci: cross-OS solx-tester suite on free macOS/Windows runners (ci:cross-os label + weekly cron) - #553

Draft
nebasuke wants to merge 3 commits into
mainfrom
cross-os-tester
Draft

ci: cross-OS solx-tester suite on free macOS/Windows runners (ci:cross-os label + weekly cron)#553
nebasuke wants to merge 3 commits into
mainfrom
cross-os-tester

Conversation

@nebasuke

@nebasuke nebasuke commented Jul 15, 2026

Copy link
Copy Markdown
Member

What

A label-gated + weekly-cron workflow running the full solx-tester suite on free macOS (arm64 + Intel) and Windows runners, plus the run-tester-suite composite action it is built from. The workflow consumes the build-toolchain composite (landed separately in #581) for its standard-config LLVM/solc prologue, so it adds no toolchain-build steps of its own.

  • Triggers: PR label ci:cross-os (on-demand, for platform-sensitive changes like process/IPC work), workflow_dispatch, and a Monday 06:00 UTC cron on main as a drift net. Advisory only — must never become a required check.
  • Scope: solx-tester only. Cross-OS legs test platform behavior (process management, IPC, paths, toolchain quirks); output quality stays on the Linux integration suite, and Hardhat/Foundry are deliberately not replicated.
  • Cost: $0 — macos-15, macos-15-intel, and windows-2025 are free for public repos; LLVM/solc come from the finished-install caches saved on main, so nothing heavier than the Rust workspace compiles.

Depends on

#581 (merged) — extracted the standard-config toolchain prologue (submodules → SFW → msys → macOS-disk → LLVM → solc) into the shared build-toolchain composite. This workflow is its sixth consumer; the only change to the composite here is adding cross-os-tests.yaml to its consumer-list description. #581 also folded in #550's fix option 2, so the bundled static libstdc++.a drop now happens inside build-toolchain for every standard-config Windows build — this workflow inherits the working shared-libstdc++ link with no step of its own.

Why

One week of evidence that this layer catches what Linux cannot:

How it works

run-tester-suite (composite action): builds solx + solx-tester (symbols kept), smoke test, then the suite under a parametrized watchdog. A run that neither crashes nor finishes gets diagnosed before the kill — on macOS, sample stacks of the tester, the top-level solx parents, and the pooled workers, plus new crash reports from DiagnosticReports; on Windows, a tasklist snapshot. Signal deaths are reported by name (killed by signal 11 (SIGSEGV)) instead of the solx-dev wrapper's "subprocess failed without exit code". Stage outcomes are aggregated at the end so diagnostics always run.

Windows specifics:

Verification

Run 29429860239: macos-15 and windows-2025 legs green, full suites — ~22 min and ~37 min job time respectively. That run predates the rebase onto build-toolchain, but the composite encodes byte-identical LLVM/solc params, so the tester results carry over; a fresh ci:cross-os run on the rebased branch confirms the composite path.

The macos-15-intel leg has no post-1.97.1 timing yet (pre-fix runs exceeded 75 min without finishing), so its watchdog (110 min) and job timeout (240 min) are a probe budget rather than the ~2x margin the other legs get. First runs establish the baseline; then tighten, or scope down to --path tests/solidity if the full suite can't finish on the free Intel minis.

Follow-ups (not in this PR)

nebasuke added 3 commits July 21, 2026 08:12
Distills the mac debug harness (PR #546) and the Windows spike into a
reusable composite action: build solx/solx-tester, smoke test, suite run
under a parametrized watchdog with per-OS hang diagnostics (macOS:
sample of tester + top-level solx parents + pooled workers, crash-report
collection; Windows: tasklist snapshot), named signals, and stage
outcomes aggregated at the end so diagnostics always run.

The workflow runs the full suite on the free macos-15 and windows-2025
runners, with the two Windows prerequisites proven by the spike: drop
the bundled static libstdc++.a (solx#550) and keep solx-solidity/test
through disk cleanup. Trigger is temporarily push-to-branch for
verification; Stage 3 switches it to a ci:cross-os label +
workflow_dispatch + weekly cron on main.
Verification run 29429860239 is green on both legs (macos-15 full suite
in ~22 min job time, windows-2025 in ~37 min, free runners), so the
temporary push trigger is replaced with the real ones: the ci:cross-os
PR label for on-demand validation of platform-sensitive changes,
workflow_dispatch, and a Monday-morning cron on main as a drift net.
Advisory only — label-gated required checks can hang the merge queue.
Also names the matrix job (the include entries leaked into the display
name) and documents the label in CLAUDE.md.
Unblocked by the rustc 1.97.1 bump (#556) fixing the enum-niche
miscompile that killed solx-tester on Intel macs (#547). Full suite,
same triggers as the other legs; no post-1.97.1 Intel timing exists,
so the watchdog (110 min) and job timeout (240 min) are a probe budget
to be tightened once real runs establish a baseline. test.yaml's Intel
leg already runs on this runner, so the finished-install LLVM/solc
caches on main serve it.
@github-actions

Copy link
Copy Markdown

Coverage Summary

Crate Line Coverage Function Coverage
solx 🟢 94.7% 🟡 68.0%
solx-benchmark-converter 🔴 0.0% 🔴 0.0%
solx-codegen-evm 🟢 88.9% 🟢 83.4%
solx-compiler-downloader 🔴 0.0% 🔴 0.0%
solx-core 🟢 96.6% 🟢 91.0%
solx-dev 🔴 2.4% 🔴 3.0%
solx-evm-assembly 🟡 70.6% 🟡 66.2%
solx-solc-test-adapter 🔴 1.7% 🔴 2.1%
solx-standard-json 🟢 95.4% 🟢 91.5%
solx-tester 🔴 38.2% 🔴 35.9%
solx-utils 🟢 83.8% 🟢 83.6%
solx-yul 🟡 78.9% 🟡 71.3%
Total 🟡 53.9% 🔴 47.0%

Codecov Report | HTML Report | Workflow Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant