Skip to content

ci: capture compile stderr in parity + compile-smoke jobs#239

Merged
proggeramlug merged 1 commit into
PerryTS:mainfrom
TheHypnoo:ci/capture-compile-stderr-for-long-tail-diag
Apr 28, 2026
Merged

ci: capture compile stderr in parity + compile-smoke jobs#239
proggeramlug merged 1 commit into
PerryTS:mainfrom
TheHypnoo:ci/capture-compile-stderr-for-long-tail-diag

Conversation

@TheHypnoo
Copy link
Copy Markdown
Contributor

Why

11 tests are tracked in test-parity/known_failures.json with status: \"ci-env\" and the same vague reason — "macOS-14 CI runner SDK/linker version interaction":

But the actual error message has never been pinned down, because both compile harnesses silently discard stderr:

Net effect: every PR that adds a Buffer / Uint8Array regression test ends up in the same skip list ("ci-env / passes-locally") on inference alone, and the underlying root cause never gets fixed. The v0.5.354 changelog itself acknowledges:

"the underlying SDK/linker gap is the real long-tail problem and a CI runner image bump (macOS-15) would clear all 9+ ci-env Buffer skips at once."

…but that hypothesis has never been verified against an actual error message.

What this PR does

Pure diagnostic plumbing — no source changes, no SKIP_TESTS edits, no runner bump, no version bump.

1. run_parity_tests.sh

On compile-fail, write the captured compile_output to test-parity/output/<test_name>.compile_error.log. The existing per-test loop is unchanged; this just persists the stderr the script was already capturing into a memory-only var.

2. .github/workflows/test.yml

Two new artifact upload steps (both if: always() + if-no-files-found: ignore so green CI runs produce no artifact noise):

  • parity-compile-errors-${{ runner.os }} from the parity job — gathers the per-test logs the script now writes.
  • compile-smoke-error-logs from compile-smoke — captures stderr per test to /tmp/perry_smoke_logs/, plus inlines head -n 30 of each failure log into the job output (so a quick scan in the GitHub UI reveals the error without needing to download the artifact).

What you'll see on the next CI run

The parity job naturally exercises every ci-env test (it doesn't have a SKIP_TESTS list — it tracks them via known_failures.json instead). With this PR, that job will produce ~11 *.compile_error.log files in the parity-compile-errors-macOS artifact, one per known failure. Reading any of them gives the actual link error.

Decision tree once we have the data

  • If the error matches the maintainer's hypothesis (link symbol mismatch / SDK ABI skew): file a follow-up PR bumping runs-on: macos-14macos-15 and removing the skips.
  • If the error is a real Perry bug (e.g. a stdlib codegen path that emits something the macOS-14 SDK linker rejects but local does accept): fix it in Perry rather than skipping it.
  • Either way: the diagnostic capture stays as defensive infra — future Buffer/typed-array regressions will surface the link error in the PR's CI run instead of getting silently added to the skip list.

Refs

…il diag)

Long-tail problem: 11 tests are tracked as `ci-env` in
test-parity/known_failures.json with the same vague reason — "macOS-14 CI
runner SDK/linker version interaction" — but no actual error message has
ever been pinned down because both compile harnesses silently discard
stderr:

- `run_parity_tests.sh:218` captures `compile_output` but only emits
  "compile error" with no detail.
- `.github/workflows/test.yml::compile-smoke` redirects with `2>/dev/null`,
  so the compile-smoke job summary lists only the failing test names.

Result: every PR that adds a Buffer/Uint8Array regression test ends up in
the same skip list ("ci-env / passes-locally") on inference alone, and the
underlying root cause never gets fixed.

Two changes, both diagnostic-only:

1. `run_parity_tests.sh` — on compile-fail, write the captured
   `compile_output` to `test-parity/output/<test_name>.compile_error.log`.
   Existing flow unchanged; no SKIP_TESTS / known_failures.json edits.

2. `.github/workflows/test.yml` — two new artifact upload steps:
   - `parity-compile-errors-${{ runner.os }}` from the parity job
     (gathers the per-test logs the script now writes).
   - `compile-smoke-error-logs` from compile-smoke, with the per-test
     stderr captured to `/tmp/perry_smoke_logs/`. The compile-smoke loop
     also now `head -n 30`'s each failure into the job output so a quick
     scan reveals the underlying error without downloading the artifact.

Both artifacts are uploaded with `if: always()` and
`if-no-files-found: ignore`, so when CI is green there's no artifact noise.

Once the actual link error is captured (next CI run on this PR will
trigger the parity job's stderr-capture path for the existing `ci-env`
tests), a follow-up PR can either bump the runner to macos-15 (the
maintainer's own hypothesis in v0.5.354 changelog) or fix the root cause
in Perry — depending on what the artifact shows.

No source changes, no SKIP_TESTS edits, no version bump. Pure diagnostic
plumbing. Once the long-tail is closed, these capture paths stay as
defensive infra for future regressions.
@proggeramlug proggeramlug merged commit d75f308 into PerryTS:main Apr 28, 2026
1 check passed
@TheHypnoo TheHypnoo deleted the ci/capture-compile-stderr-for-long-tail-diag branch April 28, 2026 13:11
@TheHypnoo TheHypnoo restored the ci/capture-compile-stderr-for-long-tail-diag branch April 28, 2026 13:46
@TheHypnoo TheHypnoo deleted the ci/capture-compile-stderr-for-long-tail-diag branch April 28, 2026 13:47
TheHypnoo added a commit to TheHypnoo/perry that referenced this pull request Apr 28, 2026
`crates/perry-codegen/src/expr.rs` and `lower_call.rs` emit
`call void @llvm.assume(i1 ...)` six times for branchless Buffer bounds
checks (the v0.5.183 PerryTS#92 numeric-read intrinsic + the v0.5.190 small-buffer
slab fast path), but the intrinsic was never declared in the LLVM IR
module's prelude. The math intrinsics (`llvm.sqrt.f64`, `llvm.floor.f64`,
etc.) are declared at runtime_decls.rs:117-121 — assume was simply missed.

Apple Clang ≥21 (Xcode 26 — what local devs run) auto-recognises the
intrinsic even when undeclared in the IR. Apple Clang 15 (LLVM 17 — what
ships on the macos-14 GitHub runner via Xcode 15.x) requires the explicit
declaration and errors out:

    /var/folders/.../perry_llvm.ll:1281:13: error: use of undefined value '@llvm.assume'
      call void @llvm.assume(i1 %r43)
    1 error generated.

This was the actual root cause behind the long-tail of `ci-env`
Buffer/typed-array test skips in `test-parity/known_failures.json`. The
diagnosis path:

1. The compile-stderr capture artifacts added in PR PerryTS#239 preserved the
   actual error message for the first time.
2. Downloading `parity-compile-errors-macOS` from a CI run revealed all
   11 failing tests share the exact same `@llvm.assume` undefined-value
   error — not the speculated SDK/linker version interaction.

Fix: one-line `module.declare_function("llvm.assume", VOID, &[I1])` in
`runtime_decls.rs`, alongside the existing math-intrinsic declarations.

Cleanup follow-on (now that the root cause is fixed):

- `.github/workflows/test.yml::compile-smoke` — remove the 11-entry
  Buffer-family `SKIP_TESTS` list (test_gap_buffer_ops, test_buffer_*,
  test_inline_uint8array_param, test_issue_167_*, test_issue_227_*,
  test_gap_fetch_response, test_stress_*). UI/timer skips stay
  (different reasons).

- `test-parity/known_failures.json` — remove 8 `ci-env` entries
  (test_gap_buffer_ops, test_gap_node_crypto_buffer,
  test_gap_typed_arrays, test_stress_buffer, test_buffer_small_alloc,
  test_buffer_numeric_read_intrinsic, test_issue_227_array_buffer_bytes,
  test_gap_fetch_response) plus 2 `bug` entries that were misdiagnosed
  but actually shared the same llvm.assume cause
  (test_inline_uint8array_param, test_issue_167_loop_alloca_stack_eat).
  test_stress_int_ops keeps its `bug` entry (separate console.log
  array-of-numbers padding bug exists independently).

Verified locally on Apple Clang 21: all 10 previously-CI-failing
Buffer/typed-array tests now compile cleanly. Stack: cargo build
--release -p perry-codegen + perry; `./target/release/perry compile
test-files/test_buffer_small_alloc.ts -o /tmp/...` produces a 0.7 MB
binary; `grep "@llvm.assume" /tmp/perry_llvm_*.ll` confirms the IR now
contains `declare void @llvm.assume(i1)`.

Refs PR PerryTS#239 (the diagnostic capture infra that revealed this).

# Conflicts:
#	.github/workflows/test.yml
#	test-parity/known_failures.json
proggeramlug pushed a commit that referenced this pull request Apr 28, 2026
`crates/perry-codegen/src/expr.rs` and `lower_call.rs` emit
`call void @llvm.assume(i1 ...)` six times for branchless Buffer bounds
checks (the v0.5.183 #92 numeric-read intrinsic + the v0.5.190 small-buffer
slab fast path), but the intrinsic was never declared in the LLVM IR
module's prelude. The math intrinsics (`llvm.sqrt.f64`, `llvm.floor.f64`,
etc.) are declared at runtime_decls.rs:117-121 — assume was simply missed.

Apple Clang ≥21 (Xcode 26 — what local devs run) auto-recognises the
intrinsic even when undeclared in the IR. Apple Clang 15 (LLVM 17 — what
ships on the macos-14 GitHub runner via Xcode 15.x) requires the explicit
declaration and errors out:

    /var/folders/.../perry_llvm.ll:1281:13: error: use of undefined value '@llvm.assume'
      call void @llvm.assume(i1 %r43)
    1 error generated.

This was the actual root cause behind the long-tail of `ci-env`
Buffer/typed-array test skips in `test-parity/known_failures.json`. The
diagnosis path:

1. The compile-stderr capture artifacts added in PR #239 preserved the
   actual error message for the first time.
2. Downloading `parity-compile-errors-macOS` from a CI run revealed all
   11 failing tests share the exact same `@llvm.assume` undefined-value
   error — not the speculated SDK/linker version interaction.

Fix: one-line `module.declare_function("llvm.assume", VOID, &[I1])` in
`runtime_decls.rs`, alongside the existing math-intrinsic declarations.

Cleanup follow-on (now that the root cause is fixed):

- `.github/workflows/test.yml::compile-smoke` — remove the 11-entry
  Buffer-family `SKIP_TESTS` list (test_gap_buffer_ops, test_buffer_*,
  test_inline_uint8array_param, test_issue_167_*, test_issue_227_*,
  test_gap_fetch_response, test_stress_*). UI/timer skips stay
  (different reasons).

- `test-parity/known_failures.json` — remove 8 `ci-env` entries
  (test_gap_buffer_ops, test_gap_node_crypto_buffer,
  test_gap_typed_arrays, test_stress_buffer, test_buffer_small_alloc,
  test_buffer_numeric_read_intrinsic, test_issue_227_array_buffer_bytes,
  test_gap_fetch_response) plus 2 `bug` entries that were misdiagnosed
  but actually shared the same llvm.assume cause
  (test_inline_uint8array_param, test_issue_167_loop_alloca_stack_eat).
  test_stress_int_ops keeps its `bug` entry (separate console.log
  array-of-numbers padding bug exists independently).

Verified locally on Apple Clang 21: all 10 previously-CI-failing
Buffer/typed-array tests now compile cleanly. Stack: cargo build
--release -p perry-codegen + perry; `./target/release/perry compile
test-files/test_buffer_small_alloc.ts -o /tmp/...` produces a 0.7 MB
binary; `grep "@llvm.assume" /tmp/perry_llvm_*.ll` confirms the IR now
contains `declare void @llvm.assume(i1)`.

Refs PR #239 (the diagnostic capture infra that revealed this).

# Conflicts:
#	.github/workflows/test.yml
#	test-parity/known_failures.json
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants