Skip to content

test: avoid undersized Boost.Test signal stacks - #252

Draft
l0rinc wants to merge 1 commit into
masterfrom
l0rinc/test-kernel-sigaltstack
Draft

test: avoid undersized Boost.Test signal stacks#252
l0rinc wants to merge 1 commit into
masterfrom
l0rinc/test-kernel-sigaltstack

Conversation

@l0rinc

@l0rinc l0rinc commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Problem: Boost.Test creates an alternate signal stack of SIGSTKSZ before either test binary runs.
On Alpine 3.24, musl's 8 KiB value can be smaller than Linux's hardware-dependent minimum on AVX-512/AMX CPUs, causing sigaltstack() to fail with ENOMEM before the first test case.
The same test_kernel setup failure has appeared on master, unrelated feature branches, and the Bitcoin Core GUI mirror.
For the same source commit, an AMX-capable Intel runner failed while an AMD runner passed.
Semgrep independently diagnosed the same Alpine 3.24 issue, measuring a 12,976-byte runtime minimum against musl's 8,192-byte SIGSTKSZ; the Linux documentation explains why the minimum depends on hardware state.

Fix: Define Boost's supported BOOST_TEST_DISABLE_ALT_STACK switch in both Boost.Test entry points.
The tests keep ordinary signal handling on the process stack; only stack-overflow reporting after that stack is exhausted is lost, and production binaries are unaffected.

Detailed explanation

An alternate signal stack is a separate memory area that the kernel can use when invoking a signal handler.
Its main benefit to Boost.Test is that a handler can still run after a test has exhausted the regular process stack.

Before invoking the handler, Linux writes a signal frame containing the interrupted process's register state to the selected stack.
The amount of state is hardware-dependent because newer x86 features add larger XSAVE areas.
The Linux kernel documentation therefore exposes the runtime minimum as AT_MINSIGSTKSZ and demonstrates allocating AT_MINSIGSTKSZ + SIGSTKSZ.

Boost.Test instead allocates only SIGSTKSZ.
With musl 1.2.6, sigaltstack() rejects a size below sysconf(_SC_MINSIGSTKSZ).
musl reports this undersized stack as ENOMEM, which Boost surfaces as the misleading setup failure:

Test setup error: system_error produced by: exp: Out of memory

The runner was not out of memory.
The failure happened while Boost.Test registered its signal stack, before test_kernel started its first case.
The failing runner was an Intel Xeon Platinum 8573C exposing AVX-512 and AMX, while the same source commit passed on an AMD EPYC 7763 without AMX.

Disabling Boost.Test's alternate stack avoids the invalid fixed-size registration entirely.
Ordinary signals can still be handled while the regular stack has space.
A future Boost fix could size its allocation dynamically, as Semgrep did for the same problem, but the supported compile-time switch is sufficient for these test binaries.

Boost.Test creates an alternate signal stack of `SIGSTKSZ` before either test binary runs.
Alpine 3.24's musl uses a fixed 8 KiB value, while Linux's required minimum grows with hardware register state such as AVX-512 and AMX.
On an affected Intel runner, `sigaltstack()` therefore returns `ENOMEM` before any test case runs:
https://github.com/l0rinc/bitcoin/actions/runs/30039746567/job/89316463276?pr=247

Linux documents the hardware-dependent minimum and recommends adding `AT_MINSIGSTKSZ` to the requested stack size:
https://www.kernel.org/doc/html/latest/arch/x86/elf_auxvec.html
A matching Alpine 3.24 incident measured a 12,976-byte minimum against musl's 8,192-byte `SIGSTKSZ`:
semgrep/ocaml#21

Define `BOOST_TEST_DISABLE_ALT_STACK` in both Boost.Test entry points.
Signal handlers use the regular process stack, avoiding the invalid registration.
The lost behavior is reporting a stack overflow after the regular stack is already exhausted.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant