`Thread.test.thread local storage` fails randomly on multiple 32-bit architectures

[alexrp]

We've previously seen this on arm/armeb and thumb/thumbeb:

test
+- test-modules
   +- test-std
      +- run test std-riscv32-linux-none-baseline_rv32-Debug failure
thread 2151089 panic: reached unreachable code
/home/ci/actions-runner4/_work/zig/zig/lib/std/posix.zig:4995:17: 0x5cfe31 in munmap (test)
        else => unreachable,
                ^
/home/ci/actions-runner4/_work/zig/zig/lib/std/Thread.zig:1507:27: 0x65d873 in join (test)
        defer posix.munmap(self.thread.mapped);
                          ^
/home/ci/actions-runner4/_work/zig/zig/lib/std/Thread.zig:466:26: 0x65715b in join (test)
    return self.impl.join();
                         ^
/home/ci/actions-runner4/_work/zig/zig/lib/std/Thread.zig:1673:17: 0x9a1021 in test.thread local storage (test)
    thread2.join();
                ^
/home/ci/actions-runner4/_work/zig/zig/lib/compiler/test_runner.zig:138:29: 0x64d61b in mainServer (test)
                test_fn.func() catch |err| switch (err) {
                            ^
/home/ci/actions-runner4/_work/zig/zig/lib/compiler/test_runner.zig:67:26: 0x64ad1b in main (test)
        return mainServer() catch @panic("internal test runner failure");
                         ^
/home/ci/actions-runner4/_work/zig/zig/lib/std/start.zig:618:22: 0x64a389 in callMain (test)
            root.main();
                     ^
error: while executing test 'Thread.test.thread local storage', the following command terminated with signal 6 (expected exited with code 0):
qemu-riscv32 /home/ci/actions-runner4/_work/zig/zig/zig-local-cache/o/6e5ac3adf343e1b75cbebaaffbf4e0e2/test --cache-dir=/home/ci/actions-runner4/_work/zig/zig/zig-local-cache --seed=0x1f99addd --listen=-

[mlugg]

I'm fairly sure this can be traced back to some kind of Linux kernel issue; not necessarily a bug per se, but at least an undocumented behavior.

Because this failure is specific to 32-bit architectures, and all of our CI runs are on 64-bit architectures which emulate the 32-bit ones with qemu, we can infer that this is probably an issue specific to QEMU's emulation of munmap for 32-bit guests on 64-bit hosts. QEMU handles this case by reserving a full 32-bit address space upfront with MAP_NORESERVE, and then lowering all calls to mmap/munmap/mremap in the guest to mmap with MAP_FIXED so that QEMU can replace a specific region of that mapping. So, the munmap call which is failing here should ultimately lower to a call which looks like:

        void *ptr = mmap(addr, len, PROT_NONE,
                         MAP_FIXED | MAP_ANONYMOUS
                         | MAP_PRIVATE | MAP_NORESERVE, -1, 0);
        return ptr == addr ? 0 : -1;

(source: do_munmap in linux-user/mmap.c in the QEMU source tree)

Assuming there isn't some deeper QEMU bug which causes part of the region to get genuinely unmapped (and I don't think there is?), the QEMU logic looks totally reasonable; and I believe the manpage for mmap indicates (if you look closely at the conditions for each error code) that this should only be able to fail with EINVAL or ENOMEM, both of which would hit a different unreachable in the stack trace. It's possible that LLVM is giving us a misleading source location by merging the unreachables, but I don't think I've ever seen that before in a Debug build, so I don't think it's likely. So, it seems to me like the kernel is returning an unexpected errno for an mmap which is replacing an existing mapping.

I tried to look for anything obviously wrong in the kernel sources, but didn't spot anything. So, I'm just going to push a commit which adds a std.debug.panic to the "unexpected errno" case of munmap (we unfortunately can't call std.posix.unexpectedErrno because std.posix.munmap is infallible), so that next time this triggers we can at least see what's being returned.

[mlugg]

923ddd9 adds the panic which will hopefully help to track this down.

[mlugg]

Assuming there isn't some deeper QEMU bug which causes part of the region to get genuinely unmapped (and I don't think there is?)

Oh, I should have more faith in myself! I literally found a bug that does exactly this in the patchset I sent to QEMU yesterday (see also #25520): https://lists.gnu.org/archive/html/qemu-devel/2025-10/msg02736.html

I'm not sure if that specific issue would have affected us (the trigger was very specific), but I absolutely would not be surprised if one of the mmap bugs fixed by this set solves this issue. The fact that we've only seen this on 32-bit targets under QEMU makes it seem somewhat likely that this is a QEMU bug. Assuming my patchset ultimately lands, we should see if this failure continues to occur after the runners upgrade to it...