Skip to content

Intermittent SIGSEGV / silent value corruption in _PyEval_EvalFrameDefault on 3.14.6 (NULL _PyStackRef deref at specialized CALL type guard) #155145

Description

@derekustruck

Bug description

Under repeated execution, _PyEval_EvalFrameDefault intermittently
dereferences a NULL _PyStackRef-tagged pointer at the +8 offset (ob_type
on an untagged PyObject*), producing SIGSEGV. In non-crashing occurrences
of what looks like the same underlying defect, a local variable is instead
observed holding the string value of a sibling parameter's name from the
same call (e.g. a float/int argument slot holding the literal string
"n" or "successes"), which then raises a spurious TypeError or
ValueError from application code that would otherwise never see a non-numeric
value there.

I could not find an existing matching report after searching the tracker for
combinations of specializer, freelist, CALL_KW, math.comb,
_PyStackRef, and the crash addresses below. Apologies in advance if this
turns out to be a duplicate — happy to have it closed and pointed at the
right issue.

Reproducer

Pure standard library, no threads, no C extensions, deterministic inputs
(only math.comb, float **, and a couple of keyword-only-argument calls
in a loop):

import math


def _require_integer(name, value, *, minimum):
    if isinstance(value, bool) or not isinstance(value, int):
        raise ValueError(f"{name} must be an integer")
    result = int(value)
    if result < minimum:
        raise ValueError(f"{name} must be at least {minimum}")
    return result


def binomial_probability_mass(n, successes, probability):
    trials = _require_integer("n", n, minimum=0)
    count = _require_integer("successes", successes, minimum=0)
    if count > trials:
        return 0.0
    return float(
        math.comb(trials, count)
        * probability**count
        * (1.0 - probability) ** (trials - count)
    )


def binomial_upper_tail(n, threshold, probability):
    trials = _require_integer("n", n, minimum=0)
    total = 0.0
    for value in range(threshold, trials + 1):
        total += binomial_probability_mass(trials, value, probability)
    return total


for i in range(2_000_000):
    binomial_upper_tail(256, (i % 200) + 1, 0.1)
print("completed without incident")

Run repeatedly (for i in $(seq 1 10); do python3.14 repro.py; echo exit=$?; done).
On this machine it fails (segfault, or a spurious TypeError/ValueError
from inside _require_integer/binomial_probability_mass as shown below)
within anywhere from the first invocation up to a few hundred thousand
iterations. An identical script never failed across 600,000+ cumulative
iterations run this way on CPython 3.12.3 on the same machine, same day.

Non-crashing corruption observed (same reproducer, different runs)

ValueError: n must be an integer
  File ".../repro.py", line 8, in _require_integer
    raise ValueError(f"{name} must be an integer")

— here value (bound to the second positional argument, which should be
an int such as 256) was observed to be the string "n", i.e. the
literal first-argument string from the same call.

TypeError: unsupported operand type(s) for -: 'str' and 'float'
  File ".../repro.py", line 26, in binomial_probability_mass
    * (1.0 - probability) ** (trials - count)

— here chance/probability had already been validated as a float a few
lines earlier in the same call, then read back as a str.

TypeError: 'cell' object is not callable
  File ".../repro.py", line 15, in binomial_probability_mass
    trials = _require_integer("n", n, minimum=0)

— here the function object bound to the name _require_integer at the
call site resolved to a cell object instead of a function.

I was not able to make the failure mode deterministic; the same script,
same inputs, same machine, with PYTHONHASHSEED fixed to a constant value,
still produces different outcomes (clean completion vs. one of the errors
above vs. SIGSEGV) across repeated invocations.

Crash evidence

Four independent SIGSEGV occurrences were captured via the kernel crash log
(WSL2's crash capture), at four different addresses, all inside
_PyEval_EvalFrameDefault, all faulting at address 0x8 (NULL + 8):

python[42842]: segfault at 8 ip 0000000000578726 sp 00007fffbb7d48b0 error 4 in python3.14[422000+333000]
Code: ... 4c 89 e8 48 83 e0 fe <48> 8b 40 08 39 88 80 01 00 00 ...

python[44518]: segfault at 8 ip 0000000000578dcd sp 00007ffc3088b730 error 4 in python3.14[422000+333000]
Code: ... 49 83 e0 fe <49> 8b 40 08 48 3d a0 aa a1 00 0f 84 70 fb ff ff ...

python[45274]: segfault at 8 ip 000000000057d4e0 sp 00007ffc998db3e0 error 4 in python3.14[422000+333000]
Code: ... 48 83 e2 fe <48> 81 7a 08 e0 76 a1 00 0f 85 90 46 00 00 ...

python[47854]: segfault at 8 ip 00000000005789ab sp 00007fff32c121a0 error 4 in python3.14[422000+333000]
Code: ... 49 83 e0 fe 66 41 89 42 fa <49> 81 78 08 a0 aa a1 00 0f 84 90 a8 00 00 ...

Disassembling each address against the installed /usr/bin/python3.14
(objdump -d) shows the same instruction shape at all four sites:

and $0xfffffffffffffffe, %reg      ; clear the low tag bit of a _PyStackRef
mov 0x8(%reg), %reg                ; load ob_type from the untagged pointer  <-- faults here when %reg == 0
cmp $<cached type constant>, %reg  ; specialized inline type guard
je  <specialized fast path>

e.g. at 0x578dcd:

578dc9: 49 83 e0 fe          and    $0xfffffffffffffffe,%r8
578dcd: 49 8b 40 08          mov    0x8(%r8),%rax          ; SIGSEGV: %r8 == 0
578dd1: 48 3d a0 aa a1 00    cmp    $0xa1aaa0,%rax
578dd7: 0f 84 70 fb ff ff    je     57894d <_PyEval_EvalFrameDefault@@Base+0xbdd>

and at 0x57d4e0:

57d4dc: 48 83 e2 fe          and    $0xfffffffffffffffe,%rdx
57d4e0: 48 81 7a 08 e0 76 a1 cmpq   $0xa176e0,0x8(%rdx)     ; SIGSEGV: %rdx == 0
        00

All four crash sites: untag a _PyStackRef (the and $-2 clearing the
low bit), then immediately dereference +0x8 for the specialized inline
type guard used by the adaptive specializing interpreter. Every occurrence
faults at address exactly 0x8, i.e. the untagged pointer was NULL in
every case rather than a valid (if stale/wrong-type) object — consistent
with a _PyStackRef stack slot holding a bare tag value (e.g. 1) instead
of a tagged object pointer at the point a specialized CALL-family
instruction reads it.

I don't have a gdb/debug build available on this machine to go further
(no root); happy to run a debug/ASan build reproducer or provide a
faulthandler-enabled Python-level traceback if that's more useful than the
raw disassembly.

Environment

$ python3.14 -VV
Python 3.14.6 (main, Jun 11 2026, 12:32:48) [GCC 13.3.0]

$ dpkg -l python3.14 python3.14-venv
ii  python3.14      3.14.6-1+noble1 amd64
ii  python3.14-venv 3.14.6-1+noble1 amd64
# installed from the deadsnakes PPA (ppa:deadsnakes/ppa) on Ubuntu 24.04

$ python3.14 -c "import sysconfig; print(sysconfig.get_config_var('Py_GIL_DISABLED'))"
0   # not a free-threaded build

$ uname -a
Linux (hostname redacted) 6.6.87.2-microsoft-standard-WSL2 #1 SMP PREEMPT_DYNAMIC Thu Jun 5 18:30:46 UTC 2025 x86_64 GNU/Linux
# WSL2 on Windows, Intel Core i9-14900K, Ubuntu 24.04.4 LTS (noble)

Not a free-threaded build, not under any memory-allocation-failure
injection, single-threaded pure-Python reproducer, no third-party C
extensions loaded.

Why I think this is real and not environment noise

  • The four crash addresses disassemble to the same instruction pattern
    (untag a _PyStackRef, load ob_type at +8, compare against a cached
    type constant) even though they're different call sites in
    _PyEval_EvalFrameDefault.
  • Every SIGSEGV faults at exactly address 0x8 — consistent with a
    NULL-valued (fully untagged-to-zero) stack slot, not a random wild
    pointer, which argues against generic hardware/RAM corruption.
  • An effectively identical reproducer never failed across 600,000+
    iterations on CPython 3.12.3, same machine, same day, ruling out
    hardware.
  • Fixing PYTHONHASHSEED to a constant value does not make the failure
    deterministic, which argues against ordinary hash-order-dependent
    behavior and toward something timing/allocation-pattern dependent (e.g.
    specialization kicking in at a slightly different call count run to run).

Thanks for looking at this — let me know what additional diagnostics would
help (I can rebuild with a debug/ASan interpreter if that's the most useful
next step, just don't have sudo/root on this particular machine for the
system package manager, but can build from source under --prefix).

Metadata

Metadata

Assignees

No one assigned

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)type-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions