repro: reproducer and root cause for #37 (free(): invalid pointer in the CPU profiler)

[korniltsev-grafanista-yolo-vibecoder239]

Draft — investigation results for #37, not a fix.

Summary

free(): invalid pointer / SIGSEGV in workers running the agent, reported
against 0.8.11/0.8.14 (celery prefork) and again against 1.2.1 (gunicorn sync
workers, Python 3.11). Both reporters' stacks are the same, and reproduce here:

#4  __GI___libc_free (mem=0x18)
#5  anyhow::error::object_drop ()   from .../pyroscope/_native...so
#6  std::sys::backtrace::__rust_begin_short_backtrace ()
#7  core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#8  std::sys::pal::unix::thread::Thread::new::thread_start ()

It is not an ABI mismatch (the issue's original hypothesis), not memory
profiling (predates it; everything here is CPU-only), and not corruption of
pyroscope's own bookkeeping — the heap is intact right up to the bogus free().

Root cause

remoteprocess::ProcessMemory::copy_struct materializes a T out of bytes
read from the profiled process:

fn copy_struct<T: Copy>(&self, addr: usize) -> Result<T, Error> {
    let mut data = vec![0; std::mem::size_of::<T>()];
    self.read(addr, &mut data)?;
    Ok(unsafe { std::ptr::read(data.as_ptr() as *const _) })
}

T: Copy does not mean every bit pattern is a valid T. On Python 3.11
py-spy's _PyInterpreterFrame has pub is_entry: bool at offset 68 — the only
niche in the struct — and rustc uses it as the discriminant of the returned
Result, because there is no room for a tag:

sizeof(Result<v3_11_0::_PyInterpreterFrame, remoteprocess::Error>) == 80
sizeof(v3_11_0::_PyInterpreterFrame)                               == 80

Confirmed by disassembling the shipped wheel
(pyroscope-io==1.2.1, _native.cpython-311-x86_64-linux-gnu.so):

0000000000900f80 <remoteprocess::ProcessMemory::copy_pointer::hfa7f1a9f30c1a49b>:
  900fbb: mov    $0x50,%r8d          ; len = 80 = sizeof(_PyInterpreterFrame)
  900fca: call   *0x29c460(%rip)     ; Process::read(addr, &mut data)
  900fd0: cmpl   $0xf,0x8(%rsp)      ; read() ok?
  900fd5: jne    901004              ;   no -> Err path
  900fd7..900fff: movups x5          ; Ok: copy all 80 target bytes verbatim,
                                     ;     byte 0x44 included, unvalidated
  901004..901012: movups x2          ; Err: 32-byte remoteprocess::Error into [0..32]
  901015: movb   $0x2,0x44(%rbx)     ; <<<< discriminant := 2, stored in `is_entry`

and the caller testing that same byte, in py_spy::stack_trace::get_stack_trace:

  904232: lea    0x100(%rsp),%rdi    ; sret slot for the Result
  904245: call   900f80 <...copy_pointer...>
  90424a: movzbl 0x144(%rsp),%eax    ; ONE byte at slot+0x44 (= 68 = is_entry)
  904252: cmp    $0x2,%al            ; the Err niche value
  904254: je     904be1              ;   -> error path
  90425a: mov    %al,0xb(%rsp)       ;   else keep it as the bool

For contrast, the next call in the same function copies a PyCodeObject
(no niche, 184 bytes) and gets a real tag word, so it cannot be confused:

  90428c: call   8fffd0 <...copy_pointer::h194f4e6da92eb3c4>
  904291: cmpl   $0x1,0x100(%rsp)

So when the sampler reads a stale frame whose byte 68 is 2, copy_pointer
silently returns Err(remoteprocess::Error) whose payload is the 32
interpreter bytes it just read. py-spy contexts it
("Failed to copy PyFrameObject") into Sample.sampling_errors, and when
pyroscope's consumer thread drops the Sample,
drop_glue<remoteprocess::Error> reads the first word as the enum
discriminant. A CPython object starts with ob_refcnt — a small integer —
which selects a String-owning variant, and the following words are freed as
a heap pointer. That is free(mem=0x18).

Evidence

Four core dumps, three organic and one from the deterministic probe; in every
one the dropped error's payload is a recognizable CPython object:

core	error payload	free() arg
wheel 1.2.1	{3, &PyCell_Type, 0x55d19f5c6250}	the cell's ob_ref
wheel 1.2.1	{3, <type>, 0x564fb95e91d0}	ditto
guard build	{3, &PyList_Type, 41, ob_item}	0x29 = 41 = that list's ob_size

With a guarding global allocator the ErrorImpl allocation is live and
intact (magic + correct size, no invalid/double free reported), and its
sibling field is a valid &'static str (len = 0x1c = 28 =
"Failed to copy PyFrameObject"). Only the remoteprocess::Error half is
CPython bytes — the Ok payload was read as an Err.

Typed frames from the debug build:

#5  _native::debug_alloc::dealloc (ptr=0x29, layout=...)
#6  core::ptr::drop_glue<remoteprocess::Error> ()
#7  core::ptr::drop_glue<anyhow::error::ContextError<&str, remoteprocess::Error>> ()
#19 core::ptr::drop_glue<(i32, anyhow::Error)> ()
#23 alloc::vec::{impl#27}::drop<(i32, anyhow::Error), alloc::alloc::Global> ()
#31 std::panicking::catch_unwind<..., _native::pyspy_backend::{impl#1}::initialize::{closure_env#0}>

Why only Python 3.11, and why it is rare

is_entry: bool exists only in the 3.11 bindings; 3.12+ use c_char there.
The frame struct is copied for every frame of every sample, chasing pointers
that go stale constantly, so 3.11 is where this bites — matching both reports
(3.11.14) and matching Python 3.13 never crashing here in 3.3+ process-hours
of the same workload. The remaining niches on 3.12+ sit in interpreter-state
structs that are copied far less often.

Two conditions must coincide in one read: byte 68 must be exactly 2
(rustc picked a single niche value, ~1/256), and the first word must be a
small integer for the drop to select a String variant — usually it is a
pointer, and the bogus error is silently harmless. Hence: bogus sampling
errors are routine, fatal frees are occasional.

Reproducers (in repro37/)

• Deterministic — poc_niche_confusion.py parks a thread and points its
  frame's previous at a private malloc'd buffer with byte 68 = 2 and a
  String pointer of 0x29. Nothing in the interpreter is corrupted; CPython
  never reads the buffer.

  $ python poc_niche_confusion.py        # 3.11 + pyroscope-io 1.2.1
  Segmentation fault                     # exit 139, seconds, 3/3 runs
  $ NO_PROFILER=1 python poc_niche_confusion.py
  survived                               # exit 0 — only the profiler frees it
  

• Organic — workload.py + runner.py, no ctypes: deep stacks, code
  objects churned so sampled addresses go stale, freed memory reused with
  pointer-filled data, thread churn. Python 3.11 + unmodified 1.2.1 wheel, CPU
  profiler only, PYTHONMALLOC=malloc, 8–10 workers → ~1 crash per 1–3
  process-hours. PYTHONMALLOC=malloc is an amplifier, not the cause: it makes
  stale reads land on live CPython objects with small refcounts more often.

Fix directions (not included here)

1. Remove the niche — emit u8/c_char instead of bool in py-spy's
   bindings. Result<T, Error> then gets a real tag and the confusion becomes
   impossible; is_entry is only used as a truthy flag.
2. Tighten copy_struct's bound so only types valid for every bit pattern can
   be read out of another process (e.g. bytemuck::AnyBitPattern instead of
   T: Copy). This is the real fix — today any future binding with a bool,
   char, enum, reference or NonNull reintroduces the bug.

Both live upstream in py-spy / remoteprocess, so pyroscope would need a
patched pin in rust/Cargo.toml.

What is in this PR

• repro37/ROOT_CAUSE.md — the full write-up: disassembly, decoded cores,
  per-version niche table, rarity analysis, ruled-out hypotheses, tooling.
• repro37/ — the reproducers, Docker images (3.11 + 1.2.1, debug-info, ASAN,
  valgrind, guarding-allocator), ingest sinks, gunicorn/fork variants,
  analyze-core.sh.
• rust/src/debug_alloc.rs — the guarding global allocator used to prove the
  error object was not corrupted. Behind the non-default debug-alloc
  feature; cargo check passes with and without it. Droppable if unwanted.

Core dumps are gitignored; ROOT_CAUSE.md cites them by path with the decoded
contents inline.

Also noticed while looking (separate, not the cause of this issue):
py-spy's copy_string builds Rust chars out of target bytes via
from_raw_parts(... as *const char), which is UB for non-scalar values and
yields Strings holding invalid UTF-8 (poc_hostile_string.py exercises it);
and shorten_filename's short_filenames cache is unbounded.

🤖 Generated with Claude Code

[github-actions]

Signed commits report

3 of 3 commits between main and reproduce-issue-37-crash could not be fully verified:

Commit	Author	Reason	Message
be223ed2	Tolyan Korniltsev	unsigned	repro: add a reproducer and root-cause analysis for #37
5fb13e27	Tolyan Korniltsev	unsigned	docs: explain why c_char is not equivalent to bool for #37
0e32ab80	Tolyan Korniltsev	unsigned	docs: note that #[repr(C)] does not prevent the niche (#37)

This repository requires all commits to be signed. See GitHub docs on commit signature verification.

[korniltsev-grafanista-yolo-vibecoder239]

just a repro, nothing to merge