fix(threads): rayon multithread verification + cgroup-aware thread cap (#263)

[d-laub]

Summary

Makes gvl's rayon-parallel paths runnable and verified under real multithreaded load, fixes the cap_threads() oversubscription bugs behind #263, and releases the GIL around the rayon FFI so parallel workers no longer serialize on it or oversubscribe/park.

Closes #263.

What changed

Thread-count resolver (python/genvarloader/_threads.py)

• GVL_FORCE_PARALLEL env knob bypasses the size gate so the multithreaded paths run on small inputs (tests, repro harnesses).
• CFS-cpu-quota-aware detection: reads cgroup v2 cpu.max (falling back to v1 cpu.cfs_quota_us/cpu.cfs_period_us). A quota is invisible to sched_getaffinity, so a 15.3-core container previously reported the full host core count and oversubscribed. Detection now takes min(affinity, quota).
• cap_threads() now overwrites RAYON_NUM_THREADS with GVL's resolved cap (was setdefault).

FFI (src/ffi/mod.rs)

• Releases the GIL around all 11 rayon parallel entry points via py.detach(...) (PyO3 0.28's rename of allow_threads). Every PyReadonly*/PyReadwrite* guard is resolved to an ndarray view before the closure; only Ungil views + POD are captured; every into_pyarray runs after. Byte-identical serial==parallel==golden parity is preserved.

Tests

• tests/integration/test_rayon_forced_parallel.py — forced-parallel dataset[:, :] is byte-identical to the serial path end-to-end (compares both ragged .data and .offsets).
• tests/integration/test_rayon_stress.py (slow) — repeated spawn-worker waves, oversubscribed rayon, with a per-launch timeout as the deadlock detector.
• tests/parity/* — marked the pytest.skip except branches NoReturn (trailing raise) to clear pre-existing pyrefly unbound-name errors, unblocking the project-wide pre-commit pyrefly hook.

⚠️ Behavior change — RAYON_NUM_THREADS is now overwritten

cap_threads() previously respected an externally-set RAYON_NUM_THREADS; it now clobbers it with GVL's cgroup-derived cap. This is the point of the fix — an inherited value (e.g. from a base image) must not defeat the cap and cause oversubscription (#263).

Escape hatch: users who want an explicit worker count should set GVL_NUM_THREADS (which becomes the resolved cap and is written to RAYON_NUM_THREADS). Setting a bare RAYON_NUM_THREADS alone no longer wins.

Root cause / Task 8

The spawn-worker stress reproducer completed all launches cleanly (no deadlock) once the oversubscription fixes + GIL release were in place — so the contingent root-cause task was not needed. The #263 hang was driven by per-worker oversubscription, addressed by the cgroup-aware cap + unconditional RAYON_NUM_THREADS overwrite + GIL release.

Verification

• cargo-test: 110 + 4 passed; cargo build --release: clean.
• test_rayon_equivalence.py (byte-identical parity): 5 passed.
• Full tree (pytest tests -q, slow excluded): 945 passed, 54 skipped, 4 xfailed.
• Slow tier (stress reproducer): 1 passed.
• pyrefly: 0 errors; ruff check/ruff format --check: clean. Pre-commit hooks pass without --no-verify.

No public API / __all__ / SKILL.md change (env-only knobs), so no skill update needed.

Note: this branch also carries the initiative's spec and plan docs (docs/superpowers/{specs,plans}/2026-06-30-rayon-multithread-verification*), which were committed to local main ahead of origin.

🤖 Generated with Claude Code