Major release: the legacy worker API is removed; contexts (py_context, py:call/3) are the only execution path. See Removed below for the full list of removed functions.
Added
isolatedcontext mode -py_context:new(#{mode => isolated})runs
CPython in a child OS process per context, with the samecall/eval/exec,
callback,erlang.send/whereis, worker-loop and pool API as the embedded
modes. It is the first mode with a hard bound:py_context:interrupt/1
stops a blocking C call (a signal in the child) andSIGKILLis the
backstop afterkill_afterms;py_context:kill/1kills at once.rlimits
(as,cpu,nofile) and a cgroup v2 directory bound the child; a
segfault in a C extension returns{error, {child_exited, {signal, 11}}}
and the node survives. The child restarts on crash within a budget
(restart,max_restarts,restart_period);py_context:child_info/1
reports its OS pid. Children are reaped by the VM and exit when the BEAM
dies (socket EOF watchdog,PR_SET_PDEATHSIGon Linux,PROC_PDEATHSIG_CTL
on FreeBSD).cgroupis refused outside Linux; rlimits apply everywhere:
asis kernel-enforced on Linux and FreeBSD and enforced by an RSS
watchdog in the child on macOS ({child_exited, {memory_limit, Bytes}}).
Validated on macOS (arm64) and FreeBSD 14.3 (OTP 28, Python 3.11).py_context:pass_fd/2- hands a file descriptor to an isolated child
over the control socket (SCM_RIGHTS), soerlang.server.serveworks out
of process: Erlang binds once, N killable children accept.- Pure-Python ETF codec (
priv/_erlang_impl/_etf.py) with the type
mapping ofpy_convert.c; the child needs no C extension. Integers beyond
64 bits round-trip exactly in isolated mode. - Shared memory -
py_shm:new/1,2,write/3,read/3,binary/3
(no copy),close/1: fixed-size regions over
iommap (optional dependency) that any
context mode maps aserlang.SharedMemory(buffer protocol, numpy
friendly).py_buffer:new(#{shared => true})is a streaming buffer over
such a region with ring backpressure, usable aswsgi.inputin isolated
contexts. Handles are plain terms and travel inside any argument or result;
py_shm:read_only/1andnew(Size, #{writable => false})hand Python a
read-only mapping.py_buffer:write/3takes a timeout for the case where
the ring is full and nobody reads (default 30 s). py:python_executable/0,py:kill/1,py_nif:os_kill/2.py_isolatedis agen_statem(statesidle,{busy, Id},looping,
stopping_loop,{restarting, Reason}):sys:get_state/1and
sys:trace/2work on isolated contexts, requests arriving during a
restart are served by the new child, andpy_context:kill/1returns once
the new child is up.- Timeouts on an isolated context cancel their own request only (queued
requests are dropped, the executing one is interrupted); the kill backstop
is bound to that request, so a busy shared context is never killed because
another caller gave up. Soak-tested: callback storms, interrupt/kill
storms, loop churn, 60 s mixed workload with resource counters checked. - Guide:
docs/isolated.md, with what each of the three modes guarantees.
Changed
- The NIF side of every context request goes through one dispatcher
(ctx_dispatch,ctx_dispatch_asyncinc_src/py_nif.c) instead of a
per-request copy of the enqueue-and-wait loop; the execute functions are
ctx_execute_*and the thread functionsctx_thread_main_*, since both
serve worker and owngil contexts. Creating a process-local env and
applying imports or paths run on the context thread inworkermode too;
the scheduler-side copies of those paths are gone. - The NIF function table is assembled from one
PY_*_NIFSmacro per area,
defined at the end of the file that owns the NIFs. py_contextkeeps the API and the reply protocol; the process body for
embedded modes moved topy_context_embedded.pydelegates streaming,
virtual environments and shared dicts topy_stream,py_venvand
py_shared_dict. The public API is unchanged.
Documentation
make check-code-map(also run by CI) verifies that every source file is
indocs/code-map.md, every Erlang module has a moduledoc and a row in
the Modules table oftest/coverage_audit.md.
Removed
- The legacy worker API (
py_nif:worker_new/0,1,worker_call,worker_eval,
worker_exec,worker_next,worker_destroy,import_module/2,
get_attr/3,set_callback_handler/2,send_callback_response/2,
resume_callback/2) and the single executor thread behind it, the
async_worker_*/async_call/async_gather/async_streamNIFs that only
returneddeprecated, the unused worker pool (pool_*NIFs), the
cancel_reader/writeraliases, and the unreachable inline executor
branches of the context NIFs. Contexts (py_context,py:call/3) are the
only execution path.py:memory_stats/0andpy:gc/0,1now run on the
calling scheduler under the GIL.
Fixed
pthread_timedjoin_npwas called without_GNU_SOURCE, an implicit
declaration on Linux that newer compilers reject.- Callback pipes waited with
select(), which is undefined for a file
descriptor above 1024: in a VM with many open files a thread callback
could time out with "Failed to spawn thread handler". The waits use
poll(), the handler ready-wait no longer holds the GIL, and
py_thread_handlerlogs a failed ready signal.