Skip to content

5.0.0

Latest

Choose a tag to compare

@benoitc benoitc released this 29 Aug 19:22
9458981

Major release: the legacy worker API is removed; contexts (py_context, py:call/3) are the only execution path. See Removed below for the full list of removed functions.

Added

  • isolated context mode - py_context:new(#{mode => isolated}) runs
    CPython in a child OS process per context, with the same call/eval/exec,
    callback, erlang.send/whereis, worker-loop and pool API as the embedded
    modes. It is the first mode with a hard bound: py_context:interrupt/1
    stops a blocking C call (a signal in the child) and SIGKILL is the
    backstop after kill_after ms; py_context:kill/1 kills at once. rlimits
    (as, cpu, nofile) and a cgroup v2 directory bound the child; a
    segfault in a C extension returns {error, {child_exited, {signal, 11}}}
    and the node survives. The child restarts on crash within a budget
    (restart, max_restarts, restart_period); py_context:child_info/1
    reports its OS pid. Children are reaped by the VM and exit when the BEAM
    dies (socket EOF watchdog, PR_SET_PDEATHSIG on Linux, PROC_PDEATHSIG_CTL
    on FreeBSD). cgroup is refused outside Linux; rlimits apply everywhere:
    as is kernel-enforced on Linux and FreeBSD and enforced by an RSS
    watchdog in the child on macOS ({child_exited, {memory_limit, Bytes}}).
    Validated on macOS (arm64) and FreeBSD 14.3 (OTP 28, Python 3.11).
  • py_context:pass_fd/2 - hands a file descriptor to an isolated child
    over the control socket (SCM_RIGHTS), so erlang.server.serve works out
    of process: Erlang binds once, N killable children accept.
  • Pure-Python ETF codec (priv/_erlang_impl/_etf.py) with the type
    mapping of py_convert.c; the child needs no C extension. Integers beyond
    64 bits round-trip exactly in isolated mode.
  • Shared memory - py_shm:new/1,2, write/3, read/3, binary/3
    (no copy), close/1: fixed-size regions over
    iommap (optional dependency) that any
    context mode maps as erlang.SharedMemory (buffer protocol, numpy
    friendly). py_buffer:new(#{shared => true}) is a streaming buffer over
    such a region with ring backpressure, usable as wsgi.input in isolated
    contexts. Handles are plain terms and travel inside any argument or result;
    py_shm:read_only/1 and new(Size, #{writable => false}) hand Python a
    read-only mapping. py_buffer:write/3 takes a timeout for the case where
    the ring is full and nobody reads (default 30 s).
  • py:python_executable/0, py:kill/1, py_nif:os_kill/2.
  • py_isolated is a gen_statem (states idle, {busy, Id}, looping,
    stopping_loop, {restarting, Reason}): sys:get_state/1 and
    sys:trace/2 work on isolated contexts, requests arriving during a
    restart are served by the new child, and py_context:kill/1 returns once
    the new child is up.
  • Timeouts on an isolated context cancel their own request only (queued
    requests are dropped, the executing one is interrupted); the kill backstop
    is bound to that request, so a busy shared context is never killed because
    another caller gave up. Soak-tested: callback storms, interrupt/kill
    storms, loop churn, 60 s mixed workload with resource counters checked.
  • Guide: docs/isolated.md, with what each of the three modes guarantees.

Changed

  • The NIF side of every context request goes through one dispatcher
    (ctx_dispatch, ctx_dispatch_async in c_src/py_nif.c) instead of a
    per-request copy of the enqueue-and-wait loop; the execute functions are
    ctx_execute_* and the thread functions ctx_thread_main_*, since both
    serve worker and owngil contexts. Creating a process-local env and
    applying imports or paths run on the context thread in worker mode too;
    the scheduler-side copies of those paths are gone.
  • The NIF function table is assembled from one PY_*_NIFS macro per area,
    defined at the end of the file that owns the NIFs.
  • py_context keeps the API and the reply protocol; the process body for
    embedded modes moved to py_context_embedded. py delegates streaming,
    virtual environments and shared dicts to py_stream, py_venv and
    py_shared_dict. The public API is unchanged.

Documentation

  • make check-code-map (also run by CI) verifies that every source file is
    in docs/code-map.md, every Erlang module has a moduledoc and a row in
    the Modules table of test/coverage_audit.md.

Removed

  • The legacy worker API (py_nif:worker_new/0,1, worker_call, worker_eval,
    worker_exec, worker_next, worker_destroy, import_module/2,
    get_attr/3, set_callback_handler/2, send_callback_response/2,
    resume_callback/2) and the single executor thread behind it, the
    async_worker_*/async_call/async_gather/async_stream NIFs that only
    returned deprecated, the unused worker pool (pool_* NIFs), the
    cancel_reader/writer aliases, and the unreachable inline executor
    branches of the context NIFs. Contexts (py_context, py:call/3) are the
    only execution path. py:memory_stats/0 and py:gc/0,1 now run on the
    calling scheduler under the GIL.

Fixed

  • pthread_timedjoin_np was called without _GNU_SOURCE, an implicit
    declaration on Linux that newer compilers reject.
  • Callback pipes waited with select(), which is undefined for a file
    descriptor above 1024: in a VM with many open files a thread callback
    could time out with "Failed to spawn thread handler". The waits use
    poll(), the handler ready-wait no longer holds the GIL, and
    py_thread_handler logs a failed ready signal.