Skip to content

v0.20.0

Choose a tag to compare

@github-actions github-actions released this 24 May 07:13
· 120 commits to main since this release

Added — memory guard prevents OOM-kill of the whole worker

A pathological PDF once made the worker consume ~13 GB of RSS, which triggered the kernel OOM-killer; the worker died without a graceful exit, lost in-flight state, and required manual recovery. Two layers of defence now:

  • RLIMIT_AS cap at startup (default 8 GB, configurable via RTFM_WORKER_MEMORY_LIMIT_GB). The next allocation past the cap raises MemoryError — catchable by the per-job handler, which marks the job failed and moves on. Converts a kernel SIGKILL into a normal Python exception.
  • RSS polling at every idle tick. Above WORKER_RSS_EXIT_MB (5 GB) the worker exits cleanly; the next hook respawns a fresh process. Catches slow leaks that wouldn't trip the per-alloc cap.

Opt out with RTFM_WORKER_MEMORY_LIMIT_GB=0 when running marker-pdf, whose ML models legitimately need 3-8 GB.

Test suite: 568 passed.