v2.3.0: Memory and CPU limits for daemon processes
Pitchfork v2.3.0 adds per-daemon memory and CPU resource limits. The supervisor periodically monitors each daemon's resource usage and automatically kills processes that exceed their configured thresholds, preventing runaway daemons from consuming all available system resources.
Added
-
memory_limitconfiguration option -- Set a maximum RSS (physical memory) for any daemon. The supervisor checks memory at each interval tick (default10s) and immediately kills the process group viaSIGTERMif it exceeds the limit. Memory is aggregated across the root process and all its descendants, so multi-process daemons (e.g. gunicorn workers) are measured correctly. The daemon is marked asErrored, allowing retry logic to kick in if configured. Accepts human-readable sizes with both SI and binary units. (#294) - @gaojunran[daemons.worker] run = "python worker.py" memory_limit = "512MB" [daemons.api] run = "node server.js" memory_limit = "2GiB" retry = 3 # Restart if killed for exceeding limit
-
cpu_limitconfiguration option -- Set a maximum CPU usage as a percentage of one core. To avoid killing daemons during transient spikes (JIT warm-up, burst responses), the supervisor requires 3 consecutive over-limit samples before terminating the process. A single sample below the limit resets the counter. Likememory_limit, CPU usage is aggregated across the entire process group. Values above 100 are valid on multi-core systems. (#294) - @gaojunran[daemons.worker] run = "python compute.py" cpu_limit = 80 # 80% of one CPU core [daemons.batch] run = "./run-batch.sh" cpu_limit = 200 # Up to 2 CPU cores
-
settings.supervisor.cpu_violation_threshold-- Controls how many consecutive CPU-over-limit samples are required before killing a daemon. Default is3. Set to1for immediate enforcement, or increase for more tolerance of short bursts. Also configurable via thePITCHFORK_CPU_VIOLATION_THRESHOLDenvironment variable. (#294) - @gaojunran[settings.supervisor] cpu_violation_threshold = 5 # More tolerant of short CPU bursts
Full Changelog: v2.2.0...v2.3.0