Skip to content

v2.3.0: Memory and CPU limits for daemon processes

Choose a tag to compare

@jdx jdx released this 31 Mar 12:27
· 181 commits to main since this release
Immutable release. Only release title and notes can be modified.
cd85cee

Pitchfork v2.3.0 adds per-daemon memory and CPU resource limits. The supervisor periodically monitors each daemon's resource usage and automatically kills processes that exceed their configured thresholds, preventing runaway daemons from consuming all available system resources.

Added

  • memory_limit configuration option -- Set a maximum RSS (physical memory) for any daemon. The supervisor checks memory at each interval tick (default 10s) and immediately kills the process group via SIGTERM if it exceeds the limit. Memory is aggregated across the root process and all its descendants, so multi-process daemons (e.g. gunicorn workers) are measured correctly. The daemon is marked as Errored, allowing retry logic to kick in if configured. Accepts human-readable sizes with both SI and binary units. (#294) - @gaojunran

    [daemons.worker]
    run = "python worker.py"
    memory_limit = "512MB"
    
    [daemons.api]
    run = "node server.js"
    memory_limit = "2GiB"
    retry = 3  # Restart if killed for exceeding limit
  • cpu_limit configuration option -- Set a maximum CPU usage as a percentage of one core. To avoid killing daemons during transient spikes (JIT warm-up, burst responses), the supervisor requires 3 consecutive over-limit samples before terminating the process. A single sample below the limit resets the counter. Like memory_limit, CPU usage is aggregated across the entire process group. Values above 100 are valid on multi-core systems. (#294) - @gaojunran

    [daemons.worker]
    run = "python compute.py"
    cpu_limit = 80     # 80% of one CPU core
    
    [daemons.batch]
    run = "./run-batch.sh"
    cpu_limit = 200    # Up to 2 CPU cores
  • settings.supervisor.cpu_violation_threshold -- Controls how many consecutive CPU-over-limit samples are required before killing a daemon. Default is 3. Set to 1 for immediate enforcement, or increase for more tolerance of short bursts. Also configurable via the PITCHFORK_CPU_VIOLATION_THRESHOLD environment variable. (#294) - @gaojunran

    [settings.supervisor]
    cpu_violation_threshold = 5  # More tolerant of short CPU bursts

Full Changelog: v2.2.0...v2.3.0