Repository navigation
0.2.0 replaces the guard of 0.1.0. Each job now runs in its own cgroup scope
with kernel memory limits, and oomwrap stops it on the memory pressure that the
kernel reports, so it no longer needs fixed floors for each machine. This is a
breaking release. Old flags fail with an error, and 0.1.0 shims and wrappers
must be written again (see Upgrade from 0.1.0).
Added
- A
cgroupbackend. With--backend auto(the default), oomwrap runs the job
in a transient scope of the systemd manager withmemory.maxand
memory.high. A stop writescgroup.kill, which also ends children that left
the process group withsetsid(). Where the scope cannot run, such as in
a container or a CI job, oomwrap uses thepollbackend. - The memory domain. oomwrap reads the size, free room and pressure of the
memory that the job competes for: the machine, or the smallest limited cgroup
above the caller, such as a container. Every share is a share of that domain. - Defaults that follow the machine:
--memory-maxis the smallest of85%, the
free room at launch minus--min-mem, and any lower limit above the job;
--memory-highis90%of it;--min-memis15%, at least256M, at most
16G. A size can be a share (85%) orinfinity. - Pressure stops from the kernel's PSI figures:
--job-pressure(60%for
30s) and--domain-pressure(10%for10s, only while the domain has
less than a quarter of its memory free). The defaults come from recorded runs
on real machines. - Stop reasons in the events:
job_limit,outside_oom_kill,job_pressure,
domain_flooranddomain_pressure. - A policy file:
/etc/oomwrap.toml, then
${XDG_CONFIG_HOME:-~/.config}/oomwrap/config.toml, with a section for each
profile. A flag wins over the user file, which wins over the system file. --require-machine-guardand--allow-no-machine-guard. Besidesearlyoom,
systemd-oomdcounts as a machine-wide guard when it watches the job. The
policy file can name more guards inmachine-guards.--memory-swap-max,--backend,--log-command,--record-readings,
--shim, anddoctor --profile <name> [--json].- An event log by default, one file per day under
${XDG_STATE_HOME:-~/.local/state}/oomwrap/, kept for 14 days.
--event-log noneturns it off. Each event has aschema_version. doctorreports the backend thatautotakes and why, the memory domain and
each setting with its source. It also finds scopes that lost their guard and
outdated shims and wrappers, and prints the command that repairs each.- Job control: Ctrl-Z,
fgandbgwork on a guarded job. - The job's
oom_score_adjis1000, so a machine-wide killer picks the job
first. - Static musl binaries for
x86_64andaarch64on each GitHub release, with
SHA-256 checksums and build provenance attestations.
Changed
- When the command ends, oomwrap ends the processes that it left behind. With
thecgroupbackend this includes daemons that left the process group, such
as build servers and compiler caches. Start a daemon that must outlive the
job outside oomwrap. - An unknown flag before the command is an error. 0.1.0 took it as the start of
the command. --memory-maxand--memory-highset the kernel limits of the job's scope.
An explicit limit needs thecgroupbackend. Without that backend, oomwrap
exits 3 and runs nothing.- Exit 3 now also means that the required backend or an explicit limit is not
available, and exit 4 that the domain has no room for the job now. - Shims and wrappers carry no policy. They call
oomwrap run --shim <tool>, and
the policy file and the defaults decide the limits. - The events changed:
stoppedwith areasonreplacesmemory_pressure_kill,
interruptedreplacesparent_signal_kill,exitedreplaceschild_exit,
andcleanupreplacesprocess_group_cleanup. Events hold the program name
only, without its arguments, unless--log-commandis given. inspect --jsonholds the wholedoctorreport in place of itsdefault_*
fields.
Removed
--min-swapand the swap floor. Linux moves idle pages to swap while much
memory stays free, so the swap floor stopped healthy jobs. Low free room and
pressure now stop the job.--require-earlyoomand--allow-no-earlyoom, replaced by
--require-machine-guardand--allow-no-machine-guard.--use-systemd-scope, replaced by--backend cgroup.- The
--min-memand--min-swapflags ofinstall-shimsandwrap, and the
variablesOOMWRAP_MIN_MEM,OOMWRAP_MIN_SWAP,OOMWRAP_PROFILE,
OOMWRAP_ALLOW_NO_EARLYOOMandOOMWRAP_EVENT_LOGof the scripts. Put these
values in the policy file.OOMWRAP_BINandOOMWRAP_REAL_<TOOL>stay.
Upgrade from 0.1.0
-
Install 0.2.0 with
cargo install oomwrap --locked, or with the binary of
the release. -
Run
oomwrap doctorand do the repairs that it names. For thecgroup
backend, turn on lingering withloginctl enable-linger. -
Remove
--min-swapfrom scripts and commands. Replace--require-earlyoom
with--require-machine-guard,--allow-no-earlyoomwith
--allow-no-machine-guard, and--use-systemd-scopewith
--backend cgroup. -
Check each
--min-mem. A fixed value such as24Gstill works, but the
default now follows the machine. Remove it, or move it into the policy file. -
Move the values that shims and wrappers took from
OOMWRAP_MIN_MEM,
OOMWRAP_MIN_SWAP,OOMWRAP_PROFILE,OOMWRAP_ALLOW_NO_EARLYOOMand
OOMWRAP_EVENT_LOGinto the policy file, for example:min-mem = "8G" [profile.vllm] require-machine-guard = false
-
Write the shims again with
oomwrap install-shims, with the--bin-dirand
--toolvalues that you used before. Then runoomwrap wrap <path>again
for each wrapped program. Both commands rewrite the 0.1.0 scripts in place.
Until then, the 0.1.0 scripts fail with exit 2. -
Update tools that read the event log to the new event names and fields.
-
Run
oomwrap doctoragain. It must name no outdated shim or wrapper.
Binaries
Static binaries for x86_64 and aarch64 Linux are attached below, with SHA256SUMS. Check one with:
sha256sum --check --ignore-missing SHA256SUMS
gh attestation verify oomwrap-0.2.0-<arch>-linux-musl --repo osolmaz/oomwrap