python: cut Klipper/Moonraker interpreter memory overhead - #271
Conversation
✅ Build ArtifactsBranch:
|
Benchmarked on carbon2u (CC1, 128 MiB), 5 reboots per variant, 120 s settle, RSS/PSS from smaps_rollup. Like-for-like runs with no swap pressure show -O costs klippy ~1.1 MiB RSS / ~1.2 MiB PSS rather than saving memory; Moonraker showed the same regression (+1.5/+2.2 MiB) and its -O was already rejected. no_debug_ranges stays on both.
pdscomp
changed the title
PYBENCH — Python
|
| Run | RSS | PSS | Mem used | Mem free | Swap used | Swap free |
|---|---|---|---|---|---|---|
| 1 | 31,500 | 27,308 | 77,492 | 6,400 | 0 | 0 |
| 2 | 30,988 | 26,792 | 77,728 | 5,420 | 0 | 0 |
| 3 | 31,568 | 27,426 | 78,820 | 5,584 | 0 | 0 |
| 4 | 30,660 | 26,530 | 78,528 | 9,960 | 0 | 0 |
| 5 | 26,572 | 22,240 | 74,560 | 7,760 | 10,240 | 219,068 |
Mean RSS 30,258 / PSS 26,059. Median RSS 30,988 / PSS 26,792.
Phase B — Moonraker with -O
SWU: opencentauri-upgrade-...-20260726225130.swu
(one-line change: -O -X no_debug_ranges in moonraker-init-d)
| Run | RSS | PSS | Mem used | Mem free | Swap used | Swap free |
|---|---|---|---|---|---|---|
| 1 | 29,440 | 25,372 | 77,000 | 10,932 | 11,264 | 218,044 |
| 2 | 32,336 | 28,864 | 80,692 | 10,280 | 0 | 0 |
| 3 | 31,044 | 26,786 | 77,684 | 7,212 | 8,960 | 220,348 |
| 4 | 33,104 | 29,543 | 80,868 | 7,740 | 0 | 0 |
| 5 | 27,432 | 23,323 | 76,060 | 13,080 | 14,080 | 215,228 |
Mean RSS 30,671 / PSS 26,778. Median RSS 31,044 / PSS 26,786.
Post-bench health: /server/info → klippy_connected: true,
klippy_state: ready.
Summary — how -O affects Moonraker memory
-O makes Moonraker use more resident memory, not less.
Naive means look like a wash (RSS +1.4%, PSS +2.8% with -O), but the swap
column confounds them: when zram swap is active at the 2-minute mark, the
kernel has already pushed Moonraker pages out, deflating RSS/PSS. Swap was
active in 1 of 5 baseline runs but 3 of 5 -O runs — itself a hint
of higher memory pressure under -O.
Comparing like-for-like runs with no swap activity:
| Baseline (runs 1–4) | -O (runs 2, 4) |
Δ | |
|---|---|---|---|
| RSS | 31,179 | 32,720 | +1,541 KiB |
| PSS | 27,014 | 29,204 | +2,190 KiB |
-O costs Moonraker roughly 1.5–2.2 MiB more resident memory at settled
idle. This reproduces the earlier APP.md screening result (~+3.2 MiB
effective memory after clean reboot) that got -O rejected for Moonraker.
Likely mechanism: -O strips asserts and docstrings, but the dominant
interpreter cost is live heap objects (tornado/asyncio state, JSON buffers),
not bytecode metadata. Optimized .pyc files are also slightly larger and
the changed bytecode defeats some sharing with the preloaded stdlib.
Recommendation: keep Moonraker unoptimized (-X no_debug_ranges only).
Klipper's pre-existing -O is untouched by this experiment. PR #271 stays
as-is; -O for Moonraker remains in the rejected-experiments list, now with
5-run confirmation data.
Phase C — Klippy with -O (stock flag, for comparison)
SWU: same as Phase B build (-20260726225130, klippy -O -X no_debug_ranges,
Moonraker also -O — Moonraker's flags do not affect Klippy's own RSS/PSS,
but Phase C MEM totals carry Moonraker's +1.5–2 MiB -O overhead).
| Run | RSS | PSS | Mem used | Mem free | Swap used | Swap free |
|---|---|---|---|---|---|---|
| 1 | 40,436 | 35,699 | 77,436 | 7,472 | 9,216 | 220,092 |
| 2 | 35,992 | 32,485 | 80,804 | 9,408 | 0 | 0 |
| 3 | 36,024 | 32,458 | 79,892 | 6,800 | 0 | 0 |
| 4 | 37,292 | 33,213 | 80,428 | 10,872 | 0 | 0 |
| 5 | 40,376 | 35,758 | 77,688 | 6,632 | 10,752 | 218,556 |
Mean RSS 38,024 / PSS 33,923. Median RSS 37,292 / PSS 33,213.
Phase D — Klippy without -O
SWU: opencentauri-upgrade-...-20260727115543.swu
(klippy -X no_debug_ranges only; Moonraker back to branch config, no -O)
| Run | RSS | PSS | Mem used | Mem free | Swap used | Swap free |
|---|---|---|---|---|---|---|
| 1 | 35,288 | 31,468 | 75,440 | 9,796 | 0 | 0 |
| 2 | 34,744 | 30,913 | 75,240 | 10,112 | 0 | 0 |
| 3 | 36,732 | 32,078 | 71,832 | 7,236 | 10,496 | 218,812 |
| 4 | 39,112 | 34,476 | 74,948 | 6,540 | 5,376 | 223,932 |
| 5 | 36,020 | 32,177 | 75,664 | 8,736 | 0 | 0 |
Mean RSS 36,379 / PSS 32,222. Median RSS 36,020 / PSS 32,078.
Post-bench health: /server/info → klippy_connected: true,
klippy_state: ready.
Summary — how -O affects Klippy memory
Same story as Moonraker, smaller magnitude: -O costs Klippy ~1.1–1.2
MiB, it doesn't save anything.
Like-for-like runs with no swap activity:
no -O (runs 1, 2, 5) |
-O (runs 2, 3, 4) |
Δ | |
|---|---|---|---|
| RSS | 35,351 | 36,436 | +1,085 KiB |
| PSS | 31,519 | 32,719 | +1,200 KiB |
Swap was active in 2/5 runs of each phase; the swap-active runs again read
lower on RSS/PSS because zram had already reclaimed pages, so naive means
understate the -O cost (means: RSS +4.5%, PSS +5.3% — inflated by Phase
C's swap-active runs also being its two highest readings; the like-for-like
table is the honest comparison).
Combined verdict
| Service | -O cost (like-for-like) |
|---|---|
| Moonraker | +1.5 MiB RSS / +2.2 MiB PSS |
| Klippy | +1.1 MiB RSS / +1.2 MiB PSS |
-O on a long-running CPython service trades a tiny bytecode-size win for
measurably higher resident memory (optimized .pyc differences and lost
assert/docstring sharing patterns on this 3.11 build). On a 128 MiB board,
dropping -O from both services reclaims roughly 2.6–3.4 MiB at
idle — worth a follow-up PR to remove Klippy's stock -O too.
Raw logs
pybench-results.txt(Phase A)pybench-results-O.txt(Phase B)pybench-klippy-O.txt(Phase C)pybench-klippy-noO.txt(Phase D)pybench.sh/pybench2.sh(capture scripts)
✅ Build ArtifactsBranch:
|
PYBENCH — Python
|
| Run | RSS | PSS | Mem used | Mem free | Swap used | Swap free |
|---|---|---|---|---|---|---|
| 1 | 31,500 | 27,308 | 77,492 | 6,400 | 0 | 0 |
| 2 | 30,988 | 26,792 | 77,728 | 5,420 | 0 | 0 |
| 3 | 31,568 | 27,426 | 78,820 | 5,584 | 0 | 0 |
| 4 | 30,660 | 26,530 | 78,528 | 9,960 | 0 | 0 |
| 5 | 26,572 | 22,240 | 74,560 | 7,760 | 10,240 | 219,068 |
Mean RSS 30,258 / PSS 26,059. Median RSS 30,988 / PSS 26,792.
Phase B — Moonraker with -O
SWU: opencentauri-upgrade-...-20260726225130.swu
(one-line change: -O -X no_debug_ranges in moonraker-init-d)
| Run | RSS | PSS | Mem used | Mem free | Swap used | Swap free |
|---|---|---|---|---|---|---|
| 1 | 29,440 | 25,372 | 77,000 | 10,932 | 11,264 | 218,044 |
| 2 | 32,336 | 28,864 | 80,692 | 10,280 | 0 | 0 |
| 3 | 31,044 | 26,786 | 77,684 | 7,212 | 8,960 | 220,348 |
| 4 | 33,104 | 29,543 | 80,868 | 7,740 | 0 | 0 |
| 5 | 27,432 | 23,323 | 76,060 | 13,080 | 14,080 | 215,228 |
Mean RSS 30,671 / PSS 26,778. Median RSS 31,044 / PSS 26,786.
Post-bench health: /server/info → klippy_connected: true,
klippy_state: ready.
Summary — how -O affects Moonraker memory
-O makes Moonraker use more resident memory, not less.
Naive means look like a wash (RSS +1.4%, PSS +2.8% with -O), but the swap
column confounds them: when zram swap is active at the 2-minute mark, the
kernel has already pushed Moonraker pages out, deflating RSS/PSS. Swap was
active in 1 of 5 baseline runs but 3 of 5 -O runs — itself a hint
of higher memory pressure under -O.
Comparing like-for-like runs with no swap activity:
| Baseline (runs 1–4) | -O (runs 2, 4) |
Δ | |
|---|---|---|---|
| RSS | 31,179 | 32,720 | +1,541 KiB |
| PSS | 27,014 | 29,204 | +2,190 KiB |
-O costs Moonraker roughly 1.5–2.2 MiB more resident memory at settled
idle. This reproduces the earlier APP.md screening result (~+3.2 MiB
effective memory after clean reboot) that got -O rejected for Moonraker.
Likely mechanism: -O strips asserts and docstrings, but the dominant
interpreter cost is live heap objects (tornado/asyncio state, JSON buffers),
not bytecode metadata. Optimized .pyc files are also slightly larger and
the changed bytecode defeats some sharing with the preloaded stdlib.
Recommendation: keep Moonraker unoptimized (-X no_debug_ranges only).
Klipper's pre-existing -O is untouched by this experiment. PR #271 stays
as-is; -O for Moonraker remains in the rejected-experiments list, now with
5-run confirmation data.
Phase C — Klippy with -O (stock flag, for comparison)
SWU: same as Phase B build (-20260726225130, klippy -O -X no_debug_ranges,
Moonraker also -O — Moonraker's flags do not affect Klippy's own RSS/PSS,
but Phase C MEM totals carry Moonraker's +1.5–2 MiB -O overhead).
| Run | RSS | PSS | Mem used | Mem free | Swap used | Swap free |
|---|---|---|---|---|---|---|
| 1 | 40,436 | 35,699 | 77,436 | 7,472 | 9,216 | 220,092 |
| 2 | 35,992 | 32,485 | 80,804 | 9,408 | 0 | 0 |
| 3 | 36,024 | 32,458 | 79,892 | 6,800 | 0 | 0 |
| 4 | 37,292 | 33,213 | 80,428 | 10,872 | 0 | 0 |
| 5 | 40,376 | 35,758 | 77,688 | 6,632 | 10,752 | 218,556 |
Mean RSS 38,024 / PSS 33,923. Median RSS 37,292 / PSS 33,213.
Phase D — Klippy without -O
SWU: opencentauri-upgrade-...-20260727115543.swu
(klippy -X no_debug_ranges only; Moonraker back to branch config, no -O)
| Run | RSS | PSS | Mem used | Mem free | Swap used | Swap free |
|---|---|---|---|---|---|---|
| 1 | 35,288 | 31,468 | 75,440 | 9,796 | 0 | 0 |
| 2 | 34,744 | 30,913 | 75,240 | 10,112 | 0 | 0 |
| 3 | 36,732 | 32,078 | 71,832 | 7,236 | 10,496 | 218,812 |
| 4 | 39,112 | 34,476 | 74,948 | 6,540 | 5,376 | 223,932 |
| 5 | 36,020 | 32,177 | 75,664 | 8,736 | 0 | 0 |
Mean RSS 36,379 / PSS 32,222. Median RSS 36,020 / PSS 32,078.
Post-bench health: /server/info → klippy_connected: true,
klippy_state: ready.
Summary — how -O affects Klippy memory
Same story as Moonraker, smaller magnitude: -O costs Klippy ~1.1–1.2
MiB, it doesn't save anything.
Like-for-like runs with no swap activity:
no -O (runs 1, 2, 5) |
-O (runs 2, 3, 4) |
Δ | |
|---|---|---|---|
| RSS | 35,351 | 36,436 | +1,085 KiB |
| PSS | 31,519 | 32,719 | +1,200 KiB |
Swap was active in 2/5 runs of each phase; the swap-active runs again read
lower on RSS/PSS because zram had already reclaimed pages, so naive means
understate the -O cost (means: RSS +4.5%, PSS +5.3% — inflated by Phase
C's swap-active runs also being its two highest readings; the like-for-like
table is the honest comparison).
Combined verdict
| Service | -O cost (like-for-like) |
|---|---|
| Moonraker | +1.5 MiB RSS / +2.2 MiB PSS |
| Klippy | +1.1 MiB RSS / +1.2 MiB PSS |
-O on a long-running CPython service trades a tiny bytecode-size win for
measurably higher resident memory (optimized .pyc differences and lost
assert/docstring sharing patterns on this 3.11 build). On a 128 MiB board,
dropping -O from both services reclaims roughly 2.6–3.4 MiB at
idle — worth a follow-up PR to remove Klippy's stock -O too.
Root-cause analysis: why -O increases RSS on Cosmos (2026-07-27)
Initial theories were all wrong. What we ruled out, with evidence:
| Theory | Result |
|---|---|
| Assert-stripped bytecode is bigger | No — marshalled bytecode is 2.2 KiB smaller with -O (4,703 vs 4,705 KiB over 271 klippy+moonraker files) |
-O strips docstrings |
No — only -OO does; 21.3 KiB of docstrings present in both |
| Different modules imported | No — sys.modules identical, 452 = 452 for Moonraker |
| More live Python objects | No — gc object counts equal; tracemalloc top-40 totals comparable |
| numpy lazy-load difference | No — bare import klippy.webhooks pulls 105 numpy modules in both modes |
| pymalloc arena slack | Partly — malloc_trim(0) recovers ~2 MiB from both, but -O still sits higher after trim |
The actual mechanism: .pyc cache-tag mismatch
The Cosmos image ships 2,253 precompiled stdlib .pyc files, all at
optimization level 0 (e.g. json/__pycache__/__init__.cpython-312.pyc).
Python's bytecode cache tag embeds the optimize level:
- plain → looks for
cpython-312.pyc→ hit, unmarshals shipped bytecode -O→ looks forcpython-312.opt-1.pyc→ miss (only 2 opt-1 files
exist in the image) → recompiles every module from.pysource
Import trace (python3 -v), same interpreter, same boot:
plain: # code object from '/usr/lib/python3.12/json/__pycache__/__init__.cpython-312.pyc'
-O: # code object from /usr/lib/python3.12/json/__init__.py
The rootfs is read-only squashfs, so -O's freshly compiled bytecode
cannot be cached — the full stdlib recompile happens on every service
start. Compiling from source runs the parser/AST/codegen pipeline, whose
transient allocations inflate the pymalloc arena high-water mark; pymalloc
never returns arenas to the OS, so the services idle ~1.5–2.2 MiB higher
forever even though the live object graph is identical.
Time series confirms the watermark shape: -O Moonraker jumps to
heap 7,928 KiB by t=20 s and plateaus; plain plateaus lower at 6,324 KiB.
The delta is entirely heap+anon (file-backed RSS flat), and identical
steady-state module/object counts rule out a "real" workload difference.
Same mechanism explains the Klippy -O regression (+1.1 MiB) — smaller
because klippy imports fewer stdlib modules than Moonraker.
Why this validates the PR
Removing -O from klippy (and never adding it to Moonraker) keeps both
services on the shipped opt-0 .pyc fast path: unmarshal instead of
compile-from-source, lower startup allocator watermark, ~1.1–2.2 MiB less
RSS per service, and faster startup as a bonus. If -O is ever wanted,
the image must ship matching opt-1 bytecode for the stdlib — previously
measured at ~856 KiB extra image space with no steady-state benefit
(APP.md "precompiled-pyc" experiment), which is the wrong trade on a
128 MiB board.
Evidence artifacts (in this repo dir)
regionbench-moonraker.txt/regionbench-klippy.txt— same-boot A/B smaps region breakdowntsbench.txt— startup time series (watermark plateau)gc-{moonraker,klippy}-opt{0,1}.json— gc/tracemalloc probesmods-mr-{plain,opt1}.json— steady-state module lists (identical)pycache-{moonraker,klippy}.tar.gz— on-device bytecode dumps viaPYTHONPYCACHEPREFIXregionbench.sh,tsbench.sh,pycapture.sh,gcprobe.py— harnesses
Raw logs
pybench-results.txt(Phase A)pybench-results-O.txt(Phase B)pybench-klippy-O.txt(Phase C)pybench-klippy-noO.txt(Phase D)pybench.sh/pybench2.sh(capture scripts)
|
That looks like some good rigourous testing @pdscomp! Nice work! I'm happy to merge this if @suchmememanyskill is also happy? |
3 participants
python: cut interpreter memory overhead on Klipper + Moonraker
TL;DR for non-specialists
Python (the language Klipper and Moonraker run on) records extra debug
bookkeeping in every running program, and on this board its "optimized"
mode (
-O) actually uses more RAM, not less. This PR turns off thewasted bookkeeping (
-X no_debug_ranges) on both services and removes-Ofrom Klipper's startup. Measured on a real printer: ~2.6–3.4 MiBless RAM used at idle across the two services versus stock — a big deal
on a 128 MiB board.
What changed
klipper-init-d:-O -X no_debug_ranges→-X no_debug_rangesmoonraker-init-d: add-X no_debug_ranges(never had-O)Two commits, two lines.
Benchmarks (carbon2u, CC1, 128 MiB)
Per variant: full image rebuild, md5-verified SWU flash, on-device flag
verification, then 5× (reboot → 120 s settle → capture). RSS/PSS from
/proc/<pid>/smaps_rollup; Mem/Swap fromfree. No heaters, no motion,no config changes. Full writeup with all 20 runs:
PYBENCH.md.
-Ocosts memory on both services (like-for-like, no swap pressure)-O-ONaive 5-run means look closer to a wash because zram swap kicked in more
often under
-O(3/5 Moonraker runs vs 1/5 baseline) and swapped-outpages deflate RSS/PSS readings — the table above compares only runs with
no swap activity, which is the honest comparison.
no_debug_ranges(kept from earlier screening)~1.4 MiB more MemAvailable at idle; restart timing unchanged (Klipper
4–5 s, Moonraker 9–11 s); APIs, camera, GPIO, I²C, PWM, remoteproc all
healthy. Earlier full test log:
APP.md
(same gist).
Post-flash health (final build)
/server/info→klippy_connected: true,klippy_state: ready.Experiments not retained
Tested on earlier screening passes and rejected with data:
-O: +1.5–2.2 MiB resident (reproduced twice)..pyc: no steady-state or startup benefit,~856 KiB image cost, complicates read-only-image updates.
MALLOC_ARENA_MAX=2: no repeatable settled-idle gain.python -S: ~0.27 MiB, bypasses.pthprocessing — not worth it.Root cause: why
-Ocosts RAM hereFollow-up analysis (same-boot A/B, smaps region diffs, gc/tracemalloc
probes, on-device bytecode capture) pinned it down. It's not the
bytecode itself —
-Obytecode is actually 2.2 KiB smaller, docstringssurvive (only
-OOstrips them), and both variants import an identicalmodule set (452 = 452 for Moonraker).
The Cosmos image ships 2,253 precompiled stdlib
.pycfiles, all atopt level 0. Python's cache tag embeds the opt level: plain Python loads
cpython-312.pyc(cache hit, fast unmarshal), while-Olooks forcpython-312.opt-1.pyc— which doesn't exist in the image — andrecompiles the entire stdlib from source on every service start. The
rootfs is read-only squashfs, so the new bytecode can't be cached; the
compile-phase allocations permanently inflate the pymalloc arena watermark
by ~1.1–2.2 MiB per service. Startup time series shows exactly this
plateau shape, and
malloc_trim(0)confirms the delta is retained arenamemory, not live objects (identical gc counts).
Import-trace proof, same boot:
So on this image
-Ois strictly a loss: no assert/docstring benefitworth having, real RAM and startup-time cost. Full evidence trail in the
PYBENCH.md gist
(root-cause section, plus raw probes and harnesses).
Scope and rollback
Two-line diff in two init scripts; revert to roll back. NumPy lazy-loading
continues separately in #257.