Skip to content

feat: persist OS tuning across reboots (sysctl.d + boot tuning unit)#56

Merged
emrebulutlar merged 1 commit into
mainfrom
fix/persist-os-tuning
Jul 4, 2026
Merged

feat: persist OS tuning across reboots (sysctl.d + boot tuning unit)#56
emrebulutlar merged 1 commit into
mainfrom
fix/persist-os-tuning

Conversation

@emrebulutlar

Copy link
Copy Markdown
Member

Closes #48 (P1.4, v0.3.0-beta).

Problem

system-tuning.sh / make optimize-os wrote runtime sysctls only. After the 2026-07-03 reboot, net.core.rmem_max/wmem_max reverted to 212992, the C media drivers crash-looped on SO_RCVBUF validation, the process manager kept starting nodes against absent drivers, and the kill-mid-write cycle corrupted node0's archive segment (the #35 replay hot-loop). Every reboot recreates this unless someone remembers to re-tune.

Changes

  • system-tuning.sh --persist (new make tune-persist):
    • writes the full sysctl set (transport-critical + the broader optimize-os set) to /etc/sysctl.d/99-openexchange.conf, commenting out keys absent on the running kernel (e.g. the pre-EEVDF kernel.sched_* knobs on 6.x), and applies it immediately so runtime == boot state
    • installs + enables openexchange-tuning.service, a oneshot boot unit for the two non-sysctl knobs: THP=never and performance governor
    • idempotent: skips when file/unit already match; degrades to WARN without sudo
  • --report drift detection: shows each persisted key that diverges from the runtime value, plus the boot unit's enablement, so make tune-report catches kernel upgrades / manual changes
  • make optimize-os now runs the persistence step at the end
  • Runbook (.claude/CLAUDE.md) + production checklist (docs/kernel-bypass.md) updated

Verification

  • Conf generation verified against the live 6.17 kernel (sched knobs correctly commented out, tcp values normalized)
  • Drift detection verified with a fixture file (value mismatch + unknown key both flagged; tab-vs-space normalization for tcp_rmem confirmed)
  • No-sudo path degrades to WARN, exit 0
  • Root-only steps (install to /etc, systemctl enable) need a privileged run on the box: sudo make tune-persist. Exit criterion (clean reboot with zero manual steps) verifiable at the next reboot via make tune-report.

🤖 Generated with Claude Code

…(match#48)

Runtime-only sysctls reverting on reboot is the root of the 2026-07-03
corruption chain: rmem_max/wmem_max fell back to 212992, the C media
drivers crash-looped on SO_RCVBUF validation, nodes started against
absent drivers, and the kill-mid-write cycle corrupted node0's archive
segment (match#35 replay hot-loop).

- system-tuning.sh --persist (make tune-persist): writes the full sysctl
  set (transport-critical + optimize-os) to
  /etc/sysctl.d/99-openexchange.conf, skipping keys absent on the running
  kernel, and installs+enables openexchange-tuning.service (oneshot) for
  the two non-sysctl knobs: THP=never and performance governor.
- --report now shows runtime-vs-persisted drift per key and the boot
  unit's enablement, so `make tune-report` catches divergence after
  kernel upgrades. Persist paths are env-overridable for tests.
- make optimize-os runs the persistence step at the end.
- Runbook + production checklist updated (verify with make tune-report
  after reboots/kernel upgrades, before starting drivers/nodes).

Closes #48

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@emrebulutlar
emrebulutlar merged commit fcb7f58 into main Jul 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P1.4] Persist OS tuning across reboots (sysctl.d + THP/governor units)

1 participant