fix(slab): the guard watches memcg-attributed truth, not the drifting counter - #348
Merged
Merged
Conversation
… counter Diagnosed live on runner-2: /proc/meminfo SUnreclaim said 1041 MiB while the root cgroup's memory.stat attributed 44 MiB, and the ~1 GiB gap was unattributable to any live cache -- per-cache sums reached ~220 MiB -- and immune to drop_caches=2 (VFS caches fell 486 MiB, the counter did not move) and to per-cache shrink. The global NR_SLAB_UNRECLAIMABLE counter drifts under container churn on 6.8.0-138; the memory it reports is not occupied. The healer was rebooting members daily to reset an integer, and the alert paged on the same phantom. The compliance observer now exports both views side by side (slab_unreclaimable_counter_bytes / _attributed_bytes), slab-heal prefers the attributed value when cgroup v2 exposes it and names its source in the heal reason, and kernel_slab_unreclaimable watches the attributed gauge. The reboot path stays armed for genuine slab growth. Closes #264 Claude-Session: https://claude.ai/code/session_01LsGid6U5RrQdFvJmvYdGCF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #264 with a diagnosis instead of a wait: the "leak" is kernel accounting drift, not occupied memory.
Measured on gha-runner-2 (6.8.0-138, 20h uptime): meminfo
SUnreclaim= 1041 MiB; root cgroupmemory.stat slab_unreclaimable= 44 MiB; per-cache sums (objects and pages, unreclaimable-flagged via/sys/kernel/slab) ≈ 220 MiB;drop_caches=2freed 486 MiB of SReclaimable and moved the counter by 0; per-cacheshrinkmoved it by 0; 361 dying memcgs neither popped nor pinned it. The global counter drifts under container churn — the healer was rebooting members daily to reset an integer.Change: compliance observer exports both views (
gha_fleet_host_slab_unreclaimable_counter_bytes+_attributed_bytes, -1 when unavailable);slab-healprefers the attributed value (helper + parser pinned by tests, source named in the heal reason, meminfo fallback for non-cgroup-v2 hosts);kernel_slab_unreclaimablealert watches the attributed gauge (metric contract test updated); maintenance-windows runbook records the story. Reboot healing stays armed for genuine growth.https://claude.ai/code/session_01LsGid6U5RrQdFvJmvYdGCF