[9.5](backport #7568) fix: size ristretto cache from GOMEMLIMIT instead of host RAM - #7577
Merged
Conversation
* fix: size ristretto cache from GOMEMLIMIT instead of host RAM memEnvLimits() called memory.TotalMemory() which returns host-node RAM, not the container cgroup limit. On a K8s node with >=16 GB RAM this sized the ristretto MaxCost at 256-512 MB -- 2-4x the pod's GOMEMLIMIT -- causing OOMKills at modest agent counts. Fix: read the current GOMEMLIMIT via debug.SetMemoryLimit(-1) and use it to select the cache tier. Falls back to memory.TotalMemory() when GOMEMLIMIT is unset (math.MaxInt64), preserving behaviour for non-containerised deployments. * test: add TestContainerMemoryMB covering GOMEMLIMIT and host-RAM paths * changelog: add fragment for ristretto cache container-memory fix * fix: use memMB() in loadLimits to respect GOMEMLIMIT for RAM warning The agent-count path in loadLimits was still calling memory.TotalMemory() directly, so the low-RAM warning would fire against host node RAM rather than the container's available memory. Switch to memMB() to stay consistent with memEnvLimits(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> (cherry picked from commit 7626dc8)
5 tasks
ycombinator
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the problem this PR solves?
memEnvLimits()ininternal/pkg/config/env_defaults.gocallsmemory.TotalMemory()to select the ristretto cache tier.memory.TotalMemory()returns the host node's total physical RAM, not the container's cgroup memory limit. On a Kubernetes node with ≥16 GB of RAM, this selects aMaxCostof 256–512 MB — 2–4× the pod's GOMEMLIMIT (128 MB at the 256 M default pod limit). The cache is allowed to grow past the GOMEMLIMIT, triggering OOMKills even when live application heap is within budget.This was the primary driver of the OOMKills observed for a high-volume serverless project in elastic/ingest-dev#8991.
How does this PR solve the problem?
Introduces
containerMemoryMB(), which reads the current GOMEMLIMIT viadebug.SetMemoryLimit(-1)and converts it to MiB. When GOMEMLIMIT is set (i.e. ≠math.MaxInt64), that value drives cache tier selection — so a pod with a 256 M limit and GOMEMLIMIT = 128 M will select a cache sized for 128 MB of available memory, not 16+ GB of node RAM.When GOMEMLIMIT is unset (
math.MaxInt64), the function falls back tomemory.TotalMemory(), preserving the existing behaviour for non-containerised deployments.memMBremains avarpointing tocontainerMemoryMB, so existing tests that stubmemMBare unaffected.How to test this PR locally
Run the config package tests:
TestContainerMemoryMBcovers both the GOMEMLIMIT path (sets a 256 MiB limit and asserts the function returns 256) and the fallback path (clears the limit and asserts the function returns host RAM in MiB).To verify the end-to-end effect: start fleet-server with
GOMEMLIMIT=128MiB(or viaserver.runtime.memory_limit) and observe that the loggedrecommended_mbvalue is ≤ 128, not 256–512.Design Checklist
Checklist
./changelog/fragmentsusing the changelog toolRelated issues
This is an automatic backport of pull request fix: size ristretto cache from GOMEMLIMIT instead of host RAM #7568 done by Mergify.