Skip to content

[9.4](backport #7568) fix: size ristretto cache from GOMEMLIMIT instead of host RAM - #7578

Merged
ycombinator merged 1 commit into
9.4from
mergify/bp/9.4/pr-7568
Aug 7, 2026
Merged

[9.4](backport #7568) fix: size ristretto cache from GOMEMLIMIT instead of host RAM#7578
ycombinator merged 1 commit into
9.4from
mergify/bp/9.4/pr-7568

Conversation

@mergify

@mergify mergify Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

What is the problem this PR solves?

memEnvLimits() in internal/pkg/config/env_defaults.go calls memory.TotalMemory() to select the ristretto cache tier. memory.TotalMemory() returns the host node's total physical RAM, not the container's cgroup memory limit. On a Kubernetes node with ≥16 GB of RAM, this selects a MaxCost of 256–512 MB — 2–4× the pod's GOMEMLIMIT (128 MB at the 256 M default pod limit). The cache is allowed to grow past the GOMEMLIMIT, triggering OOMKills even when live application heap is within budget.

This was the primary driver of the OOMKills observed for a high-volume serverless project in elastic/ingest-dev#8991.

How does this PR solve the problem?

Introduces containerMemoryMB(), which reads the current GOMEMLIMIT via debug.SetMemoryLimit(-1) and converts it to MiB. When GOMEMLIMIT is set (i.e. ≠ math.MaxInt64), that value drives cache tier selection — so a pod with a 256 M limit and GOMEMLIMIT = 128 M will select a cache sized for 128 MB of available memory, not 16+ GB of node RAM.

When GOMEMLIMIT is unset (math.MaxInt64), the function falls back to memory.TotalMemory(), preserving the existing behaviour for non-containerised deployments.

memMB remains a var pointing to containerMemoryMB, so existing tests that stub memMB are unaffected.

How to test this PR locally

Run the config package tests:

go test ./internal/pkg/config/... -run TestContainerMemoryMB -v

TestContainerMemoryMB covers both the GOMEMLIMIT path (sets a 256 MiB limit and asserts the function returns 256) and the fallback path (clears the limit and asserts the function returns host RAM in MiB).

To verify the end-to-end effect: start fleet-server with GOMEMLIMIT=128MiB (or via server.runtime.memory_limit) and observe that the logged recommended_mb value is ≤ 128, not 256–512.

Design Checklist

  • I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
  • I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
  • I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool

Related issues

* fix: size ristretto cache from GOMEMLIMIT instead of host RAM

memEnvLimits() called memory.TotalMemory() which returns host-node RAM,
not the container cgroup limit. On a K8s node with >=16 GB RAM this sized
the ristretto MaxCost at 256-512 MB -- 2-4x the pod's GOMEMLIMIT -- causing
OOMKills at modest agent counts.

Fix: read the current GOMEMLIMIT via debug.SetMemoryLimit(-1) and use it
to select the cache tier. Falls back to memory.TotalMemory() when GOMEMLIMIT
is unset (math.MaxInt64), preserving behaviour for non-containerised deployments.

* test: add TestContainerMemoryMB covering GOMEMLIMIT and host-RAM paths

* changelog: add fragment for ristretto cache container-memory fix

* fix: use memMB() in loadLimits to respect GOMEMLIMIT for RAM warning

The agent-count path in loadLimits was still calling memory.TotalMemory()
directly, so the low-RAM warning would fire against host node RAM rather
than the container's available memory. Switch to memMB() to stay
consistent with memEnvLimits().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
(cherry picked from commit 7626dc8)
@mergify mergify Bot added the backport label Aug 7, 2026
@mergify
mergify Bot requested a review from a team as a code owner August 7, 2026 01:43
@mergify
mergify Bot requested review from macdewee and ycombinator August 7, 2026 01:43
@mergify mergify Bot added the backport label Aug 7, 2026
@github-actions github-actions Bot added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Aug 7, 2026
@ycombinator
ycombinator merged commit 00cc774 into 9.4 Aug 7, 2026
13 checks passed
@ycombinator
ycombinator deleted the mergify/bp/9.4/pr-7568 branch August 7, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant