Dgraph Alpha nodes repeatedly run out of memory and get OOM-killed in Kubernetes

[abhishek-ashish-rafay]

Gradual memory growth in Dgraph Alpha eventually leads to OOM kills

Hi Dgraph team,

We are running Dgraph v24.0.5 on Kubernetes with three Alpha replicas and are observing gradual memory growth over time, eventually leading to container OOM kills and restarts.

Environment

• Dgraph version: v24.0.5
• Deployment: Kubernetes StatefulSet, 3 Alpha replicas
• Previous container memory limit: 1 GiB
• Current configuration under investigation: 2 GiB memory request/limit per Alpha
• GOMEMLIMIT=1200MiB

Observed behavior

Memory consumption tends to increase gradually over time rather than only during isolated spikes. Eventually, the container reaches its memory limit and is terminated with OOMKilled (exit code 137). One Alpha pod had accumulated 37 restarts during our investigation.

In one approximately two-hour observation window:

• Go HeapAlloc increased from ~153 MiB to ~177 MiB.
• Process RSS remained around ~524 MiB.
• Cgroup memory usage remained around ~666 MiB.

These measurements did not capture the eventual OOM event, so we cannot yet determine whether the growth is caused by the Go heap, Badger, memory-mapped files, or other process memory.

Relevant configuration

env:
  - name: GOMEMLIMIT
    value: "1200MiB"

resources:
  requests:
    cpu: 500m
    memory: 2Gi
  limits:
    cpu: "2"
    memory: 2Gi

Alpha command flags:

--cache "size-mb=256; percentage=0,80,20;"
--badger "compression=snappy; numgoroutines=8; memtablesize=16777216;"
--trace "ratio=0;"

We are testing the increased memory limit while keeping the cache and Badger settings unchanged.

Questions

1. Is gradual memory growth that eventually reaches the container limit expected under normal Dgraph workloads, or could this indicate a memory leak or issue in v24.0.5?
2. What metrics or profiling tools would you recommend to identify which part of Alpha is responsible for the growth, particularly when container memory usage differs from Go heap usage?
3. Are there any known issues or recommended configuration changes for v24.0.5 that could explain this behavior?

We have attached a screenshot of the memory usage for reference and can provide sanitized logs or memory profiles if helpful.

Thanks for your guidance.

Attaching ScreenShot for reference.

[matthewmcneely]

Hi @abhishek-ashish-rafay,

To your first question: a slow climb toward the limit isn't expected, but I don't think v24.0.5 has a known leak that matches what you're seeing. There have been real posting-list cache fixes since v24.0.5. In that version the posting-list cache gives every entry a cost of zero, so it never evicts (fixed in #9515), and resizing it at runtime did nothing (fixed in #9526). Your config turns that cache off, though. The first value in percentage=0,80,20 is the posting-list cache share, and with it at 0 the cache is never created. Your 256 MB goes to Badger instead: about 205 MB of block cache and 51 MB of index cache.

The gap between RSS and Go heap is most likely that Badger block cache. With compression=snappy, Badger decompresses blocks into memory it allocates itself (through jemalloc in the official builds), outside the Go heap. So it never appears in HeapAlloc, and GOMEMLIMIT can't see it or limit it -- this has been a unfortunately side-effect of jemalloc. A full ~205 MB block cache plus Go runtime overhead fits your ~524 MiB RSS against ~170 MiB of heap pretty well. The remaining ~140 MiB between RSS and cgroup usage is probably page cache from Badger's SST and value-log files. The kernel can usually reclaim that, but it still counts against the container.

Related: GOMEMLIMIT only covers the Go heap. If it was also 1200MiB when the limit was 1 GiB, the Go runtime thought it had more room than the container did, and that alone could explain a lot of those 37 restarts. With 2 GiB and 1200MiB you have some headroom now, but it's worth keeping the gap between the two at least as big as your off-heap usage.

To figure out which part is growing, the most useful thing is a series of samples running right up to an OOM, from at least one alpha:

• /debug/pprof/heap (Go heap)
• /debug/jemalloc (memory allocated outside the Go heap, broken down by allocator)
• dgraph_memory_inuse_bytes, dgraph_memory_alloc_bytes, and dgraph_memory_proc_bytes from /debug/prometheus_metrics (Go heap in use, jemalloc-allocated bytes, and RSS)
• container_memory_working_set_bytes from cAdvisor, or /sys/fs/cgroup/memory.stat from inside the pod (the anon and file lines matter most), since that's what the OOM killer goes by
• dgraph debuginfo -a :8080 -z :6080 collects most of this into one archive if that's easier. If you run it every so often while memory climbs, the trend should show whether the growth is Go heap, off-heap, or page cache.

A couple of questions:

1. Do you have any vector (HNSW) indexes? v24.0.x had an OOM in the HNSW index code that was fixed in fix(core): fix visited uids bitset memory #9221 (in v24.1.x).
2. What's the workload like? Mostly queries, steady mutations, or periodic bulk writes? And does the growth line up with any of them?

On upgrading: v24.1.9 has #9221 and the posting-cache rework from #9237. The cache-sizing fixes (#9515, #9526) are only in v25.x. Since your posting-list cache is off, I wouldn't expect an upgrade alone to change this, but it's worth doing anyway. If you move to v25 and want the posting-list cache back on, it's now sized properly.

If you can share a heap profile and a jemalloc dump from close to an OOM, I'm happy to take a look.

[abhishek-ashish-rafay]

Hi @matthewmcneely,

Thanks for the detailed reply. We dug further; here's where we are.

The main thing we see: memory climbs in steps and mostly stays there

After every restart the alpha's memory rises steadily, usually in steps, and it mostly holds each new level instead of returning to where it was. We've seen a couple of partial drops without a restart (~140–280 MiB of working set, possibly page cache), but the overall trend between restarts is always up.

• Longest uninterrupted run we have metrics for (Oct 21 → Oct 26, ~4.7 days; 1 GiB limit, default cache, no GOMEMLIMIT): container RSS went from ~400–460 MiB just after the restart to ~670–730 MiB on all three alphas, roughly 2–3 MiB/hour. It was still at that level on Oct 26 during a quiet hour with almost no writes.
• Current run (restarted Oct 28 with 2 GiB / GOMEMLIMIT=1200MiB): working set went from ~190 MiB to ~760 MiB within the first day.
• In the original post, HeapAlloc went from 153 to 177 MiB in two hours.

Screenshot below: working set of the dgraph-dgraph-alpha-1 pod , Oct 22–29 (times in IST). The deep drops are restarts. The requests/limits in the legend are pod totals including the Istio sidecar, at today's values; the Dgraph container limit was 2 GiB for most of this window and 1 GiB for part of Oct 26–27.

So the gradual growth is real and the same on every replica. The OOM kills themselves came from short spikes on top of that raised baseline (see the last section).

Your questions

We don't use any vector (HNSW) indexes. Our DQL schema uses standard scalar and UID predicates with indexes such as hash, exact, trigram, and others. The GraphQL schema doesn't use @embedding or Float32Vector.

Workload: One client (our API server) using /graphql over HTTP, load-balanced across the 3 alphas:

• ~3 requests/s in total; ~2.9 KB per request, ~3.6 KB per response, ~14 ms average latency
• ~93% queries, ~7% mutations (per alpha since the last restart: ~40.6k Server.Query, ~3.2k Server.Mutate)
• Small dataset: ~159.5k edges, 236–435 MB in p; mutations are mostly updates to existing nodes
• Writes come in bursts: disk writes are near zero most of the time, with stretches of 10–60 KiB/s
• CPU ~0.05 cores median; no bulk or live loader

The gradual growth continues through both busy and quiet periods, so we can't tie it to a specific operation yet.

Where the memory is now (MiB, Oct 28 18:00 UTC, same as the attached captures)

	HeapAlloc	HeapInuse	Go idle, not released	jemalloc	RSS	cgroup anon	cgroup file (mapped)
alpha-0	169	231	34	43	494	395	88 (23)
alpha-1	163	222	40	49	763	405	318 (286)
alpha-2	195	248	22	36	461	389	73 (18)

• jemalloc is only 36–49 MiB, all in Table.Decompress, so the off-heap block cache isn't the big piece (block-cache hit ratio is 99.5%).
• Most of the anon memory is the Go heap. About 60–70 MiB of anon isn't covered by Go runtime memory plus jemalloc's live allocations.
• alpha-1's higher RSS is mostly mmapped Badger files (286 MiB file_mapped).

Correction to our original post (GOMEMLIMIT)

The StatefulSet revision history shows GOMEMLIMIT wasn't set when the OOM kills happened:

• Oct 19 (one alpha, about 30 OOM kills over 3 hours) and Oct 21 (all three alphas in the same hour): 2 GiB limit, no GOMEMLIMIT, default cache flags.
• Oct 26: OOM loops caused by our own change to a 1 GiB limit (still no GOMEMLIMIT).
• 1 GiB with GOMEMLIMIT=1536MiB ran Oct 27–28 without any OOM kills.

So GOMEMLIMIT being above the container limit doesn't explain these restarts.

The OOM kills were short spikes on top of the baseline

Just before each OOM, cAdvisor showed RSS at about 0.6–1.0 GiB; the jump to the 2 GiB limit happened between two 30-second scrapes. Both events happened during write activity, and on Oct 21 all three alphas were killed within the same hour, which points at the replicated write path rather than a single query. On the current run, alpha-1's cgroup memory.peak reached 1.93 GiB of 2 GiB, while 1-minute metrics never showed it above ~820 MiB. The higher the baseline creeps, the less room is left for these spikes.

Questions

1. For the gradual growth, would heap profiles taken a few hours apart (for -diff_base) help, or is there something else you'd like us to capture?
2. Any idea what on the write path could allocate over 1 GiB in a few seconds on a dataset this small?

Thanks!

Sharing the Grafana graph showing Dgraph alpha container RSS memory usage across all three replicas over the last 7 days. It illustrates the memory usage trend over time, including the gradual increase between restarts.

[matthewmcneely]

Hi @abhishek-ashish-rafay, quick question before I go further. How often does the GraphQL schema get applied to the cluster? For example, does your API server push it on every startup or deploy, on a timer, or only when the schema actually changes?

If you're not sure, the alpha logs will show it. Every replica logs Max open files limit each time it applies a schema update, so kubectl logs <alpha-pod> | grep "Max open files limit" (or with --previous for the run before a restart) should show how often it happens and whether it lines up with the OOMs.

[abhishek-ashish-rafay]

HI @matthewmcneely,

Thanks — here's what we found.

How often the schema is applied: only when our API server starts, never on a timer. On every startup of an API-server pod (we run 2 replicas), it checks the stored schema version:

• If the version is unchanged and a few "canary" fields are present in the live schema, it skips the apply.
• It calls updateGQLSchema with the full schema when the version changed, when the canary check fails, or when it can't verify the schema.

So in practice it's applied on deploys that change the schema. It can also happen on other restarts or deploys: during a rolling deploy with mixed image versions, an older pod can re-apply its older schema, and the newer pod then re-applies the current one. The schema is about 40 KB / 67 types, and changes to it can add or alter @search indexes.

Alpha logs: kubectl logs <alpha> | grep "Max open files limit" returns 0 lines on all three alphas for the current run. They started at 2026-10-28 09:52 UTC, about 10 hours ago, with 0 restarts. Our API-server pods started at 07:04 UTC, before the alphas restarted, so no schema apply has happened during this run. Over those 10 hours we still saw the gradual growth, and alpha-1's short spike to a memory.peak of 1.93 GiB.

Unfortunately we can't check the runs that had the OOMs (Oct 19 and Oct 21). Those pods have been recreated since, so their logs and --previous aren't available. We'll check whether API-server deploys or restarts happened around those times (Oct 19 ~20:00 UTC and Oct 21 ~14:00 UTC) and follow up.

[matthewmcneely]

Hey @abhishek-ashish-rafay,

OK thanks, the reason I was asking was because of #9301 (TLDR; applying a schema could alloc a stupid amount of mem) -- which was fixed in v25.

I'll dig into this more tomorrow.

[abhishek-ashish-rafay]

Hi @matthewmcneely ,

One additional question based on our latest memory breakdown.

We observed that alpha-1 has around 320 MiB of file-backed memory, compared with roughly 95–100 MiB on alpha-0 and alpha-2. This appears to be primarily the Linux page cache for Badger's memory-mapped data files. Anonymous memory is relatively similar across the three replicas.

Is there a supported Dgraph or Badger configuration to limit this file-backed cache, or a way to encourage the kernel to reclaim idle cached pages under container memory pressure? Since this memory also contributes to cgroup memory usage, could it meaningfully reduce the headroom available for the short memory spikes that lead to OOM kills?

Also, based on our observations, would you consider this file-cache growth expected behavior, or should we investigate why the cache accumulates differently across replicas? We want to distinguish normal, reclaimable page-cache growth from the anonymous-memory growth that appears to raise the baseline over time.

[matthewmcneely]

Hi @abhishek-ashish-rafay,

Thanks for checking the logs. With no schema apply during the current run, #9301 can't explain alpha-1 reaching a memory.peak of 1.93 GiB. It could still explain Oct 19 and Oct 21, though. Both of those happened on a 2 GiB limit, and on Oct 21 all three alphas went down in the same hour. A schema update is applied through Raft, so every replica runs it at the same moment, which would fit.

One quick check settles whether #9301 is possible on your setup at all. The size of that buffer depends on the alpha's open-file soft limit:

kubectl exec <alpha-pod> -c <dgraph-container> -- grep "Max open files" /proc/1/limits

If the soft limit is in the billions (1073741816 is common on containerd), a schema apply allocates about 2 GiB at once. If it's around 1M or less, the buffer is a few MiB and we can drop that theory.

The 1.93 GiB peak may not be a spike. memory.peak tracks memory.current, and that includes page cache, both active and inactive. container_memory_working_set_bytes subtracts inactive_file. So a peak near the limit while working set stays around 820 MiB is consistent with inactive page cache filling the cgroup until the kernel reclaims it. That's normal kernel behavior. memory.events on the pod will show which one it was:

• max counts how often usage hit the limit and the kernel had to reclaim
• oom_kill counts actual kills

If max is non-zero and oom_kill is 0, the kernel reclaimed page cache and nothing was in danger.

For the real OOM kills on Oct 19 and Oct 21, the kernel log on the node is the best evidence we can get. But I think you indicated those were no longer available? The OOM killer logs a line like Killed process <pid> (dgraph) ... anon-rss:...kB, file-rss:...kB, shmem-rss:...kB. If those nodes haven't been recycled (or maybe when a node OOMs in the future), journalctl -k --since ... | grep -i "killed process" (or dmesg -T) should still have it. If anon-rss is near 2 GiB, the spike was real heap memory. If it's mostly file-rss, it's a different problem.

To your questions about File-backed mem, no, there's no Dgraph or Badger setting that limits it. Badger in v24.0.5 (v4.3.0) always mmaps its SST tables, value log files, and memtable WAL files, with no option for plain file reads. I'd consider the growth expected. Each pod's file cache holds whatever that process has read recently. That depends on which queries land on it and which tables its compactions touched, and with no memory pressure the kernel has no reason to evict anything. So differences between replicas are normal and not worth chasing.

It also shouldn't reduce your headroom for spikes much. Clean file pages, which is nearly all of Badger's read side, are reclaimed before the OOM killer runs. Dirty pages have to be written back first, but at your write rate there aren't many. The one real cost is that dashboards built on working set overstate how close you are to the limit. For the baseline question I'd focus on anon memory, which is what you're already doing.

I found one thing in v24.0.5 worth checking related the this "gradual growth" issue. Besides the posting-list cache you've disabled, v24.0.5 has a second cache just for dgraph.type index keys (#9068), and --cache percentage doesn't turn it off. It holds a copy of the UID list for each type. Every commit that sets dgraph.type appends its delta to that copy, and only a rollup of that key clears it. At low write rates, those rollups can be delayed for a long time. Every GraphQL query roots at type(X), so every query reads from this cache and clones the pending deltas. The cache was removed in #9237, which shipped in v24.1.0.

I'm not sure it's big enough on its own to account for 2–3 MiB/hour. It only grows when a mutation sets dgraph.type, which for GraphQL is mostly add mutations. If nearly all your writes are updates to existing nodes, this probably isn't it.

The heap diff will tell us either way. Take /debug/pprof/heap a few hours apart and compare:

go tool pprof -diff_base heap-early.pb.gz -sample_index=inuse_space heap-late.pb.gz

If most of the growth is under posting.(*List).setMutationAfterCommit, posting.copyList, or posting.(*Txn).UpdateCachedKeys, it's this cache, and moving to v24.1.9 should fix it. Please grab /debug/pprof/goroutine?debug=1 at the same times too. A slowly rising goroutine count won't show up in the heap diff.

Finally, if you have a staging environment I'd urge you to try v25.5.1. Among many small memory improvements was a pretty big change (#9515) to actually get the posting list cache to work correctly. My spidey sense tells me that you disabled the cache entirely to try to keep the alphas within their assigned memory space. Of course other things arrive on the heap, so it's not a hard upper limit but the posting list cache now works as intended.

Finally (part 2) -- in the past I've seen rogue queries spike as you've described. Worth checking IMO, in v24 graphql responses have an extensions.touched_uids that you could track in your API to see if any queries are fanning out beyond reasonable limits.

[abhishek-ashish-rafay]

Hi @matthewmcneely ,

Thanks, this was very helpful. Results below, including one correction to what I told you earlier.

Correction: Oct 19–26 was not a 2 GiB limit. The StatefulSet template said 2Gi, but the running alpha pods were never recreated and kept a 1 GiB limit (512Mi request). Our 2 GiB limit only started with the current run (Oct 28).

#9301: the open-file soft limit is 65535 on all three alphas, so we can drop that theory.

memory.events: all three alphas show max 0, oom 0, oom_kill 0, and pgscan/pgsteal are 0 on the alpha with the 1.93 GiB peak. So it never hit the limit and the kernel never reclaimed. On that alpha, go_memstats_heap_sys_bytes has been about 387 MiB in every snapshot, so the Go heap never got near the peak. The rest was jemalloc, or page cache that was freed when files were deleted (its p directory is only ~414 MiB). We can't tell which from what we have.

Kernel OOM log (available only for the node running alpha-1):

time (UTC)	memcg limit	anon at kill	file at kill
Oct 6 11:55	4 GiB	4074 MiB	~0
Oct 7 20:58 / Oct 8 14:17	4 GiB	4075 MiB	~0
Oct 21 14:12–14:16 (6 kills)	1 GiB	1008–1018 MiB	0–3 MiB

Every kill was anon, so the spikes are real allocations, not page cache. Two things stand out:

• On Oct 6–8 the alpha filled a 4 GiB limit with anon.
• On Oct 21 it crash-looped: each restart reached ~1 GiB anon within 5–80 s of starting, so startup itself needed more than 1 GiB at that point.

dgraph.type cache (#9068): setMutationAfterCommit, copyList and UpdateCachedKeys show 0 bytes in all 30 heap profiles (3 alphas × 10 snapshots over 21h). Goroutines are flat at 115–135.

Gradual growth: over the last ~20h, after warm-up, anon rose only 12–25 MiB per alpha (under 1 MiB/h). jemalloc rose 2–4 MiB and file cache was flat, all within normal GC variation. The only steady item in the heap diffs is OpenCensus span data, a few MB and bounded. So we don't see a leak in this run, and with the 1 GiB limit corrected, the earlier "growth" was mostly a limit that was too small.

Two questions:

• On Oct 21 each restart went from 0 to ~1 GiB anon within seconds. At startup in v24.0.5, what allocates that much? WAL replay, the opRollup that starts at boot, or catching up from the leader?
• Since all the kills were anon and the Go heap stayed small in this run, would a jemalloc-heavy path (e.g. Badger compaction table builders, or snappy decompression during a large iteration) be your first suspect for the spike?

[matthewmcneely]

Hi @abhishek-ashish-rafay,

Thanks, this is a really useful round of data.

"What allocates ~1 GiB at startup?": My best guess is a schema change that rebuilds indexes, replayed on every restart. When a schema update adds or changes an index, v24.0.5 rebuilds every affected predicate at the same time. Each rebuild opens its own temporary Badger instance with default options, starting at about 83 MB each. The new schema is only saved once the rebuild finishes. If the alpha dies part way through, it replays the change from the Raft log on restart, starts every rebuild again, and dies again. Schema changes go through Raft, so all replicas rebuild at once.

I reproduced this on v24.0.5 with a 1 GiB limit, 20k nodes and 40 string predicates. Indexing one predicate added about 100 MiB. Indexing 39 in one alter OOM-killed the alpha at about 1 GB of anon memory, then crash-looped, with each restart killed within seconds. That matches your Oct 21 pattern. It's worth checking whether an API-server deploy changed @search directives around those times. The rolling-deploy case where an older pod re-applies its schema would trigger it twice.

"Is a jemalloc-heavy path the spike?": Not compaction or snappy, I don't think. Both work in small, bounded pieces. The off-heap user that scales with a single request is the query response encoder. In my tests, a query returning a few million nodes put 512 MiB under the OutputNode.Encoder tag in /debug/jemalloc, while the Go heap stayed around 265 MB. Your current-run peak had heap_sys flat and no schema apply, which fits that pattern. If you catch /debug/jemalloc during a spike, look for that tag.

If you want to try it, I've put together a monitoring bundle in #9878 (contrib/monitoring). It has a Grafana dashboard that splits alpha memory into Go heap, jemalloc, anon and page cache, with alert rules. A small sidecar saves a heap profile, goroutine dump and jemalloc stats when memory spikes, plus the memory timeline leading into any restart. It works with v24 (and v25). It isn't merged yet, so you'd build the sidecar image from the Dockerfile.

Also, if you do have the ability to try a deploy to a staging cluster, I'd strongly recommend giving v25 a try. There have been numerous improvements to memory usage since v24.0.5. v25 is binary compatible with v24, so no data migration would be necessary.