etcd raft memory usage keep increasing and causing etcd to OOM error

[pratik-thacker]

Summary:

We have a use case to store values size of 100KB to 650KB in etcd, however the total number of keys are not more than 2000.
But we observe etcd memory consumption keeps on increasing and eventually leading to OOM error, sometime it takes just few hours and sometime 2 to 3 days.

Details:

• ETCD Version:

  bash-4.4$ etcd -version
  etcd Version: 3.3.11
  Git SHA: 2cf9e51d2
  Go Version: go1.10.7
  Go OS/Arch: linux/amd64
  bash-4.4$
  

• ETCD Members: 3
• ETCD Quota Backend Size: 256 MB
• Container Max Memory: 1 GB (We also tried to make it 2Gi but facing same problem)
• We tried to set/reduce below parameters one by one with the hope of improvement, but it didn't seem to be helping:

  ETCD_AUTO_COMPACTION_RETENTION=1
  ETCD_SNAPSHOT_COUNT=100
  GOGC=20
  GODEBUG="madvdontneed=1
  

Steps to Reproduce:

To quickly reproduce the issue, store huge value (1MB) size KV in etcd same key frequently. (This is nearly similar to our use case well)

1. Execute below command to store 1MB size values in same key repeatedly for 250 times:

for i in {1..250}; do dd if=/dev/urandom bs=1024 count=1024  | ETCDCTL_API=3 etcdctl put key  || break; done

2. Execute below commands to compact and defragment the etcd backend database:

etcdctl compact `(etcdctl endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9]*')`
etcdctl defrag
etcdctl alarm disarm

3. Repeat Step 1& 2 for 3 to 4 times and you will see each time etcd memory consumption is increasing, and eventually it leads to OOM with more such repetitions.
4. Collect the heap profile from https://:2379/debug/pprof/heap, it will show increased heap usage for raft Entries, which keep on increasing with put entries and never gets released, below is the snippet of it:

File: etcd
Type: inuse_space
Time: Dec 14, 2020 at 2:45pm (IST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top10
Showing nodes accounting for 790.67MB, 98.83% of 800.03MB total
Dropped 49 nodes (cum <= 4MB)
Showing top 10 nodes out of 33
      flat  flat%   sum%        cum   cum%
  497.65MB 62.20% 62.20%   497.65MB 62.20%  github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raftpb.(*Entry).Unmarshal
  282.54MB 35.32% 97.52%   282.54MB 35.32%  github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.(*InternalRaftRequest).Marshal
    5.24MB  0.65% 98.17%     5.24MB  0.65%  github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*streamWriter).closeUnlocked
    5.24MB  0.65% 98.83%     5.24MB  0.65%  github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.startPeer
         0     0% 98.83%   282.54MB 35.32%  github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).Put
         0     0% 98.83%     5.24MB  0.65%  github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).apply
         0     0% 98.83%     5.24MB  0.65%  github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyAll
         0     0% 98.83%     5.24MB  0.65%  github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyConfChange
         0     0% 98.83%     5.24MB  0.65%  github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyEntries
         0     0% 98.83%   282.54MB 35.32%  github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).processInternalRaftRequestOnce
(pprof)

Please let us know if any more details needed to help you to reproduce the issue.

[tangcong]

        // DefaultSnapshotCatchUpEntries is the number of entries for a slow follower
	// to catch-up after compacting the raft storage entries.
	// We expect the follower has a millisecond level latency with the leader.
	// The max throughput is around 10K. Keep a 5K entries is enough for helping
	// follower to catch up.
	DefaultSnapshotCatchUpEntries uint64 = 5000

@pratik-thacker The increase in etcd memory is caused by the DefaultSnapshotCatchUpEntries parameter. If your key/value is 1M, it may occupy etcd memory up to 5G. Is your key/value relatively large? Hope this parameter can be configured?

[pratik-thacker]

Yes our key/values are relatively large, value_size is around 1 MB.

Few further queries:

1. Does etcd raft stores all revisions (upto SnapshotCatchUpEntries) in the memory or it stores latest key value in memory?
2. If it stores all the last 5000 revisions in memory then for each KV of 1 MB it will may consume upto 5G memory.
   And if we store 1000 such uniq keys (having 1MB value size) then will it need 5G * 1000 = 5TB of memory?
3. What is env variable name to configure SnapshotCatchUpEntries?
   I couldn't find this on etcd website and I tried with ETCD_SNAPSHOT_CATCHUP_ENTRIES but it seems a wrong guess.

[tangcong]

Each write request such as put/txn will generate a raft log entry, which has nothing to do with the number of keys. At least 5000 such log entries will be kept, which is not currently configurable. @pratik-thacker

[pratik-thacker]

Thank you for the clarification.
So raft will need memory around 5000 * "average size of put entry", right?

[tangcong]

yes

…

---Original--- From: "pratik-thacker"<notifications@github.com&gt; Date: Mon, Dec 21, 2020 18:42 PM To: "etcd-io/etcd"<etcd@noreply.github.com&gt;; Cc: "Comment"<comment@noreply.github.com&gt;;"tangcong"<tangcong506@foxmail.com&gt;; Subject: Re: [etcd-io/etcd] etcd raft memory usage keep increasing and causing etcd to OOM error (#12548) Thank you for the clarification. So raft will need memory around 5000 * "average size of put entry", right? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.