possible Issues(?) bumping etcd 8 GB upper limit (or not!)

[dims]

What would you like to be added?

Last night i took a dive in the 8/10 GB limits in etcd to see why we recommend what we recommend

First order of business was current state:

• MaxQuotaBytes (8GB) at

  etcd/server/storage/quota.go

  Line 33 in 822eeaf

  MaxQuotaBytes = int64(8 * 1024 * 1024 * 1024) // 8GB

• initialMmapSize (10GB)

  etcd/server/storage/backend/backend.go

  Line 42 in 822eeaf

  initialMmapSize = uint64(10 * 1024 * 1024 * 1024)

• bopts.InitialMmapSize

  etcd/server/storage/backend/backend.go

  Line 178 in 822eeaf

  bopts.InitialMmapSize = bcfg.mmapSize()

Configurable from command line:

• quota-backend-bytes https://github.com/search?q=repo%3Aetcd-io%2Fetcd%20QuotaBackendBytes&type=code and https://github.com/etcd-io/etcd/pull/4901/files

Then saw the following being logged:

• "quota-maximum-size-bytes" : https://github.com/search?q=repo%3Aetcd-io%2Fetcd%20MaxQuotaBytes&type=code
• "quota-size-bytes" : https://github.com/search?q=repo%3Aetcd-io%2Fetcd%20QuotaBackendBytes&type=code

Issues:

• Proposal: Increase quota-backend-bytes default to 8GB #9771
• Query : quota-backend-bytes max value #7045

Also a couple of discussions:

• https://groups.google.com/g/etcd-dev/c/vCeSLBKC_M8/m/2pPBXpc9BwAJ
• https://etcd.io/docs/v3.4/faq/#what-does-the-etcd-warning-snapshotting-is-taking-more-than-x-seconds-to-finish--mean

interesting tidbit from google groups discussion

we can recover 2GB of data within 20 seconds on good hardware. We cannot do that for 1TB data due to today's hardware limitation

here's where we added 10% head room:

• https://github.com/etcd-io/etcd/pull/7525/files

Interesting tidbit from ^^^ is Mechanism for permitting mmaps > 10GB.

There are a few references to things > 10 GB, example:

• Inconsistent revision and data occurs #13766 (comment) points to 17GB (--quota-backend-bytes 17179869184)
• https://fossies.org/linux/cilium/Documentation/operations/performance/scalability/report.rst (quota-backend-bytes=17179869184)

Our won @ahrtr says If your VM has big memory (e.g. 64GB), it’s OK to set a value > 8GB. in https://etcd.io/blog/2023/how_to_debug_large_db_size_issue/

the TL;DR is that back in December 2016, we took the MTTR as a guide given the the then status of hardware to come up with a self-imposed limits for the different values we then cargo culted into hard limits. Also the lower limit of 2GB was to support 32bit architectures for windows and arm.

Is this right? Anything else i am missing?

[dims]

I think i am trying to convince myself that it's ok to bump quota-backend-bytes :)

[ahrtr]

Thanks @dims for raising this topic.

We need performance/benchmark data to support the decision.

I may spend some time to optimize boltDB in the next couple of weeks/months. FYI etcd-io/bbolt#401. I am struggling to downloading the huge boltDB file (about 100GiB) from google drive for now.

[ahrtr]

I did some benchmark test on 16GB boltDB files, and the performance looks pretty good. So the bbolt isn't a blocker for bumping quota-backend-bytes to a value <= 16G.
FYI. etcd-io/bbolt#401 (comment)

The only concern is snapshot transport and restoring from the snapshot at a slow follower. We don't have such data yet.

[ahrtr]

Note the 8GB MaxQuotaBytes isn't a hard limitation. Users can still set a value bigger than 8GB to --quota-backend-bytes; etcd just prints a warning in this case.

[dims]

/close

[ahrtr]

thx @dims for the confirmation.

[rchincha]

Anyone aware of recent benchmarks with spinning disks vs SSDs?

Also, dgraph-io/badger#1668 - an option for write-heavy operations. However if the bottleneck is snapshot/restore, not sure it would help.