Provide simple option for limiting total memory usage

[gonzojive]

What version of Go are you using (go version)?

$ go version
go version go1.14 linux/amd64

What version of Badger are you using?

v1.6.0

Does this issue reproduce with the latest master?

As far as I know, yes.

What are the hardware specifications of the machine (RAM, OS, Disk)?

32 GB RAM
AMD Ryzen 9 3900X 12-core, 24-Thread
1 TB Samsung SSD

What did you do?

Used the default options to populate a table with about 1000 key/val pairs where each value is roughly 30MB.

The badger database directory is 101GB according to du. There are 84 .vlog files.

When I start my server up, it quickly consumes 10 GB of ram and dies due to OOM. dmesg output:

[654397.093709] Out of memory: Killed process 15281 (taskserver) total-vm:20565228kB, anon-rss:12610116kB, file-rss:0kB, shmem-rss:0kB

What did you expect to see?

I would expect the database to provide a simple option to limit memory usage to an approximate cap.

What did you see instead?

1. The recommended mechanism of tweaking a many-dimension parameter space is confusing and hasn't worked for me.

2. The memory related parameters are not explained in much detail. For example, the docstring for options.MemoryMap doen't indicate roughly how expensive MemoryMap is vs FileIO.

3. I haven't managed to successfully reduce memory usage using the following parameters:

func opts(dbPath string) badger.Options {
	return badger.DefaultOptions(dbPath).
		WithValueLogLoadingMode(options.FileIO).
		WithTableLoadingMode(options.FileIO).
		WithNumMemtables(1)
}

I can create an example program if the issue is of interest.

[Kleissner]

Agree. We use it for a simple key-value lookup with a couple of billion records (database directory is 700 GB).

It uses about 200 GB of RAM which is unacceptable. The culprit are memory mapped files according to Process Explorer.
Good thing we have a lot of RAM, but there should be an easy well-defined max memory limit to set.

[gonzojive]

I am attempting to restore from backup and running out of memory. Here's a memory profile:

latest options:

func badgerOpts(dbPath string) badger.Options {
	return badger.DefaultOptions(dbPath).
		WithValueLogLoadingMode(options.FileIO).
		WithTableLoadingMode(options.FileIO).
		WithNumMemtables(1).
		WithCompression(options.Snappy).
		WithKeepL0InMemory(false).WithLogger(&gLogger{})
}

[jarifibrahim]

@gonzojive I will have a look at the high memory usage. What is the size of your data directory and the size of the backup file?

[gonzojive]

The backup file is 100GB (99,969,891,936 bytes)

In this case, backup.go's Load function. seems to be a major offender. It does not account for the size of the values at all. Added logging shows huge key/value accumulation and no flushing:

I0326 09:16:43.884962    5195 taskstorage.go:147] not flushing with 1269 entries, 73.2K key size, 3.2G combined size, 9.6M limit

I'm guessing there are many places where value size is not accounted for when making memory management decisions.

[gonzojive]

I modified backup.go to flush when accumulated key + value size exceeds 100 MB. I can send a pull request for this at some point.

Before backup.go modifications the process consumes memory until the OS kills it:

After:
(it's basically flat at 21 GB).... but I didn't manage to grab a screenshot because of Ubuntu/gnome flakiness.

When I set the threshold to 500MB instead of 100 MB, memory usage still causes a crash for some reason.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gonzojive]

Although I have a fix for the backup restoration issue, this issue as a whole has not been addressed.

I'm not aware of what causes badger to take up the amount of memory that it does. That seems like the first step towards introducing a flag for setting a fixed memory limit. May someone from the badger team weigh in?

[jarifibrahim]

I'm not aware of what causes badger to take up the amount of memory that it does. That seems like the first step towards introducing a flag for setting a fixed memory limit. May someone from the badger team weigh in?

The amount of memory being used depends on your DB options. For instance, each table has a bloom filter and these bloom filters are kept in memory. Each bloomfilter takes up 5 MB of memory. So if you have 100 GB of data, that means you have (1001000/64) = 1562 tables, and 15625 MB is about 7.8 GB of memory. So your bloom filters alone would take up 7.8 GB of memory. We have a separate cache in badger v2 to reduce the memory used by bloom filters.

Other things that might affect memory usage is the table loading mode. If you set the table loading mode to fileIO, the memory usage should reduce but then your reads would be very slow.