Restoring large databases eats up all the RAM

[magik6k]

When trying to restore a large backup, Badger starts taking lots of memory, eventually crashing with OOM.

• Backup size: 107G (11457947 keys)
• System: Linux (4.20.6, Arch), 64G RAM
• Badger commit: 2f79792

[magik6k]

Heap dump+binary used around 20-30G ram use - https://ipfs.io/ipfs/QmWxZ2L4vUQX5vBuq7hHgipNDnf3FQFdNvNnPhrAKMWApy

https://ipfs.io/ipfs/QmWxZ2L4vUQX5vBuq7hHgipNDnf3FQFdNvNnPhrAKMWApy/profile001.svg

[magik6k]

I also tried setting opts.ValueLogLoadingMode = options.FileIO in badger/cmd/restore.go as I had issues with mmaped pages getting locked up in the past, but it didn't help.

[manishrjain]

Based on the heap, it is using around 5GB of RAM, which is not extreme but not ideal either. I think what's happening is that the restore is going full throttle, without any bounds, i.e. it is creating writes, without considering how many are in fly already. We can add a throttle control, so it doesn't create too many pending writes.

Changing this to case over a buffered channel would be the easiest way to throttle it:

badger/backup.go

Line 152 in 3196cc1

default:

[manishrjain]

There's also the Throttle lib that I wrote, available in y package, which can do this in a better way.

[magik6k]

Might also be worth noting - this ran restoring to/from NFS storage on ZFS HDDs, and the backup file might have been cached in RAM, so writes could be way slower than reads.

[ashish-goswami]

Fixed in #762