Switch on-disk database from LevelDB

[michaelsproul]

Description

Currently we use LevelDB as our on-disk store, however, the Rust bindings for it are very minimally maintained and missing a few useful features like range queries, i.e. "give me all the values between key X and X + N". See: https://github.com/skade/leveldb

In contrast, the RocksDB bindings are much more actively maintained (updates within the last month), and support more features (including range queries). RocksDB is also meant to be a successor to LevelDB, and is used by Parity: https://github.com/paritytech/parity-ethereum/tree/master/parity/db

Alternatively, we could switch to something other than RocksDB. Suggestions welcome!

[paulhauner]

I remember @AgeManning having some issues with RocksDB and I think I remember @kirk-baird saying that someone at Parity said they wished they used LevelDB instead of Rocks.

My understanding is that Rocks is a bit faster, however it is more susceptible to corruption. This understanding is pretty loose though, I might try and ask around :)

[jtomanik]

Have you guys considered LMDB?

Monero team is using it successfully as well as Mozilla. Here are some considerations behind their choice:
https://monero.stackexchange.com/questions/702/why-did-monero-choose-lmdb-over-alternative-database-types/784

https://mozilla.github.io/firefox-browser-architecture/text/0015-rkv.html

https://mozilla.github.io/firefox-browser-architecture/text/0017-lmdb-vs-leveldb.html

Apart from that Mozilla maintains Rust bindings for LMDB https://github.com/mozilla/rkv

[michaelsproul]

Thanks for the tip @jtomanik, we hadn't! LMDB does look like a strong contender, particularly with bindings maintained by Mozilla.

There's a lot going on with Lighthouse at the moment, but in a few weeks we should have time to do some benchmarks and try some alternatives.

[michaelsproul]

I think we should move forward with this before mainnet. LevelDB has multiple drawbacks:

• Doesn't support read-write transactions (only write batches)
• Doesn't support range queries Actually, it does
• Doesn't run on Windows
• Not actively maintained any more

I've been using LMDB in the slasher (#1567) and have been quite happy with its performance and features. It still might be worth giving RocksDB a try too.

Interesting to note is that TurboGeth uses LMDB, presumably for good reason. We may even want to take inspiration from some of their optimisations 👀 https://github.com/ledgerwatch/turbo-geth/blob/master/docs/programmers_guide/db_walkthrough.MD

[AgeManning]

I can't recall the issues I've had with rocks in the past. I believe a lot of them came from compiling on various linux distros and OS's.