Add lmdb store

[willemolding]

PR summary

This PR adds a new implementation of both CAS and EAV that uses LMDB. This is the main store used by the Monero cryptocurrency as well as being used my Mozilla in the firefox browser. As such it is well supported and has excellent Rust bindings currently maintained by Mozilla.

LMDB maintains a sorted order of keys which, with a clever key naming scheme, allows for a huge optimization in performing queries that match exactly on an entity (the most common type of query). Rather than iterating the entire database it is possible to jump to the beginning of EAVI entries for the entry and read until the next entry.

Benchmarks show comparable performance to PickleDB for CAS and EAV get_all retrievals but a significant improvement in get_exact queries.

test cas::lmdb::tests::bench_lmdb_cas_add         ... bench:      14,914 ns/iter (+/- 638)
test cas::lmdb::tests::bench_lmdb_cas_fetch       ... bench:      12,686 ns/iter (+/- 882)
test eav::lmdb::tests::bench_lmdb_eav_add         ... bench:      29,498 ns/iter (+/- 1,510)
test eav::lmdb::tests::bench_lmdb_eav_fetch_all   ... bench:   2,235,458 ns/iter (+/- 57,434)
test eav::lmdb::tests::bench_lmdb_eav_fetch_exact ... bench:       2,760 ns/iter (+/- 173)


test cas::pickle::tests::bench_pickle_cas_add         ... bench:      12,762 ns/iter (+/- 509)
test cas::pickle::tests::bench_pickle_cas_fetch       ... bench:      12,168 ns/iter (+/- 429)
test eav::pickle::tests::bench_pickle_eav_add         ... bench:      25,724 ns/iter (+/- 815)
test eav::pickle::tests::bench_pickle_eav_fetch_all   ... bench:   2,238,151 ns/iter (+/- 37,428)
test eav::pickle::tests::bench_pickle_eav_fetch_exact ... bench:     119,686 ns/iter (+/- 4,885)

Future questions

• How should we decide the initial memory map size and when/how should this be increased?

Review checklist

• The story has unit or integration tests
• No new bugs, and any tech-debt is identified and justified
• There is enough API documentation (how to use)
• There is enough code documentation (how the code works)

[struktured]

Nice! I was thinking we need to change the api to iterator style rather than in memory collections but one thing at a time.

[willemolding]

Nice! I was thinking we need to change the api to iterator style rather than in memory collections but one thing at a time.

Yeah Ashanti and I talked about that and I had a bit of a crack but not as easy as I first thought. Agreed that should be next though.

[maackle]

This looks great, fundamentally. I made some minor comments about things to change, like the From<&[u8]>, and the DRYness, which would be nice to fix before. Also @AshantiMutinta didn't request changes, but she made some important points that should probably be addressed before merging.

[maackle]

Should be OK though right? Since our stores are all monotonic?

[willemolding]

I believe it opens up the possibility of corrupting the whole database by writing to the wrong places in memory since it doesn't verify before syncing to disk. But Rust should stop us from doing that right?

[thedavidmeister]

keen to see if this translates to something like indexed db in the future in the browser :)

[willemolding]

keen to see if this translates to something like indexed db in the future in the browser :)

Yeah I saw your post in the mattermost. That could probably even better support EAV queries as I believe it has support for custom indexes as well as sorted keys.