FoundationDB as the KV store

[mumoshu]

// I'm posting this feature request here by following https://github.com/surrealdb/surrealdb/blob/main/CONTRIBUTING.md#introducing-new-features but if you have another channel for that please let me know!

Would there be any chance that SurrealDB could add support for FoundationDB as a KV store?

My primary motivation is benchmarking at this point- For me, TiKV and FDB are the two biggest transactional KV stores(although they have several differences as explained in places like tikv/tikv#2986). I prefer FDB for its extensive test suite for ensuring correctness.

I'm not yet totally sold on either of those though- It would be great if I could write/run some benchmarks against SurrealDB on TiKV and FDB, compare, and decide which one to use as the backing store for awesome SurrealDB, and encourage other folks to do the same so that we can collaborate and I'm not alone in the Internet who might be using SurrealDB on FDB, hence this feature request 😄

If it happened I believe it might look very similar to one for TiKV. It would also require a new dependency to foundationdb-rs. (I actually have a PoC that compiles successfully

WDYT?

// P.S. Perhaps you've already tried FDB and opted to go with TiKV? Then I'd greatly appreciate it if you could share your experience. Thanks in advance for taking your time.

And lastly... Keep up the great work! I'm impressed to all your work on SurrealDB and looking forward to every "coming-soon" feature you mentioned in the documentation/site!

[tobiemh]

Hi @mumoshu, thanks for this idea and starting the discussion!

We've definitely had an eye on FoundationDB for a long, long time now (since 2015 I believe)!

I am open to having FoundationDB as a potential backing store, albeit the design limitations may result in a number of queries failing in SurrealDB. Is your PoC source code visible anywhere (I'd love to take a look)?

The limitations which were a semi-deciding factor for us going with TiKV are:

Large transactions

Transaction size cannot exceed 10,000,000 bytes of affected data. Keys, values, and ranges that you write are included as affected data. Keys and ranges that you read are also included as affected data, but values that you read are not. Likewise, conflict ranges that you add or remove (using a snapshot read or a transaction option) are also added or removed from the scope of affected data.

If any single transaction exceeds one megabyte of affected data, you should modify your design. In the current version, these large transactions can cause performance issues and database availability can (briefly) be impacted.

Long running transactions

FoundationDB currently does not support transactions running for over five seconds. In particular, after 5 seconds from the first read in a transaction:

• subsequent reads that go to the database will usually raise a transaction_too_old error (although reads cached by the client will not);
• a commit with any write will raise a transaction_too_old or not_committed error.

Long running read/write transactions are a design limitation, see the discussion in Anti-Features.

The full page of design limitations can be found here.

Having said that, FoundationDB does have a very nice key-watching feature, which I wish TiKV had too.

[mumoshu]

Thanks for your reply!

I have a bit more complete reply at #25 (comment).

In addition to that, let me add one thing:

Long running transactions

In case you missed it, just for the purpose of sharing what I know, FoundationDB is trying to push the limits with their new Redwood storage engine. I think it will make more long-running transactions possible with some caveats(like it's not still "unlimited", as it's fundamentally about what to use to store historical keys and values, either main memory or the storage).

It isn't fully documented yet but I can recommend reading https://forums.foundationdb.org/t/foundationdb-summit-2019-redwood-storage-engine-update/1750.
It's envisioned to be a storage engine with a built-in MVCC functionality(FDB currently does MVCC in-memory, in a layer higher than storage engines which is bound to the limited main memory size), although the MVCC functionality isn't exposed to the FDB internals yet in FDB 7.0 or the latest 7.1.

[mumoshu]

a very nice key-watching feature

I find it useful too!

I'd also like to mention that they seem to be working on adding change feeds = range watches 👀
It might be useful for implementing Change-Data-Capture, although it is obviously not a part of SurrealDB's KeyValueStore as of today so I'm not sure how it can be leveraged by it(perhaps for SurrealDB could use changefeeds for FDB KVS and use a SurreaDB native feature for other KVS>

[tobiemh]

In case you missed it, just for the purpose of sharing what I know, FoundationDB is trying to push the limits with their new Redwood storage engine. I think it will make more long-running transactions possible with some caveats(like it's not still "unlimited", as it's fundamentally about what to use to store historical keys and values, either main memory or the storage).

Oh awesome @mumoshu , I did not know about this new storage engine. I will definitely watch that YouTube video!

[mumoshu]

@tobiemh Hey! Thanks a lot for your response ☺️

Large transactions
Long running transactions

Thanks! I'm aware of it and I'm still optimistic about it and that's why I'm going to benchmark the two. I would greatly appreciate it if you could review/validate my idea!

You may already be aware of all these points but please let me explain them fully anyway.

So, my understanding is that it's mainly due to that (1) FDB focuses more on higher throughput FDB queries and wants the FDB layer developer to use snapshot isolation to disable serializable transactions OR the layer to implement its own MVCC on top of FDB and (2)FDB's MVCC for 5 seconds is implemented with non-MVCC data on disk plus the MVCC layers maintained in-memory https://forums.foundationdb.org/t/large-range-scans-avoid-5s-limit/292/6. The main memory is relatively more constrained than the storage hence they heuristically decided to use lower limits.

An example of (1) I know of is https://github.com/losfair/mvsqlite which had implemented its own MVCC backed by SQLite transactions and WAL.

TiKV seems to do MVCC at the storage layer by prefixing RocksDB keys with the kv version https://tikv.org/docs/5.1/reference/architecture/storage/#mvcc. Theoretically, FDB could do the same and there might be pros/cons for doing so. If you weigh more on throughputs you'd better use FDB by compiling it with more generous limits(I thought they are just constants in their source code) and run it on the machines with more memory. If you weigh more on the economy you might better use TiKV.

Considering all, I guess "I" am not yet at the stage where relying solely on TiKV for its potential to handle larger and log-running transactions, although I can't say FDB's better either 😁

That said, I'm still interested in what kind of SurrealDB queries you have in your mind that may fail and be non-functional when run on FDB?

To my humble knowledge, I can imagine:

• a large OLAP query or
• a large atomic schema migration(like updating millions of items' values along with their metadata to reflect the latest schema)
• a large index (re)building done atomically

would have a hard life if they are supposed to be made "serializable" naively on FDB, without SurrealDB's own MVCC layer.

I'm not yet sure if making a whole large KV-store transaction serializable is always practical, though. It needs to write transaction logs/WAL to the disk at least. Should I always let it additionally write versioned or layered KVs for MVCC? It might result in more write amplication in exchange for larger transactions.

If we learned later that TiKV could actually make large transactions serializable without serious problems- it's totally awesome! I might happily use TiKV as the backing store for SurrealDB, or FDB could be updated to do a similar storage-level MVCC to catch up.

// In other words, I do respect TiKV and its authors and maintainers a lot. However until I see real numbers, I can't justify me saying "no hard-coded limit = more scalability". At least from what their respective docs say, I can't stop thinking that they should have similar architectural limits although TiKV seems to have more generous hard-coded limits than FDB.

Well, this turned out to be a very long read. It's my pleasure to discuss this kind of thing as I'm very much interested in this area so I couldn't stop writing this as detailed as I can think. Sorry about that! I hope it's readable.

Thanks in advance for taking your time 🙏

[tobiemh]

Hi @mumoshu first of all thanks for the contribution which has just merged. We'll look to get this feature available in the next release (in the Linux version for the moment).

So, my understanding is that it's mainly due to that (1) FDB focuses more on higher throughput FDB queries and wants the FDB layer developer to use snapshot isolation to disable serializable transactions OR the layer to implement its own MVCC on top of FDB and (2)FDB's MVCC for 5 seconds is implemented with non-MVCC data on disk plus the MVCC layers maintained in-memory https://forums.foundationdb.org/t/large-range-scans-avoid-5s-limit/292/6. The main memory is relatively more constrained than the storage hence they heuristically decided to use lower limits.

Yeah this is my understanding too.

An example of (1) I know of is https://github.com/losfair/mvsqlite which had implemented its own MVCC backed by SQLite transactions and WAL.

This is really interesting. I didn't know of mvsqlite before!

TiKV seems to do MVCC at the storage layer by prefixing RocksDB keys with the kv version https://tikv.org/docs/5.1/reference/architecture/storage/#mvcc. Theoretically, FDB could do the same and there might be pros/cons for doing so. If you weigh more on throughputs you'd better use FDB by compiling it with more generous limits(I thought they are just constants in their source code) and run it on the machines with more memory. If you weigh more on the economy you might better use TiKV.

Yes although I'm not sure compiling it with more generous limits will solve the problem, but that's just a guess!

That said, I'm still interested in what kind of SurrealDB queries you have in your mind that may fail and be non-functional when run on FDB?

• a large OLAP query or
• a large atomic schema migration(like updating millions of items' values along with their metadata to reflect the latest schema)
• a large index (re)building done atomically

Absolutely, these queries are exactly right. With the graph traversal, a complicated query could last an unknown amount of time. In addition, index (re)building (depending on the number of items) could be tricky. To be honest, these are all choices that the developer can make themselves. If they choose to build on top of SurrealDB on FoundationDB, then those limits will be something they would have to understand.

would have a hard life if they are supposed to be made "serializable" naively on FDB, without SurrealDB's own MVCC layer.

I'm not yet sure if making a whole large KV-store transaction serializable is always practical, though. It needs to write transaction logs/WAL to the disk at least. Should I always let it additionally write versioned or layered KVs for MVCC? It might result in more write amplication in exchange for larger transactions.

Yes you're spot on. At the moment the SurrealDB query layer, and the key-value store are completely separate. At the moment that separation is quite important as it enables SurrealDB to be run in many different scenarios. But yes, large schema changes, or large re-indexing jobs could be an issue here. It might be better down the line to have an intermediary layer in between SurrealDB and FoundationDB to provide the MVCC logic.

// In other words, I do respect TiKV and its authors and maintainers a lot. However until I see real numbers, I can't justify me saying "no hard-coded limit = more scalability". At least from what their respective docs say, I can't stop thinking that they should have similar architectural limits although TiKV seems to have more generous hard-coded limits than FDB.

So TiKV (as you probably already know) is being used in some pretty large use-cases and production scenarios. Our own testing, benchmarking, and understanding of the limitations is of course necessary, and something we plan to do extensively in due course.

But that brings me on to a slightly different (but completely related) topic of discussion which you asked on the Discord forum. I'll answer over there... 😀!

[tobiemh]

I forgot to add...

Thanks! I'm aware of it and I'm still optimistic about it and that's why I'm going to benchmark the two.

There are a number of performance improvements which we will be making to SurrealDB in due course. A couple of these are listed on the Github Issues list, and one will be added in due course. We know where the code can be improved, and these improvements will make a big difference to the overall performance of the SurrealDB query and computation layer.

Just to make you aware!

[mumoshu]

Is your PoC source code visible anywhere

I'll publish it in a few days once I confirmed it works at least with my local installation of FDB 😄

[mumoshu]

@tobiemh Hey! I've successfully confirmed it's working and I'm sharing all my changes in a form of a pull request for ease of reviewing.

#37

It would be great if you could take a look. Thanks in advance for taking your time 🙏

[tobiemh]

Hi @mumoshu - pleasure to take a look! Thank you for the contribution and for getting involved! We've got some big plans for SurrealDB and it's great to have you involved at this early stage!