[EPIC] Full-Blown Persistent Storage

[tobixdev]

Currently, RdfFusion only has an in-memory storage. While this is good for testing and exploring the ideas behind RdfFusion, at some point we need a persistent storage format for our graphs.

I believe a storage format based on Parquet makes the most sense as DataFusion has excellent support for it. The general approach (sorted quad indices) can be also used with Parquet.

Could be interesting:

• https://sferrada.com/publication/2025-iswc-arenas-guerrero-cottas/

[tobixdev]

Relevant discussion: #113

From my naïve current standpoint, I think RDF Fusion's storage format should be

1. Open, I don't think specing and implementing a custom format makes sense for this project
2. The should be existing support for reading and writing files in the Arrow/DataFusion ecosystem so we have battle-tested and performant implementations that we can rely upon
3. Ideally, the writer implementation would allow us to easily integrate features like transactions and sorting.

Additional (maybe obvious) requirements:
4. Good Performance

Another relevant paper: https://dl.acm.org/doi/10.1145/3749163

Other projects that we could check-out:

• https://tonbo.io/ (LSM tree based on Parquet)

[jonrmayer]

Also, have a look at Lance Format

Lance and Vortex have similar performance characteristics - e.g. Random Access - 100 * parquet performance but I strongly suspect that Vortex gets the win for a graph [ ignorance speaking ]

[jonrmayer]

Interesting opinion/update regarding F3 with the backstory - https://www.linkedin.com/feed/update/urn:li:activity:7380871334068895744/

AnyBlox looks interesting

[tobixdev]

Thanks again for your suggestions! After some thought, I think for now I'll be going with a Parquet-based solution.

However, I am aiming to make the concepts of persistent quad index file-independent and then hopefully one can relatively easily implement another file format and we can compare the performance.