Serialization & Deserialization of the tree

[Stentonian]

We likely need this to be super efficient for the use-case where the tree has been created and we want to save it to a file, which we must be able to quickly read back into memory on a new program.

File size could be 200GB for the number of users we are catering for. Serde may not be fast enough. Here are some other options.

• https://parquet.apache.org/ (file format for fast reads)
• https://github.com/apache/arrow-rs (Apache arrow which I think interacts with parquet files) (maybe also https://github.com/jorgecarleitao/arrow2 which is being used by Ozgur for Skar)
• https://www.pola.rs/ (alternative to the above)
• https://duckdb.org/docs/data/parquet/overview.html (we could maybe use this DB as apposed to a hashmap)

[Stentonian]

"Because, you can kind of mimic a hashmap in terms of key value pairs with an arrow table. Serializing/deserializing would be super fast. You can paralleize access to it and it will be way more efficient in terms of memory/space. But you'll take a knock on random access."

[Stentonian]

Going to close this as all serde stuff has been completed