Binary Serialization of the Cache File to save space?

[JD-The-65th]

Hey, I found your project cool, but noticed in the video you're saving the cache as a plaintext file, which could possibly be an issue for filesystems with very large amounts of data. Can you consider using Binary Serialization of the Cache file? I asked an A.I. search engine about this, and it referenced the serde_json and bincode crates.

[conaticus]

Hey thanks for the suggestion! Yes this would absolutely be better than storing the mass JSON in the cache file. I got a little lazy and started rushing everything towards the end to get the video done.

This should definitely be a consideration moving forward if I do come back to this project :)

[RaphGL]

if we're going this route why not also encrypt the serialized binary with something like zstd? zstd is so fast that some linux distros even compress the entire RAM with it so more the system can have more space available.

Also looking into the cache loading function:

pub fn load_cache(state_mux: &StateSafe) {
    let state = &mut state_mux.lock().unwrap();
    let file_contents = fs::read_to_string(CACHE_FILE_PATH).unwrap();
    state.drive_cache = serde_json::from_str(&file_contents).unwrap();
}

Depending on the user's system this can result in big memory usage, so maybe instead of loading the whole thing in, the cache could be lazily loaded using something like Rust's iterator, which is itself lazy. We could get something that could be iterated over and load and unload a few chunks of memory at a time without having to care too about manually handling it.

[ProtogenDelta]

if we're going this route why not also encrypt the serialized binary with something like zstd? zstd is so fast that some linux distros even compress the entire RAM with it so more the system can have more space available.

Compress, not encrypt, but yes, this does look like a good idea. Had tested this on a local branch with deflate and decided against it for performance reasons, but zstd is looking like a much better option. Will take a look at it later today.

[ProtogenDelta]

Depending on the user's system this can result in big memory usage, so maybe instead of loading the whole thing in, the cache could be lazily loaded using something like Rust's iterator, which is itself lazy. We could get something that could be iterated over and load and unload a few chunks of memory at a time without having to care too about manually handling it.

This might be suitable for another issue.

[ProtogenDelta]

#31 merged.