Where and how to persist blobs?

[sandreae]

The actual data of blobs published and replicated in pieces (max 256KB in size). These pieces need to be persisted for replication and serving the resultant blob files over a http server. I can see 2 possible solutions for the persistence of blobs+/pieces.

1. Store pieces in the database: during replication we can retrieve the pieces easily from the database, when a request for the blob arrives at the http server we retrieve all the pieces and concatenate into the complete blob.
2. Store pieces in the database and blobs on the filesystem: for replication pieces come from the database, for serving via http server the blobs come from the file system.

The downside of 2) is that the blobs are stored twice, taking up double the space on a device. The downside of 1) is that we need to handle the file serving logic ourselves, whereas if the files are directly on the system then we can use https://docs.rs/tower-http/latest/tower_http/services/struct.ServeDir.html.

[sandreae]

Maybe there's an option 3:

3. Store complete blobs on fs, don't store pieces in the database: during replication we'd need to break them up again into pieces....

[sandreae]

Ok, it's likely option 2) we'll do, but to clarify, we'll store the blob pieces in their encoded form in the database (for easy replication) and then store the full blobs on the file system (for serving over http). This still means there is some duplication, but it seems worth it for saving on regularly encoding+/decoding blob data during replication.