Streaming?

[cedws]

Is streaming possible with bincode? It looks like all the serialisation functions have a Sized bound, but I'm fuzzy on how exactly this works.

I need to serialise a struct that will sometimes be too large to hold in memory all at one time, because it will have a field that contains file data. The only way I can see around this is to use a BufReader so that bytes can be read as a stream, but BufReader does not derive serde::Serialize.

Thanks.

[TyOverby]

What do you mean by streaming?

Also, even if you managed to serialize the file via streaming, what would you do to deserialize it?

[cedws]

I mean Reader/Writer, the ability to process in chunks. I assume that I would be able to use the deserialize_from function in order to deserialize the large amount of data.

[TyOverby]

are you serializing a bunch of smaller structs?

Rust doesn't deal with uninitialized data very well, so having partially constructed structs isn't really possible.

[cedws]

Not necessarily a bunch, it could be just one. One of the values in the struct will be the contents of a file though, which could be gigabytes, and might not always fit in memory all at once. I think I'm going to look into other methods of... chunked serialisation? I think flatbuffers might do it.

[TyOverby]

Yeah, I don't think that Bincode can help you here.