add type checking #14
Conversation
| //! Shardio storage is intended for transient use within a single processing task, not for archival purposes. | ||
| //! You can't read shardio files created by binaries for a different Rust version, | ||
| //! or where type definitions have changed. It recommended | ||
| //! to only use interact with a shardio file from a single Rust binary. |
There was a problem hiding this comment.
I am a bit concerned about this because if I have a unit test that takes in a shard file as input (let's say this is a unit test for a stage code), won't that break when we upgrade the toolchain? Or should we not make such unit tests?
There was a problem hiding this comment.
yeah good point. I thought that bincode also could change formats between versions, but it looks like bincode should be forward compatible with new Rust versions and new bincode versions as long as the definition of you type doesn't change. When you have a type mismatch, the most common thing is to get a 'failed to fill buffer' error or a crazy big allocation that panics without a good stack trace. So maybe using https://docs.rs/bincode/1.1.4/bincode/struct.Config.html#method.limit & setting the limit to a reasonable value, and returning a better error message if the limit is hit is a good compromise.
No description provided.