Vectors are deserialized with spare capacity

[cswinter]

Thanks for a great library!

Serializing/deserializing a Vec<u8> with len > 4096 will give a Vec with capacity equal to a power of two. This is undesirably for use cases where the deserialized data is not modified. Is there a way to deserialize a Vec without allocating spare capacity?

Reproduce with https://github.com/cswinter/bincode_overalloc:

extern crate bincode;

use bincode::{serialize, deserialize};

fn main() {
    for i in 0..(1<<30) {
        let vec = vec![0u8; i];
        assert_eq!(vec.len(), vec.capacity());
        let serialized = serialize(&vec).unwrap();
        let deserialized: Vec<u8> = deserialize(&serialized).unwrap();
        if deserialized.len() != deserialized.capacity() {
            println!("{} {}", deserialized.len(), deserialized.capacity());
        }
    }
}

[ZoeyR]

You could make deserialized be a &[u8] instead of a Vec<u8> if you aren't going to modify the data.

[cswinter]

I see. In this case, I want to return the deserialized value and keep it alive permanently/until eviction. I can't think of a good way of achieving that without having ownership of the value.

[JoshMcguigan]

This functionality actually lives in the root serde crate, not in bincode. It looks like they purposefully only pre-allocate up to len 4096, and after that rely on the standard allocation behavior of the vec type, which explains the behavior you are seeing.

https://github.com/serde-rs/serde/blob/97920be33aee4019bf7bd86bc3cf957a4de91413/serde/src/de/impls.rs#L862
https://github.com/serde-rs/serde/blob/97920be33aee4019bf7bd86bc3cf957a4de91413/serde/src/private/de.rs#L198

I found this issue on the serde repo discussing this behavior, with a possible work around.

[ZoeyR]

Closing since this is behavior from serde itself.