How do I decode binary data?

[Svedrin]

Hi,

I'm implementing an auth mechanism based on bson and sodiumoxide, and I'm having trouble decoding the challenge. Sodiumoxide treats plaintexts as a &[u8], so I'd like to use a binary string instead of a Unicode string for the challenge, but I can't seem to get bson to decode it correctly.

Code

With a "standard" string, everything works fine:

#[derive(Serialize, Deserialize, Debug)]
pub struct AuthChallenge {
    pub challenge: String,
}

// [...]

    let challenge: AuthChallenge = bson::from_bson(
        bson::Bson::Document(
            bson::decode_document(
                &mut Cursor::new(&message.payload[..])
            ).chain_err(|| "Could not decode bson")?
        )
    ).expect("Decoding failed");
    info!("Challenge: {:?}", challenge.challenge);

This yields:

2018-03-08T08:22:04+01:00 - INFO - Received message ReqAuthentication (payload: [53, 0, 0, 0, 2, 99, 104, 97, 108, 108, 101, 110, 103, 101, 0, 33, 0, 0, 0, 49, 56, 55, 54, 50, 98, 57, 56, 98, 55, 99, 51, 52, 99, 50, 53, 98, 102, 57, 100, 99, 51, 49, 53, 52, 101, 52, 97, 53, 99, 97, 51, 0, 0])
2018-03-08T08:22:04+01:00 - INFO - Challenge: "18762b98b7c34c25bf9dc3154e4a5ca3"

However, if I changeString to Vec<u8> in the struct and change the server side to send a binary string (5) instead of a standard string (2), I get this:

2018-03-08T08:28:12+01:00 - INFO - Received message ReqAuthentication (payload: [53, 0, 0, 0, 5, 99, 104, 97, 108, 108, 101, 110, 103, 101, 0, 32, 0, 0, 0, 0, 54, 55, 98, 54, 100, 53, 50, 99, 50, 101, 48, 51, 52, 52, 56, 49, 98, 52, 57, 101, 102, 51, 56, 56, 101, 100, 100, 54, 51, 98, 50, 102, 0])
thread 'main' panicked at 'Decoding failed: InvalidType("a sequence")', /checkout/src/libcore/result.rs:906:4

I'm also having trouble encoding the signature, because to_bson would always complain that there are no unsigned types in bson, so I ended up doing the encoding manually:

    bson::Bson::Binary(
        bson::spec::BinarySubtype::Generic,
        Vec::from(&signature[..])
    )

Am I doing it wrong, or does bson not currently support binary strings correctly? Can I help in fixing it somehow?

[zonyitoo]

If I understand your problem correctly, what you mean is decoding data with

#[derive(Serialize, Deserialize, Debug)]
pub struct AuthChallenge {
    pub challenge: Vec<u8>,
}

will failed with Decoding failed: InvalidType("a sequence")?

I am trying to reproduce it ...

[zonyitoo]

Ok. I know why you have to:

I'm also having trouble encoding the signature, because to_bson would always complain that there are no unsigned types in bson, so I ended up doing the encoding manually:

That's because you are trying to serialize AuthChallenge to bson, which contains a challenge field with type Vec<u8>. According to Serde doc, Vec<u8> will be treated the same as Vec<T> and processed by seq. So serializer will iterate this Vec and serialize each u8 value one by one.

And of course, BSON does not support any kind of unsigned value, so u8 is not supported. And that's why you get an error UnsupportedUnsignedType.

Solution? Hmm...

---

And for the same reason, you cannot deserialize Bson::Binary with Vec<u8>.

[Svedrin]

outch... I know next to nothing about serde, but sounds like we're hitting a design limitation here... Can you catch Vec<u8> in to_bson and return the same thing I used? that still won't get the deserialization part fixed though, and it's only really useful if both work.

[Svedrin]

I've switched my code to use CBOR now, and I found out that I had to also use serde_bytes for the encoding to work, like so:

#[derive(Serialize, Deserialize, Debug)]
pub struct AuthChallenge {
    #[serde(with = "serde_bytes")]
    pub challenge: Vec<u8>,
}

Maybe this would also work with BSON, but the CBOR module's API is also a lot simpler, so I'm sticking with it. Sorry about that :/

[zonyitoo]

Never mind, thank you for pointing this problem out. :)

[TypicalFence]

Hello all, serde_bytes does not work in conjunction with bson for me, which is a shame.

Has someone tried using it and can confirm that it is working?

Maybe I am just doing something wrong...

[zonyitoo]

@AlexFence Can you provide an example for your problem?

[TypicalFence]

@zonyitoo here is my rather long but complete example.

as @Svedrin pointed out:

Maybe this would also work with BSON

So I tried it out.

But I was unsure if I did something else wrong, which is why I now made this small example.

The following code is a minimal example for what I am trying to do, there is much more data in my actual project, but this is only a example.

Its basically reads in a image file (in the example "test.jpg") and stores that alongside other data in a bson.
In the example the other data is just a string called "name". After storing it inside a bson file, I would like to be able to open it at a later point.

Also I used those depndencies:

[dependencies]
serde = "1.0.10"
serde_derive = "1.0"
bson = "0.11"
serde_bytes = "0.10"

extern crate serde;
#[macro_use]
extern crate serde_derive;
extern crate serde_bytes;
extern crate bson;

use std::io::{Read, Write};
use std::path::Path;
use std::fs::File;
use std::error::Error;

#[derive(Serialize, Deserialize, Clone)]
pub struct SpecialImage {
    #[serde(with = "serde_bytes")]
    pub data: Vec<u8>,
    pub name: String,
}

impl SpecialImage {

    pub fn from_file(p: &Path, name: String) -> Result<Self, std::io::Error> {
        let mut file = File::open(p)?;
        let mut data = Vec::new();
        file.read_to_end(&mut data);

        Ok(Self {data, name})
    }

    pub fn open(p: &Path) -> Result<Self, Box<Error>> {
        let mut file = File::open(p)?;
        let document = bson::decode_document(&mut file)?;
        let u = bson::from_bson(bson::Bson::Document(document))?;

        Ok(u)
    }

    pub fn save(&self, p: &Path) {
        let bson_data = bson::to_bson(&self).unwrap();
        if let bson::Bson::Document(document) = bson_data {
            let mut buf = Vec::new();
            bson::encode_document(&mut buf, &document).unwrap();
            let mut f = File::create(p).unwrap();
            f.write_all(&buf);
        }
    }
}

fn main() {
    let my_img = SpecialImage::from_file(Path::new("test.jpg"), "test".to_string()).unwrap();
    my_img.save(Path::new("test.bson"));

    let my_bson_img = SpecialImage::open(Path::new("test.bson")).unwrap();

    if my_img.name == my_bson_img.name {
        println!("yay!")
    }
}

Sadly I still get a UnsupportedUnsignedType Error.

As I understand it serde_bytes would handle the bytes in some friendly way to make it work.
But I am still a beginner regarding that kind of stuff, so I am not really sure.

[zonyitoo]

This issue was never fixed... @AlexFence please try bson with the latest commit.

[TypicalFence]

@zonyitoo by running my example I get the expected result

fence@fence-desktop ~/c/r/example> cargo r
    Finished dev [unoptimized + debuginfo] target(s) in 0.03s
     Running `target/debug/example`
yay!

It appears to be working now!
thank you for your quick support!

[zonyitoo]

@kyeah I have updated bson to v0.12.0 with all the latest dependencies. Can mongo driver check if there are any problems?

This issue is fixed in v0.12.0. Will be published to crates.io later.