-
Notifications
You must be signed in to change notification settings - Fork 122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make ReadRefReader.buf public #300
Comments
I haven't tried parsing concatenated messages like this, so I'm not sure if the reader is reading only as little as necessary, but if it is, then you should be able to use |
Thanks for the prompt response! From what I've been currently testing, it consumes only what is necessary - and I'm parsing 10k's of concatenated messages, which use various types/lists/maps/extdata. Cursor, unfortunatelly, does not have AsRef trait, therefore, it cannot be wrapped into ReadRefReader. As for the |
|
I think I got it (or did I?), but... it produces an error. pub fn parse<'a>(contents: &'a [u8]) -> Vec<Message<'a>> {
let mut result = Vec::new();
let mut deserializer = Deserializer::from_read_ref(&mut contents.as_ref());
loop {
if let Ok(message) = Message::deserialize(&mut deserializer) {
result.push(message);
}
else {
break;
}
}
result
// ^^^^^^ returns a value referencing data owned by the current function
} |
I'm really sorry, but I'm short of any ideas. Would you be kind enough to provide a short example? Either with |
So, this is working - in a sense that the pub fn parse(contents: &[u8]) -> Vec<Message> {
let mut result = Vec::new();
let cursor = &mut contents.as_ref(); // or &mut &contents
let mut deserializer = Deserializer::from_read_ref(cursor);
loop {
let r = Message::deserialize(&mut deserializer);
if let Ok(message) = r {
result.push(message);
}
else {
break;
}
}
// result
Vec::new()
} However, I still can't return the result from the function. I think I understand the reason, but I don't know how to work around it. Except for maybe rewriting the method as EDIT: let log = std::fs::read(r#"logs\log.msgpack"#).unwrap();
let log = &mut &log[..]; // wtf
let messages = message::parser::parse_binlog(binlog); |
edit: nevermind. I mistakenly thought What I've meant about the cursor, etc. was for
|
But there isn't
Replacing |
I mean this: #[derive(serde::Serialize, serde::Deserialize)]
struct Test {
msg: String,
}
fn main() {
let mut ser = Vec::new();
rmp_serde::encode::write(&mut ser, &Test {msg: "Hello".into()}).unwrap();
rmp_serde::encode::write(&mut ser, &Test {msg: "World".into()}).unwrap();
/////////////////////
let mut reader = ser.as_slice();
let one: Test = rmp_serde::from_read(&mut reader).unwrap();
let two: Test = rmp_serde::from_read(&mut reader).unwrap();
println!("{} {}", one.msg, two.msg);
} You don't need direct use of deserializer or any special buffer manipulation, because |
Thanks, I really appreciate you answering, however, I stressed multiple times that I'm specifically interested in zero-copy deserialization, and this example, unfortunatelly, doesn't work: #[derive(serde::Serialize, serde::Deserialize)]
struct Test<'a> {
msg: &'a str,
}
fn main() {
let mut ser = Vec::new();
rmp_serde::encode::write(&mut ser, &Test {msg: "Hello"}).unwrap();
rmp_serde::encode::write(&mut ser, &Test {msg: "World"}).unwrap();
/////////////////////
let mut reader = ser.as_slice();
let one: Test = rmp_serde::from_read_ref(&mut reader).unwrap();
let two: Test = rmp_serde::from_read_ref(&mut reader).unwrap();
println!("{} {}", one.msg, two.msg);
} |
Ah, yes. I forgot about this complication. |
Thanks @kornelski ! |
Hi,
I want to deserialize stream of MessagePack structures with zero-copy.
But when I create ReadRefReader with:
... and then deserialize a first struct:
I have no way of knowing where did the deserializer end.
However, if I would add this into the
impl<'de, R> Deserializer<ReadRefReader<'de, R>>
:... and used it like this:
It would work.
So the question is - am I missing something? Is there a way how to deserialize streaming MessagePack structs from buffer reference? If not - would it be possible to add the
get_buf()
method? And if not... could you think of a better way how to implement this?The text was updated successfully, but these errors were encountered: