Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

possibility to have a Streaming serializer ? #345

Closed
allan-simon opened this issue Aug 20, 2017 · 4 comments
Closed

possibility to have a Streaming serializer ? #345

allan-simon opened this issue Aug 20, 2017 · 4 comments
Labels

Comments

@allan-simon
Copy link

@allan-simon allan-simon commented Aug 20, 2017

Imagine I have a huge request from my database that I would like to serialize as a json array to send it over the network

currently (if i haven't missed anything), I have to accumulate my objects into a vector , which I then will be able to serialize in one time.

it's fine most of the time but here looking to the size of the vector

  • my program needs to keep a copy in memory of all the data before being able to send a single byte (so huge RAM consumption)
  • it needs to wait the vector creation to be completed to sent the first byte (so huge latency)

If Serde was to provide a way to serialize things as it go, I would be able to something like that

stream.write(serializer.start_array())
for one_row from all_database_rows.iter() {
      let my_object = object_from_row(one_row);
      stream.write(serializer.array_item(my_object));
}
stream.write(serializer.end_array())
@oli-obk

This comment has been minimized.

Copy link
Member

@oli-obk oli-obk commented Aug 20, 2017

This is possible already. You need a helper type, but inside that type's Serialize impl you can call serialize_seq, which gives you an object you can keep adding elements one by one by calling serialize_element on it. Once you are done, you call the end method

@oli-obk oli-obk added the support label Aug 20, 2017
@allan-simon

This comment has been minimized.

Copy link
Author

@allan-simon allan-simon commented Aug 20, 2017

thanks for your answer, as i'm still a bit new with both rust and serde I just want to confirm some point:

my helper type actually would hold for example the rust-postgres' "Rows" (which is an iterator)
and serializing it will only then unroll the iterator right ? which solve the problem of having an intermediate Vec

in that case it's great but actually only solve one part of my "problem" , the second is that now I still have an intermediate string representing the whole array serialized as json, and I would like to know if i can get rid of it completly by directlying writing in the stream rather than in the String

@oli-obk

This comment has been minimized.

Copy link
Member

@oli-obk oli-obk commented Aug 20, 2017

my helper type actually would hold for example the rust-postgres' "Rows" (which is an iterator)
and serializing it will only then unroll the iterator right ?

Yes

he second is that now I still have an intermediate string representing the whole array serialized as json, and I would like to know if i can get rid of it completly by directlying writing in the stream rather than in the String

That's possible by using to_writer instead of to_string

@dtolnay

This comment has been minimized.

Copy link
Member

@dtolnay dtolnay commented Aug 20, 2017

I wouldn't bother with a wrapper type:

let rows = /* whatever iterator */;
let out = std::io::stdout();

let mut ser = serde_json::Serializer::new(out);
let mut seq = ser.serialize_seq(Some(rows.len()))?; // or None if unknown
for row in rows {
    seq.serialize_element(&row)?;
}
seq.end()?;
@dtolnay dtolnay closed this Nov 19, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
You can’t perform that action at this time.