Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support lazily reading a serialized Vector #233

Closed
rookboom opened this issue Apr 20, 2018 · 2 comments
Closed

Support lazily reading a serialized Vector #233

rookboom opened this issue Apr 20, 2018 · 2 comments

Comments

@rookboom
Copy link

@rookboom rookboom commented Apr 20, 2018

It would be great if it was possible to incrementally read a large file that contains a serialized Vec.

A large file may contain 10 million records and you may not be able to load everything into memory
before processing it. Being able to iterate over elements individually or in chunks would be very helpful.
It would allow for doing something like:

decode_vec(file_reader)
.into_iter()
.filter(|person| person.last_name.starts_with("Jones") )
.map(|person| person.age)
.sum()

Or whatever query one might want to do on a large corpus of structured data.

@dtolnay
Copy link
Collaborator

@dtolnay dtolnay commented Apr 20, 2018

Would this approach work? The example there is very similar to yours. https://serde.rs/stream-array.html

@ZoeyR
Copy link
Collaborator

@ZoeyR ZoeyR commented Apr 1, 2020

Closing this since its been 2 years since response. dtolnay's provided answer seems to be a good way to do this.

@ZoeyR ZoeyR closed this Apr 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.