Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fst files can be streamed on a row by row basis #15

Open
MarcusKlik opened this issue Feb 1, 2017 · 0 comments
Open

fst files can be streamed on a row by row basis #15

MarcusKlik opened this issue Feb 1, 2017 · 0 comments

Comments

@MarcusKlik
Copy link
Collaborator

MarcusKlik commented Feb 1, 2017

That would involve creating a fst file-connection object (similar to base-R file method). With that object data can be streamed row-by-row until the file is depleted (or the connection is closed). The binary file format allows for streaming from compressed fst files as well. In addition, a connection object could also be used to stream to a fst file. The fst format needs to accommodate multiple chunks for that options and the connection object needs a (custom sized) buffer for maximum performance.

@fstpackage fstpackage added this to the In read algorithms milestone Feb 1, 2017
@MarcusKlik MarcusKlik removed this from the Interface milestone Sep 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants