Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a pipeline system for compression #10

Open
xiaodaigh opened this issue Oct 19, 2019 · 0 comments
Open

Implement a pipeline system for compression #10

xiaodaigh opened this issue Oct 19, 2019 · 0 comments
Projects

Comments

@xiaodaigh
Copy link
Owner

Thinking about JDF it can actually be thought of as a pipeline

raw data -> blosc compressed/rle compressed -> written

Conceivably, we can have more elaborate pipelines like

raw data -> rle compress -> blosc compress-> written

and before we start, we do not know which compression is better e.g.

raw string data -> string array -> blosc compress -> written
or
raw string data -> rle compress -> blosc compress -> written

so we can have this comparison pipeline concept. So we can compare both compress before writing to disk, but all we have to do is to remember the operations in a pipeline and and then retry them.

@xiaodaigh xiaodaigh added this to To do in v0.3.0 Nov 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
v0.3.0
  
To do
Development

No branches or pull requests

1 participant