Add support for Gzip and Zlib compression #51
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This allows the S3 active storage server to work with data that was compressed using one of these algorithms before it was written to the object store. Compression is configured using the optional `compression` field in the API JSON request data, and if present should be set to "gzip" or "zlib". We are using the standard flate2 library for decompression. There may be more performant options to consider in future, but this works well as a first pass, and changing in future will not affect the API.
When data is compressed, the size parameter refers to the size of the compressed data. Typically this is not equal to the size of the uncompressed data, so we can't validate it against the data type size. This change skips initial size/dtype validation when compression is applied, instead performing it once the data has been decompressed. It also adds an additional validation that the size matches the shape, when a shape has been specified.
markgoddard
changed the title
This allows us to more easily support compression algorithms that have decompression parameters.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This allows the S3 active storage server to work with data that was
compressed using one of these algorithms before it was written to the
object store.
Compression is configured using the optional
compressionfield in theAPI JSON request data, and if present should be set to "gzip" or "zlib".
We are using the standard flate2 library for decompression. There may be
more performant options to consider in future, but this works well as a
first pass, and changing in future will not affect the API.