-
Notifications
You must be signed in to change notification settings - Fork 0
Add quantization codec and implement compression/decompression of floating point data #31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add quantization codec and implement compression/decompression of floating point data #31
Conversation
|
👋 Thank you for your draft pull request! Do you know that you can use |
77d8ab7 to
2850c86
Compare
|
I think lossy tests are close to passing but due to the issue mentioned in #40 the floating point values don't match exactly: I think we should adjust the tests to compare the data read in by both libraries rather than comparing to the very original data. |
42d9834 to
d55034b
Compare
d3bc227 to
3cee76c
Compare
| """ | ||
|
|
||
|
|
||
| class Quantize(Codec): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we can have this inherit Codec if we aren't going to implement encode and decode as it will fail the ABC check if numcodecs is importable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's fine and we are going to remove references to Codec anyway so we can just change this to not inherit from anything?
| ) | ||
| settings["itemsize"] = tile_data.size // int(np.product(actual_tile_shape)) | ||
|
|
||
| lossless = len(cdata) == 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This might want a better variable name, you can have lossless data in the COMPRESSED_DATA column if it was compressed directly with GZIP without quantization? Am I right in thinking that this is "this tile is losslessly compressed when the others are quantized"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes that's correct
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
gzip_fallback?
Co-authored-by: Stuart Mumford <stuart@cadair.com>
|
@Cadair - I found some time to implement the comments |
Fixes #14
Fixes #40