Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix gzip Decompression Support #8335

Merged
merged 2 commits into from
Aug 29, 2023
Merged

Conversation

BJap
Copy link

@BJap BJap commented Aug 27, 2023

Currently, gzip decompression fails. I noticed when trying to decompress RESTful content bytes in my project that uses gzip for content-type that the headers are not read correctly and I get a failure message of -3.

I suspected I could get around this by slicing the byte array of the raw content. I came across a comment from @FoamyGuy , whom knowingly or not, also found this bug, and had the same idea. That workaround also works in my own code.

I know there are some tests in tests/circuitpython/zlib_decompress.py, but not sure how these things are done here.

My project partner and I worked together to come up with the solution shown in this PR. The tested inputs used are the the same text input compressed each using deflate gzip, and zlib. The new code can reverse each into the original decompressed text. These are included below:


deflate: dtst.txt

b'\x0b\xc9HU(,\xcdL\xceVH*\xca/\xcfSH\xcb\xafP\xc8*\xcd-(V\xc8/K-R(\x01J\xe7$VU*\x94gd\x96\xa4*\xa4\xe4\xa7+\x18\x1a\x19\x9b\x98\x9a\x99[X\x1a(\x94d\xe6\xa6\x16\xeb\x01\x00'


gzip: gtst.txt

b'\x1f\x8b\x08\x08\x94\xaf\xead\x00\x03tst.txt\x00\x0b\xc9HU(,\xcdL\xceVH*\xca/\xcfSH\xcb\xafP\xc8*\xcd-(V\xc8/K-R(\x01J\xe7$VU*\x94gd\x96\xa4*\xa4\xe4\xa7+\x18\x1a\x19\x9b\x98\x9a\x99[X\x1a(\x94d\xe6\xa6\x16\xeb\x01\x00\xbe\x98\x07*C\x00\x00\x00'


zlib: ztst.txt

b'x\x9c\x0b\xc9HU(,\xcdL\xceVH*\xca/\xcfSH\xcb\xafP\xc8*\xcd-(V\xc8/K-R(\x01J\xe7$VU*\x94gd\x96\xa4*\xa4\xe4\xa7+\x18\x1a\x19\x9b\x98\x9a\x99[X\x1a(\x94d\xe6\xa6\x16\xeb\x01\x00.x\x16\xb8'


decompressed: tst.txt

b'The quick brown fox jumps over the lazy white dog 1234567890 times.'

@gamblor21
Copy link
Member

I have not got a chance to test this yet, but just wanted to leave my quick comment after looking it over.
I had left GZIP on the original as we were discussing better ways to include it. I do not see any downside to including this change to allow those who want to the ability to use it a bit easier.

Issue #6284 has some discussion about changes in micropython about uzlib and at some point it is worth looking at those and an overall plan for decompression in circuitpython. I just do not have the time to dig into that deeper now.

@tannewt
Copy link
Member

tannewt commented Aug 28, 2023

Isn't this implementation being used: https://github.com/adafruit/circuitpython/blob/main/shared-bindings/zlib/__init__.c#L51 ?

Copy link
Member

@tannewt tannewt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks!

@tannewt tannewt merged commit 7c62de3 into adafruit:main Aug 29, 2023
355 checks passed
@BJap BJap deleted the zlib-decompress-bug branch August 29, 2023 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants