-
-
Notifications
You must be signed in to change notification settings - Fork 253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix deflateBound for HASH_BITS > 15 #1039
Conversation
This check was changed from HASH_BITS != 15 to HASH_BITS < 15 in commit e7bb6db, leading zlib-ng to produce upper bounds that are too small in certain cases.
The reason why this was changed (in the wrong commit it seems) is related to cbc3962. So, we default to 16 now and where 15 was the previous default and also used to be the max, 16 is currently always used unless manually changed in deflate.h. (This might change) 16 tends to compress better than 15, and the original check enabled a calculation with more buffer space when it is not 15. That was obviously correct before, but with the new default being 16 this change would never get the "tight" bound activated. So, perhaps the real problem is that the tight bound calculation is too tight in some cases perhaps? @atdt How did you trigger the problem? What compression level did you use? Was it level 1 (deflate_quick) by any chance? |
@Dead2 Sorry, I should have included a reproduction case. I ran into this in the form of a test failure in Apache Arrow's unit tests (TestGZipInputStream/CompressedInputStreamTest.RandomData). Here's a minimum repro:
With zlib, the output I get is:
With
The Z_OK return value of deflate() with zlib-ng indicates that there was not enough output space. |
@Dead2 The shift amounts are way too high at the bottom of deflateBound().... I'm getting 768, 192 and 0 for the results of the shifts. I would expect the highest number to be closer to 300000. |
@Dead2 I have not been feeling well. Maybe I can take a look at another time. |
@nmoinvaz Health and family first, always 😄 |
This was fixed by #1071 |
This check was changed from HASH_BITS != 15 to HASH_BITS < 15 in commit e7bb6db, leading zlib-ng to produce upper bounds that are too small in certain cases.