Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stable release 2.0.7 #1393

Merged
merged 82 commits into from
Mar 17, 2023
Merged

Stable release 2.0.7 #1393

merged 82 commits into from
Mar 17, 2023

Conversation

Dead2
Copy link
Member

@Dead2 Dead2 commented Dec 26, 2022

Changes since 2.0.6:

@Dead2 Dead2 added the Next Stable Targeting next stable release (does not mean accepted) label Dec 26, 2022
@phprus
Copy link
Contributor

phprus commented Dec 26, 2022

Missing commit:
#1267 - fix for bug in 3a5a2c4

@Dead2
Copy link
Member Author

Dead2 commented Dec 26, 2022

Missing commit: #1267 - fix for bug in 3a5a2c4

What do you mean? The commit is in this PR already, and there is no other commit in the source PR.

Edit: Actually the commit is already there. But github does not show it in the above list, only in the commits tab. The list above seems to only show about half the actual commits, weird..

@phprus
Copy link
Contributor

phprus commented Dec 26, 2022

Oh! I am sorry!
For some unknown reason, not all commits and changes (on "Files changed" tab) were loaded for me.

@mtl1979
Copy link
Collaborator

mtl1979 commented Dec 26, 2022

Build breaks initially at 9bb926a ... After that everything is noise.

That commit use zmemcpy_* macros that were not available.

Next break seems to be at 0ce59bf... That's where tests start to fail if I drop the commit above.

deflate.h Outdated Show resolved Hide resolved
Copy link
Collaborator

@mtl1979 mtl1979 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Broken put_short().

test/minideflate.c Outdated Show resolved Hide resolved
KungFuJesus and others added 21 commits December 27, 2022 13:47
Technically, we weren't actually doing this the way C wants us to,
legally.  The zmemcpy's turn into NOPs for pretty much all > 0
optimization levels and this gets us defined behavior with the
sanitizer, putting the optimized load by arbitrary alignment into the
compiler's hands instead of ours.

Backport note: Replaced zmemcpy with direct memcpy, as that is what we
end up with in a later commit anyway.
* This avoids conditional branch when it's known at build time that TZCNT instructions are always supported
It would seem that on some platforms, namely those which are
!UNALIGNED64_OK, there was a likelihood of chunkmemset_safe_c copying all
the bytes before passing control flow to chunkcopy, a function which is
explicitly unsafe to be called with a zero length copy.

This fixes that bug for those platforms.
Sample input from https://www.openwall.com/lists/oss-security/2022/03/26/1.

Co-authored-by: Tavis Ormandy <taviso@users.noreply.github.com>
Co-authored-by: Eric Biggers <ebiggers@kernel.org>
…for small lengths due to shift returning 0.

* Treat 0 byte input as 1 byte input when calculating compressBound and deflateBound
* Test both compressBound() and deflateBound() as those share same code fragment.
Dead2 and others added 14 commits February 10, 2023 13:10
Removed tests for features not supported in 2.0.x:
- cxx related settings, as stable does not use gtest/gbench.
- Emscripten
- Add_subdirectory
- Symbol prefix
- oss-fuzz, their buildfile is incompatible with this branch
…n and congestion.

The free Github Actions VMs have 2 cores, the dedicated s390x VM has 4 cores.
…d the past < chunksize bytes in the window. See #1245.

Co-authored-by: Adam Stylinski <kungfujesus06@gmail.com>

Backported from commit c882034.
* Add __msan_unpoison() calls to DFLTCC inline assembly.
* Make parameter block sizes symbolic constants.
* Move dfltcc() definition after struct dfltcc_param_v0 definition.

Backported from commit 1f5ddcc.
gzsetparams() now returns a Z_STREAM_ERROR in this case.
memLevel 9 would cause deflateBound() to assume the use of fixed
blocks, even if the compression level was 0, which forces stored
blocks. That could result in a bound less than the size of the
compressed data. Now level 0 always uses the stored blocks bound.
A fixed block could be chosen when a stored block was smaller. Now
the smaller of the two is always chosen.
@nmoinvaz
Copy link
Member

All checks pass!

@mtl1979
Copy link
Collaborator

mtl1979 commented Feb 11, 2023

All checks pass!

We're getting closer and closer to next stable release... Still need some testing before we can do official release...

@rhpvorderman
Copy link

I don't know if it is very helpful but for the python bindings I test the compatibility with proper zlib. I simply take all possible compression levels, memory levels and windowbits settings and run them through zlib-ng's deflate and see if zlib's inflate can properly handle them. Then I do the same thing the other way round. This results in thousands of compatibility tests.
I just checked this branch and on Windows x86/64 (msvc), Linux x86/64(gcc) and MacOS x86/64(clang) everything seems to be fine.

Changes since 2.0.6:
- Fix CVE-2022-37434 #1328
- Fix chunkmemset #1196
- Fix deflateBound too small #1236
- Fix Z_SOLO #1263
- Fix ACLE variant of crc32 #1274
- Fix inflateBack #1311
- Fix deflate_quick windowsize #1431
- Fix DFLTCC bugs related to adler32 #1349 and #1390
- Fix warnings #1194 #1312 #1362
- MacOS build fix #1198
- Add invalid windowBits handling #1293
- Support for Force TZCNT #1186
- Support for aligned_alloc() #1360
- Minideflate improvements #1175 #1238
- Dont use unaligned access for memcpy #1309
- Build system #1209 #1233 #1267 #1273 #1278 #1292 #1316 #1318 #1365
- Test improvements #1208 #1227 #1241 #1353
- Cleanup #1266
- Documentation #1205 #1359
- Misc improvements #1294 #1297 #1306 #1344 #1348
- Backported zlib fixes
- Backported CI workflows from Develop branch
@Dead2 Dead2 changed the title Stable pre release Stable release 2.0.7 Mar 7, 2023
@Dead2 Dead2 added Needs testing Please help test this Reviews wanted labels Mar 7, 2023
@Dead2
Copy link
Member Author

Dead2 commented Mar 17, 2023

Heads up: This is getting released later today, unless something comes up in last minute testing.

I'd like it if people would do a quick review of this PR. (You don't need to read every line, just say what you did do; cursory review, tested on x86-64, agree with the proposed changes, etc).
(Merge protection is on, so reviews are also required before merging, and I'd like to avoid having to bypass those protections)

I meant to release this late last week, but I have been out with the flu for a week now, and I am finally able to spend a little time in front of a monitor again.

I have been running this on a production web server for a while now without problems, unfortunately I no longer have access to big server deployments since I am no longer employed (I became 100% disabled recently).
But confidence is high that there are no glaring bugs in this, but corner cases are always the hardest ones to test.

@Dead2
Copy link
Member Author

Dead2 commented Mar 17, 2023

GCC 12.2 on x86-64 i9-9900K (no AVX-512)

2.0.6

 Tool: minigzip   Levels: 1-9
 Runs: 45         Trim worst: 5

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 1     47.618%      1.852/1.881/1.900/0.011        0.897/0.921/0.936/0.010      100,930,174
 2     35.518%      2.716/2.745/2.766/0.013        0.990/1.006/1.021/0.007       75,282,961
 3     34.198%      3.106/3.157/3.206/0.025        0.953/0.979/0.990/0.008       72,485,726
 4     32.928%      3.562/3.622/3.670/0.031        0.922/0.935/0.947/0.007       69,793,995
 5     32.662%      3.817/3.869/3.938/0.034        0.908/0.927/0.938/0.007       69,228,858
 6     32.507%      4.308/4.412/4.473/0.047        0.908/0.927/0.938/0.007       68,901,209
 7     32.255%      5.925/6.066/6.185/0.072        0.887/0.908/0.923/0.009       68,366,800
 8     32.167%      8.655/8.787/9.027/0.092        0.894/0.908/0.920/0.007       68,180,776
 9     32.156%   11.603/11.802/12.225/0.151        0.876/0.907/0.921/0.010       68,156,160

 avg1  34.668%                        5.149                          0.935
 tot                               1853.663                        336.716      661,326,659

   text    data     bss     dec     hex filename
 112085    1296      32  113413   1bb05 libz-ng.so.2

2.0.7

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 1     44.407%      1.777/1.791/1.811/0.009        0.975/0.994/1.008/0.010       94,124,801
 2     35.518%      2.753/2.777/2.798/0.012        1.013/1.036/1.047/0.007       75,282,961
 3     34.198%      3.118/3.180/3.227/0.027        0.989/1.008/1.020/0.008       72,485,726
 4     32.928%      3.585/3.659/3.715/0.034        0.947/0.963/0.975/0.007       69,793,995
 5     32.662%      3.813/3.891/3.959/0.037        0.945/0.958/0.970/0.007       69,228,858
 6     32.507%      4.321/4.452/4.566/0.059        0.928/0.954/0.964/0.007       68,901,209
 7     32.255%      5.977/6.092/6.207/0.062        0.901/0.935/0.951/0.010       68,366,800
 8     32.167%      8.646/8.793/9.029/0.096        0.910/0.928/0.945/0.010       68,180,776
 9     32.156%   11.625/11.800/12.054/0.106        0.903/0.927/0.939/0.008       68,156,160

 avg1  34.311%                        5.159                          0.967
 tot                               1857.406                        348.108      654,521,286

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 1     44.407%      1.779/1.799/1.815/0.010        0.976/0.993/1.008/0.009       94,124,801
 2     35.518%      2.731/2.780/2.799/0.012        1.023/1.038/1.052/0.008       75,282,961
 3     34.198%      3.116/3.179/3.225/0.025        0.982/1.005/1.017/0.009       72,485,726
 4     32.928%      3.599/3.656/3.705/0.028        0.943/0.963/0.976/0.008       69,793,995
 5     32.662%      3.816/3.897/3.961/0.039        0.940/0.956/0.968/0.008       69,228,858
 6     32.507%      4.336/4.436/4.512/0.054        0.928/0.955/0.966/0.008       68,901,209
 7     32.255%      5.949/6.085/6.202/0.067        0.903/0.932/0.945/0.010       68,366,800
 8     32.167%      8.658/8.817/9.079/0.124        0.909/0.928/0.940/0.009       68,180,776
 9     32.156%   11.649/11.815/12.112/0.121        0.908/0.928/0.942/0.009       68,156,160

 avg1  34.311%                        5.163                          0.966
 tot                               1858.591                        347.925      654,521,286

   text    data     bss     dec     hex filename
 111613    1296      32  112941   1b92d libz-ng.so.2

Code size is slightly smaller.
Compression and Decompression is slightly slower overall, but not by much.
Level 1 compression has improved by quite a bit, but it is also 4.5% faster.

@Dead2 Dead2 merged commit a583e21 into stable Mar 17, 2023
@iii-i
Copy link
Member

iii-i commented Mar 20, 2023

Thank you very much for the release! FWIW, I ran some additional tests on s390x, and all is looking good.

@rhpvorderman
Copy link

Thanks very much for releasing. I updated the python bindings to include the new version. Since the wbits bug is fixed these bindings can be used as a one-to-one replacement for python's zlib module.

@Dead2 Dead2 deleted the stable-pre-release branch June 7, 2023 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Needs testing Please help test this Next Stable Targeting next stable release (does not mean accepted) Reviews wanted
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet