Update ZLib-Intel to v1.2.11

[vkvenkat]

This PR fixes #28013 and updates ZLib-Intel to v1.2.11 as available in https://github.com/jtkukunas/zlib.

The below performance numbers were captured with the Canterbury Corpus workload on a machine with an Intel® Core™ i7-8700K CPU on Windows.

When compared to ZLib-vanilla v1.2.11:

• Compression performance is up to 36% faster. This speedup is similar to the improvements made in PR Build clrcompression.dll with optimized Zlib-Intel #5674.
• Decompression performance is up to 31% faster. This is net new speedup using decompression optimizations in ZLib-Intel v1.2.11.

In addition, this release has security and bug fixes from the latest upstream ZLib.

Here is a detailed performance comparison for ZLib-Intel v1.2.11 (https://github.com/jtkukunas/zlib) vs ZLib v1.2.11 (https://github.com/madler/zlib) by streams & compression levels:

Here are the compression ratios for DeflateStream with the Fastest compression level:

CompressionLevel	Library	Initial Size(KB)	Compressed Size(KB)	Ratio (Compressed/Initial)
Fastest	ZLib-vanilla v1.2.11	2936	976	0.33
Fastest	ZLib-Intel v1.2.11.1	2936	944	0.32

Here are the compression ratios for DeflateStream with the Optimal compression level:

CompressionLevel	Library	Initial Size(KB)	Compressed Size(KB)	Ratio (Compressed/Initial)
Optimal	ZLib-vanilla v1.2.11	2936	850	0.29
Optimal	ZLib-Intel v1.2.11.1	2936	869	0.30

Similar to PR #5674, there are small increases in the compression ratio for Optimal.

@ianhays @joshfree @atsushikan PTAL

[Wraith2]

From the PR you linked:

Please note that due to a gzip incompatibility I have modified the library to no longer use the agressive deflate_quick strategy and instead use the traditional deflate_fast strategy for CompressionLevel Fastest. This causes performance for CompressionLevel Fastest to be similar to Zlib-Adler.

Is this no longer needed?

[danmoseley]

@ianhays

[ianhays]

FYI: @ahsonkhan

[joshfree]

LGTM. @ianhays @ahsonkhan can you please complete some loop runs / stress testing on this change?

[stephentoub]

I'm curious, what causes some of these copyright dates to differ, e.g. https://github.com/dotnet/corefx/pull/32732/files#diff-d1964061700ad02918d1ccd83e7e4a86R2 being updated to 2016 and this being updated to 2017?

[vkvenkat]

The copyright headers (including the dates) were retained from the upstream ZLib for existing files that we modified.

[vkvenkat]

From the PR you linked:

Please note that due to a gzip incompatibility I have modified the library to no longer use the agressive deflate_quick strategy and instead use the traditional deflate_fast strategy for CompressionLevel Fastest. This causes performance for CompressionLevel Fastest to be similar to Zlib-Adler.

Is this no longer needed?

@Wraith2 The deflate_fast strategy is the current default for CompressionLevel Fastest in the latest version of ZLib-Intel. I have updated the PR with performance details for various compression levels.

[stephentoub]

Why were these previously zlib when the rest were zlib-intel? Is it that these files weren't previously specialized in the Intel implementation, and now they are?

[vkvenkat]

Yes, we added new optimizations for decompression in the latest ZLib-Intel. This was not available previously.

[stephentoub]

Is this the update that was referenced as being expected by middle of June?
#30362 (comment)

If yes, it's fine that it's delayed, but I'd like to understand why the four month delay to understand if there's something to be on the look out for as a result. Was there some kind of risk that caused the slip, some failures that required mitigation and lots of extra testing, etc.?

[vkvenkat]

Is this the update that was referenced as being expected by middle of June?
#30362 (comment)

If yes, it's fine that it's delayed, but I'd like to understand why the four month delay to understand if there's something to be on the look out for as a result. Was there some kind of risk that caused the slip, some failures that required mitigation and lots of extra testing, etc.?

@stephentoub Yes it is referencing this update. The initial version of ZLib-Intel with optimizations for upstream ZLib v1.2.11 was available in the June time-frame. Some of the delay since June was from evaluating additional optimizations and testing. We checked to ensure our schedule aligned with the .NET Core 3.0 release cycle.

[joshfree]

@stephentoub the delay is mostly our fault due to various leaves / time off on our end; we asked @vkvenkat to take their time and delay until more people were back on our side to help review and test the changes.

[stephentoub]

Ok, thanks. I mainly just wanted to know if there were any significant issues we should be aware of, whether we should be looking at beefing up testing in any particular areas, etc.

[danmoseley]

@vkvenkat do you have any performance numbers for non-Intel hardware?

[vkvenkat]

@danmosemsft We don't have any performance numbers for non-Intel hardware.

[tannergooding]

Do we need this, since CoreCLR requires SSE2 for both x86 and x64? Is this to support other runtimes using CoreFX that might not have the same requirement?

[joshfree]

@vkvenkat could you resolve the merge conflict and then we're good to merge this PR to master.

Thank you

[vkvenkat]

@joshfree Done.

[stephentoub]

Thanks, @vkvenkat.

[ViktorHofer]

This change introduced the following warnings in master:

c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.h(141): warning C4267: 'function': conversion from 'size_t' to 'long', possible loss of data (compiling source file C:\git\corefx\src\Native\Windows\clrcompression\zlib-intel\inffast.c) [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.h(150): warning C4267: 'function': conversion from 'size_t' to 'uInt', possible loss of data (compiling source file C:\git\corefx\src\Native\Windows\clrcompression\zlib-intel\inffast.c) [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.h(154): warning C4267: 'function': conversion from 'size_t' to 'uInt', possible loss of data (compiling source file C:\git\corefx\src\Native\Windows\clrcompression\zlib-intel\inffast.c) [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.h(174): warning C4267: '-=': conversion from 'size_t' to 'uInt', possible loss of data (compiling source file C:\git\corefx\src\Native\Windows\clrcompression\zlib-intel\inffast.c) [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.h(189): warning C4267: '-=': conversion from 'size_t' to 'unsigned int', possible loss of data (compiling source file C:\git\corefx\src\Native\Windows\clrcompression\zlib-intel\inffast.c) [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.h(190): warning C4267: '+=': conversion from 'size_t' to 'unsigned int', possible loss of data (compiling source file C:\git\corefx\src\Native\Windows\clrcompression\zlib-intel\inffast.c) [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inffast.c(102): warning C4244: '=': conversion from '__int64' to 'unsigned int', possible loss of data [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inffast.c(236): warning C4244: '=': conversion from '__int64' to 'unsigned int', possible loss of data [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.h(141): warning C4267: 'function': conversion from 'size_t' to 'long', possible loss of data (compiling source file C:\git\corefx\src\Native\Windows\clrcompression\zlib-intel\inflate.c) [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.h(150): warning C4267: 'function': conversion from 'size_t' to 'uInt', possible loss of data (compiling source file C:\git\corefx\src\Native\Windows\clrcompression\zlib-intel\inflate.c) [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.h(154): warning C4267: 'function': conversion from 'size_t' to 'uInt', possible loss of data (compiling source file C:\git\corefx\src\Native\Windows\clrcompression\zlib-intel\inflate.c) [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.h(174): warning C4267: '-=': conversion from 'size_t' to 'uInt', possible loss of data (compiling source file C:\git\corefx\src\Native\Windows\clrcompression\zlib-intel\inflate.c) [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.h(189): warning C4267: '-=': conversion from 'size_t' to 'unsigned int', possible loss of data (compiling source file C:\git\corefx\src\Native\Windows\clrcompression\zlib-intel\inflate.c) [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.h(190): warning C4267: '+=': conversion from 'size_t' to 'unsigned int', possible loss of data (compiling source file C:\git\corefx\src\Native\Windows\clrcompression\zlib-intel\inflate.c) [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.c(719): warning C4244: '=': conversion from '__int64' to 'unsigned int', possible loss of data [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.c(878): warning C4244: '=': conversion from '__int64' to 'unsigned int', possible loss of data [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.c(946): warning C4244: '=': conversion from '__int64' to 'unsigned int', possible loss of data [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.c(949): warning C4244: '=': conversion from '__int64' to 'unsigned long', possible loss of data [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.c(1090): warning C4244: '=': conversion from '__int64' to 'unsigned int', possible loss of data [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.c(1195): warning C4244: '=': conversion from '__int64' to 'unsigned int', possible loss of data [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.c(1201): warning C4244: '=': conversion from '__int64' to 'unsigned long', possible loss of data [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.c(1238): warning C4244: '=': conversion from '__int64' to 'unsigned int', possible loss of data [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.c(1247): warning C4244: '=': conversion from '__int64' to 'unsigned int', possible loss of data [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.c(1312): warning C4244: '=': conversion from '__int64' to 'unsigned int', possible loss of data [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.c(941): warning C4244: 'initializing': conversion from '__int64' to 'unsigned long', possible loss of data [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\inflate.c(1192): warning C4244: 'initializing': conversion from '__int64' to 'unsigned long', possible loss of data [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\x86.c(49): warning C4047: 'function': 'int *' differs in levels of indirection from 'int (*)[4]' [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]
c:\git\corefx\src\native\windows\clrcompression\zlib-intel\x86.c(49): warning C4024: '__cpuid': different types for formal and actual parameter 1 [C:\git\corefx\artifacts\obj\native\netcoreapp-Windows_NT-Debug-x64-aot\clrcompression\clrcompression.vcxproj] [C:\git\corefx\src\Native\build-native.proj]

[stephentoub]

@ViktorHofer, can you open an issue for this?

@vkvenkat, can you please address this with a follow-up PR? We really want our build to be warning and error free. At the moment there are a few other warnings due to an issue with one of the tools used in the build, but hopefully those will also be addressed soon.

[vkvenkat]

@stephentoub Sure, I will make a follow-up PR to fix this.

[ViktorHofer]

https://github.com/dotnet/corefx/issues/33277

[stephentoub]

Thanks, @vkvenkat and @ViktorHofer.

[vkvenkat]

Warnings fixed by PR #33357

[danmoseley]

Thanks @vkvenkat