Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate to zlib-ng, part 2: consume it in runtime #102403

Draft
wants to merge 22 commits into
base: main
Choose a base branch
from
Draft

Migrate to zlib-ng, part 2: consume it in runtime #102403

wants to merge 22 commits into from

Conversation

carlossanlop
Copy link
Member

@carlossanlop carlossanlop commented May 17, 2024 •
edited
Loading

Still in draft, please don't review yet until I get all the platforms working in CI.

Contributes to: #101465

  • Copied latest zlib-ng release and updated license-related files: Migrate to zlib-ng, part 1: copy source code only #102520
  • Updated cmake code
  • Confirmed full local build on Linux passed
  • Confirmed full local build on Windows passed (including NativeAOT)
  • Confirmed full local build on MacOS passed
  • Tests pass locally on Linux and Windows (System.IO.Compression, ZipFile, Brotli, System.IO.Packaging)
  • Fix MacOS issues in CI
  • Fix some quick temporary workarounds
  • Compare performance before and after on Linux and Windows, on x64 and arm64
  • Run runtime-community and fix any issues
  • Run runtime-extra-platforms and fix any issues
  • Document the future upgrading process
  • Deleted zlib and zlib-intel references
  • If zlib-ng is found installed in the current distro, use that instead (add it to a separate PR)

Performance comparisons

Before: 6e43b6a (main)
After: 619b721 (from this PR)
Ran the dotnet/performance System.IO.Compression microbenchmarks.

Ubuntu 22.04 x64 (WSL) summary: better: 38, geomean: 1.715 worse: 32, geomean: 1.038 total diff: 70
Slower diff/base Base Median (ns) Diff Median (ns) Modality
System.IO.Compression.Brotli.Compress(level: Optimal, file: "TestDocument.pdf") 1.18 192337.89 226048.52
System.IO.Compression.Brotli.Compress(level: Optimal, file: "sum") 1.08 3159.96 3421.86 bimodal
System.IO.Compression.Brotli.Decompress(level: Fastest, file: "TestDocument.pdf" 1.07 290766.68 312166.62
System.IO.Compression.Brotli.Decompress(level: Optimal, file: "TestDocument.pdf" 1.07 322551.99 346187.95
System.IO.Compression.Brotli.Decompress_WithoutState(level: Fastest, file: "Test 1.07 275812.51 294142.33
System.IO.Compression.Brotli.Decompress_WithState(level: Fastest, file: "TestDoc 1.06 276224.78 294076.54
System.IO.Compression.Brotli.Decompress_WithoutState(level: Optimal, file: "Test 1.06 310487.62 330175.91
System.IO.Compression.Brotli.Decompress_WithState(level: Optimal, file: "TestDoc 1.06 311455.12 331184.97
System.IO.Compression.Brotli.Compress(level: Fastest, file: "TestDocument.pdf") 1.06 384733.58 406323.28 bimodal
System.IO.Compression.Brotli.Decompress_WithState(level: Fastest, file: "alice29 1.05 422509.72 442234.80
System.IO.Compression.Brotli.Compress(level: Fastest, file: "alice29.txt") 1.04 856743.04 891092.01
System.IO.Compression.Brotli.Decompress_WithoutState(level: Fastest, file: "alic 1.04 422384.77 437716.92
System.IO.Compression.Brotli.Compress_WithState(level: Fastest, file: "alice29.t 1.04 759858.95 786494.05
System.IO.Compression.Brotli.Compress_WithoutState(level: Optimal, file: "TestDo 1.03 467828989.00 483341893.00
System.IO.Compression.Brotli.Compress_WithState(level: Optimal, file: "TestDocum 1.03 467929298.50 482845800.00
System.IO.Compression.Brotli.Decompress(level: Fastest, file: "sum") 1.03 117844.45 121368.56
System.IO.Compression.Brotli.Compress_WithState(level: Fastest, file: "TestDocum 1.03 302991.91 311955.75
System.IO.Compression.Brotli.Decompress_WithState(level: Fastest, file: "sum") 1.02 112810.47 115470.21
System.IO.Compression.Brotli.Compress_WithoutState(level: Optimal, file: "sum") 1.02 48053373.75 49091125.25
System.IO.Compression.Brotli.Decompress(level: Optimal, file: "sum") 1.02 121051.31 123624.03
System.IO.Compression.Brotli.Decompress(level: Fastest, file: "alice29.txt") 1.02 448704.71 457814.62
System.IO.Compression.Brotli.Decompress_WithoutState(level: Fastest, file: "sum" 1.02 112492.33 114594.62
System.IO.Compression.Brotli.Decompress_WithState(level: Optimal, file: "alice29 1.02 362410.90 369060.57
System.IO.Compression.Brotli.Decompress_WithState(level: Optimal, file: "sum") 1.02 115182.90 117289.21
System.IO.Compression.Brotli.Compress_WithState(level: Optimal, file: "sum") 1.02 47923622.25 48783400.00
System.IO.Compression.Brotli.Compress_WithoutState(level: Fastest, file: "TestDo 1.02 305730.75 311134.31
System.IO.Compression.Brotli.Compress_WithoutState(level: Fastest, file: "alice2 1.02 756251.18 769102.68
System.IO.Compression.Brotli.Compress_WithState(level: Fastest, file: "sum") 1.01 138117.44 139961.59
System.IO.Compression.Brotli.Compress_WithoutState(level: Fastest, file: "sum") 1.01 136391.67 138110.64
System.IO.Compression.Brotli.Decompress_WithoutState(level: Optimal, file: "alic 1.01 362289.44 366212.65
System.IO.Compression.Brotli.Decompress_WithoutState(level: Optimal, file: "sum" 1.01 115232.62 116098.99
System.IO.Compression.Brotli.Compress_WithState(level: Optimal, file: "alice29.t 1.00 202616422.00 203507000.00
Faster base/diff Base Median (ns) Diff Median (ns) Modality
System.IO.Compression.Gzip.Compress(level: Optimal, file: "sum") 2.85 1097465.62 384723.33
System.IO.Compression.ZLib.Compress(level: Optimal, file: "sum") 2.82 1084193.76 383910.37
System.IO.Compression.Deflate.Compress(level: Optimal, file: "sum") 2.71 1085387.48 400419.55
System.IO.Compression.ZLib.Compress(level: Fastest, file: "alice29.txt") 2.68 1779124.98 663022.18
System.IO.Compression.Gzip.Compress(level: Fastest, file: "alice29.txt") 2.68 1797944.78 670268.46
System.IO.Compression.Deflate.Compress(level: Fastest, file: "alice29.txt") 2.62 1725938.17 659444.94
System.IO.Compression.Gzip.Compress(level: Optimal, file: "alice29.txt") 2.43 6178248.86 2545687.55
System.IO.Compression.ZLib.Compress(level: Optimal, file: "alice29.txt") 2.40 6118452.21 2553188.04
System.IO.Compression.Deflate.Compress(level: Optimal, file: "alice29.txt") 2.31 6050531.13 2615256.24
System.IO.Compression.Gzip.Compress(level: Fastest, file: "TestDocument.pdf") 2.27 2307834.82 1017390.62
System.IO.Compression.ZLib.Compress(level: Fastest, file: "TestDocument.pdf") 2.25 2293871.42 1021050.35
System.IO.Compression.Deflate.Compress(level: Fastest, file: "TestDocument.pdf") 2.23 2271449.05 1016917.91
System.IO.Compression.Gzip.Decompress(level: Fastest, file: "sum") 2.05 149727.56 73008.16
System.IO.Compression.Gzip.Compress(level: Fastest, file: "sum") 2.02 288366.05 142457.16
System.IO.Compression.Gzip.Decompress(level: Optimal, file: "sum") 1.97 137549.53 69697.27
System.IO.Compression.ZLib.Compress(level: Fastest, file: "sum") 1.94 276941.67 142801.42
System.IO.Compression.Deflate.Compress(level: Fastest, file: "sum") 1.91 273130.99 142896.90
System.IO.Compression.ZLib.Decompress(level: Fastest, file: "sum") 1.89 137509.85 72854.60
System.IO.Compression.ZLib.Decompress(level: Optimal, file: "sum") 1.78 123422.10 69235.10
System.IO.Compression.Deflate.Decompress(level: Fastest, file: "sum") 1.76 128101.29 72767.57
System.IO.Compression.Gzip.Decompress(level: Optimal, file: "alice29.txt") 1.70 585702.87 344260.40
System.IO.Compression.Deflate.Decompress(level: Optimal, file: "sum") 1.63 113690.46 69811.35
System.IO.Compression.ZLib.Decompress(level: Optimal, file: "alice29.txt") 1.55 533825.87 344802.72
System.IO.Compression.Deflate.Decompress(level: Optimal, file: "alice29.txt") 1.43 492854.63 345130.00
System.IO.Compression.Gzip.Decompress(level: Optimal, file: "TestDocument.pdf") 1.43 367603.19 257902.87
System.IO.Compression.Gzip.Decompress(level: Fastest, file: "alice29.txt") 1.40 607602.50 433918.42
System.IO.Compression.Gzip.Decompress(level: Fastest, file: "TestDocument.pdf") 1.34 370051.94 276042.33
System.IO.Compression.ZLib.Decompress(level: Fastest, file: "alice29.txt") 1.30 558262.72 428688.68
System.IO.Compression.ZLib.Decompress(level: Optimal, file: "TestDocument.pdf") 1.26 324012.91 257498.05
System.IO.Compression.ZLib.Decompress(level: Fastest, file: "TestDocument.pdf") 1.21 332792.28 275080.92
System.IO.Compression.Deflate.Decompress(level: Fastest, file: "alice29.txt") 1.21 515055.61 427031.22
System.IO.Compression.Gzip.Compress(level: Optimal, file: "TestDocument.pdf") 1.16 2515867.13 2175346.41
System.IO.Compression.Deflate.Decompress(level: Optimal, file: "TestDocument.pdf 1.14 291316.86 256608.40
System.IO.Compression.Deflate.Decompress(level: Fastest, file: "TestDocument.pdf 1.11 302148.51 272684.50
System.IO.Compression.Deflate.Compress(level: Optimal, file: "TestDocument.pdf") 1.08 2360974.97 2181609.85
System.IO.Compression.ZLib.Compress(level: Optimal, file: "TestDocument.pdf") 1.08 2375183.47 2197691.48
System.IO.Compression.Brotli.Compress(level: Fastest, file: "sum") 1.06 162542.80 153978.63
System.IO.Compression.Brotli.Compress(level: Optimal, file: "alice29.txt") 1.01 1989422.69 1972818.00
Windows 11 arm64 (Surface Pro X)

summary:
better: 50, geomean: 1.362
worse: 13, geomean: 1.030
total diff: 63

Slower diff/base Base Median (ns) Diff Median (ns) Modality
System.IO.Compression.Brotli.Compress(level: Optimal, file: "sum") 1.19 5270.81 6291.58
System.IO.Compression.Brotli.Compress_WithoutState(level: Optimal, file: "TestDo 1.05 607769900.00 638531100.00
System.IO.Compression.Brotli.Compress_WithState(level: Optimal, file: "TestDocum 1.05 608341100.00 636545000.00
System.IO.Compression.Brotli.Decompress_WithState(level: Optimal, file: "sum") 1.02 182812.10 187125.91
System.IO.Compression.Brotli.Decompress(level: Fastest, file: "alice29.txt") 1.02 792524.69 806630.31
System.IO.Compression.Brotli.Decompress_WithState(level: Optimal, file: "alice29 1.02 609244.40 619227.34
System.IO.Compression.Brotli.Compress(level: Optimal, file: "alice29.txt") 1.01 3142830.63 3187173.75
System.IO.Compression.Brotli.Decompress(level: Optimal, file: "alice29.txt") 1.01 656019.27 664360.03
System.IO.Compression.Brotli.Decompress_WithState(level: Fastest, file: "sum") 1.01 184681.32 186634.00
System.IO.Compression.Brotli.Compress(level: Optimal, file: "TestDocument.pdf") 1.01 587505.21 592723.80
System.IO.Compression.Brotli.Decompress(level: Fastest, file: "sum") 1.01 202022.00 203072.12
System.IO.Compression.Brotli.Decompress_WithState(level: Fastest, file: "alice29 1.00 787389.14 791031.25
System.IO.Compression.Brotli.Compress(level: Fastest, file: "TestDocument.pdf") 1.00 1172639.29 1177367.86
Faster base/diff Base Median (ns) Diff Median (ns) Modality
System.IO.Compression.Gzip.Compress(level: Optimal, file: "sum") 2.47 2150283.59 869373.96
System.IO.Compression.Deflate.Compress(level: Optimal, file: "sum") 2.47 2134192.19 865086.46
System.IO.Compression.ZLib.Compress(level: Optimal, file: "sum") 2.44 2149914.84 881053.99
System.IO.Compression.Gzip.Compress(level: Fastest, file: "alice29.txt") 2.41 2980981.25 1238357.93
System.IO.Compression.Deflate.Compress(level: Fastest, file: "alice29.txt") 2.37 2916461.46 1232579.33
System.IO.Compression.ZLib.Compress(level: Fastest, file: "alice29.txt") 2.30 2971314.58 1290300.96
System.IO.Compression.Gzip.Compress(level: Optimal, file: "alice29.txt") 2.30 10052270.31 4365707.81
System.IO.Compression.Deflate.Compress(level: Optimal, file: "alice29.txt") 2.29 9987428.13 4369489.06
System.IO.Compression.ZLib.Compress(level: Optimal, file: "alice29.txt") 2.28 10036128.13 4406249.22
System.IO.Compression.Gzip.Compress(level: Fastest, file: "TestDocument.pdf") 1.98 4009726.56 2025188.28
System.IO.Compression.Deflate.Compress(level: Fastest, file: "TestDocument.pdf") 1.96 3958289.06 2017482.42
System.IO.Compression.ZLib.Compress(level: Fastest, file: "TestDocument.pdf") 1.75 4004092.97 2289694.20 several?
System.IO.Compression.Brotli.Decompress_WithoutState(level: Optimal, file: "Test 1.57 908589.96 578646.88 bimodal
System.IO.Compression.Gzip.Compress(level: Fastest, file: "sum") 1.54 611528.49 396808.65
System.IO.Compression.Gzip.Decompress(level: Fastest, file: "sum") 1.54 171008.22 110966.15
System.IO.Compression.Deflate.Compress(level: Fastest, file: "sum") 1.52 596260.88 393349.22
System.IO.Compression.Deflate.Decompress(level: Fastest, file: "sum") 1.45 156066.36 107914.81
System.IO.Compression.Gzip.Decompress(level: Optimal, file: "sum") 1.44 158872.22 110563.78
System.IO.Compression.ZLib.Decompress(level: Fastest, file: "sum") 1.34 168747.92 126002.68
System.IO.Compression.Deflate.Decompress(level: Optimal, file: "sum") 1.32 142294.77 107644.81
System.IO.Compression.ZLib.Decompress(level: Optimal, file: "sum") 1.28 156282.56 121858.87
System.IO.Compression.Gzip.Decompress(level: Fastest, file: "TestDocument.pdf") 1.26 486365.72 384855.87
System.IO.Compression.Gzip.Decompress(level: Optimal, file: "alice29.txt") 1.26 617596.00 489310.74
System.IO.Compression.Gzip.Decompress(level: Optimal, file: "TestDocument.pdf") 1.26 482179.88 383280.00
System.IO.Compression.ZLib.Compress(level: Fastest, file: "sum") 1.22 607766.35 498967.62 several?
System.IO.Compression.Deflate.Decompress(level: Fastest, file: "TestDocument.pdf 1.16 435230.63 374555.11
System.IO.Compression.Deflate.Decompress(level: Optimal, file: "alice29.txt") 1.16 554758.48 477472.54
System.IO.Compression.Deflate.Decompress(level: Optimal, file: "TestDocument.pdf 1.15 430319.02 373470.83
System.IO.Compression.Gzip.Compress(level: Optimal, file: "TestDocument.pdf") 1.15 4153676.56 3612664.38
System.IO.Compression.Gzip.Decompress(level: Fastest, file: "alice29.txt") 1.14 654035.16 572136.81
System.IO.Compression.ZLib.Decompress(level: Optimal, file: "alice29.txt") 1.14 607173.56 531369.17
System.IO.Compression.ZLib.Compress(level: Optimal, file: "TestDocument.pdf") 1.14 4154059.38 3653118.13
System.IO.Compression.Deflate.Compress(level: Optimal, file: "TestDocument.pdf") 1.14 4099540.63 3610190.63
System.IO.Compression.ZLib.Decompress(level: Optimal, file: "TestDocument.pdf") 1.13 474408.71 418232.24
System.IO.Compression.Deflate.Decompress(level: Fastest, file: "alice29.txt") 1.06 590469.11 558907.81
System.IO.Compression.ZLib.Decompress(level: Fastest, file: "TestDocument.pdf") 1.05 479221.31 458220.02 several?
System.IO.Compression.Brotli.Decompress_WithoutState(level: Optimal, file: "alic 1.04 645995.44 618226.36
System.IO.Compression.Brotli.Compress_WithoutState(level: Fastest, file: "TestDo 1.04 1189231.01 1146041.11
System.IO.Compression.Brotli.Decompress_WithoutState(level: Fastest, file: "sum" 1.04 193030.59 186201.11 several?
System.IO.Compression.Brotli.Decompress_WithoutState(level: Fastest, file: "alic 1.03 817060.07 790537.99
System.IO.Compression.Brotli.Compress_WithState(level: Fastest, file: "sum") 1.02 431906.07 423876.23
System.IO.Compression.Brotli.Compress_WithState(level: Fastest, file: "TestDocum 1.02 1164633.17 1143570.91
System.IO.Compression.Brotli.Decompress(level: Optimal, file: "sum") 1.02 204358.49 200671.75
System.IO.Compression.Brotli.Compress_WithoutState(level: Fastest, file: "alice2 1.01 1798088.89 1772452.08
System.IO.Compression.Brotli.Compress_WithState(level: Fastest, file: "alice29.t 1.01 1795713.54 1779367.36
System.IO.Compression.Brotli.Compress(level: Fastest, file: "sum") 1.01 429493.41 426665.37
System.IO.Compression.Brotli.Decompress_WithState(level: Fastest, file: "TestDoc 1.01 511434.78 508288.51
System.IO.Compression.Brotli.Compress_WithState(level: Optimal, file: "sum") 1.01 63332225.00 62946050.00
System.IO.Compression.Brotli.Compress(level: Fastest, file: "alice29.txt") 1.01 1768822.92 1759184.03
System.IO.Compression.Brotli.Decompress_WithState(level: Optimal, file: "TestDoc 1.00 548117.08 547487.96
macOS Monterey 12.7.5 (MacBook Pro Retina)

summary:
better: 37, geomean: 1.480
worse: 18, geomean: 1.019
total diff: 55

Slower diff/base Base Median (ns) Diff Median (ns) Modality
System.IO.Compression.Deflate.Decompress(level: Fastest, file: "alice29.txt") 1.06 549618.75 584521.27
System.IO.Compression.Gzip.Decompress(level: Fastest, file: "alice29.txt") 1.06 560990.62 595117.25
System.IO.Compression.ZLib.Decompress(level: Fastest, file: "alice29.txt") 1.05 558433.87 586991.67
System.IO.Compression.Brotli.Compress_WithState(level: Optimal, file: "sum") 1.02 53760106.25 54846369.25
System.IO.Compression.Brotli.Compress_WithoutState(level: Fastest, file: "alice2 1.02 912386.47 929932.59
System.IO.Compression.Brotli.Compress_WithoutState(level: Fastest, file: "TestDo 1.02 334342.43 340315.11
System.IO.Compression.Brotli.Compress_WithState(level: Fastest, file: "sum") 1.02 176230.43 179152.99
System.IO.Compression.Brotli.Compress(level: Optimal, file: "sum") 1.01 9114.75 9242.05
System.IO.Compression.Brotli.Decompress(level: Fastest, file: "sum") 1.01 148637.61 150488.17
System.IO.Compression.Brotli.Compress_WithoutState(level: Optimal, file: "sum") 1.01 54129042.50 54723757.25
System.IO.Compression.Brotli.Compress_WithState(level: Fastest, file: "alice29.t 1.01 920275.82 929710.49
System.IO.Compression.Brotli.Compress(level: Fastest, file: "TestDocument.pdf") 1.01 555177.83 560305.22
System.IO.Compression.Brotli.Decompress_WithoutState(level: Fastest, file: "sum" 1.01 131600.86 132618.99
System.IO.Compression.Deflate.Decompress(level: Fastest, file: "TestDocument.pdf 1.01 348107.75 350787.92
System.IO.Compression.Brotli.Compress_WithoutState(level: Fastest, file: "sum") 1.01 177518.88 178738.14
System.IO.Compression.Brotli.Decompress(level: Fastest, file: "alice29.txt") 1.01 550253.89 553875.38
System.IO.Compression.Brotli.Decompress_WithState(level: Fastest, file: "sum") 1.00 131204.18 131814.53
System.IO.Compression.Brotli.Decompress_WithoutState(level: Optimal, file: "sum" 1.00 134184.06 134599.37
Faster base/diff Base Median (ns) Diff Median (ns) Modality
System.IO.Compression.Deflate.Compress(level: Optimal, file: "sum") 3.42 1812515.64 529920.91
System.IO.Compression.ZLib.Compress(level: Optimal, file: "sum") 3.42 1808531.92 528815.01
System.IO.Compression.Gzip.Compress(level: Optimal, file: "sum") 3.39 1803433.29 531642.70
System.IO.Compression.Gzip.Compress(level: Optimal, file: "alice29.txt") 2.70 7895141.33 2923967.60
System.IO.Compression.ZLib.Compress(level: Optimal, file: "alice29.txt") 2.70 7893726.34 2928123.30
System.IO.Compression.Deflate.Compress(level: Optimal, file: "alice29.txt") 2.69 7891106.95 2928246.65
System.IO.Compression.ZLib.Compress(level: Fastest, file: "alice29.txt") 2.06 1900947.80 924022.95
System.IO.Compression.Deflate.Compress(level: Fastest, file: "alice29.txt") 2.05 1885564.55 919178.03
System.IO.Compression.Gzip.Compress(level: Fastest, file: "alice29.txt") 2.04 1894650.97 930557.78
System.IO.Compression.Gzip.Compress(level: Fastest, file: "TestDocument.pdf") 1.93 2529325.45 1309898.46
System.IO.Compression.ZLib.Compress(level: Fastest, file: "TestDocument.pdf") 1.93 2525721.55 1311404.21
System.IO.Compression.Deflate.Compress(level: Fastest, file: "TestDocument.pdf") 1.92 2517268.68 1310960.16
System.IO.Compression.ZLib.Compress(level: Fastest, file: "sum") 1.58 427243.90 269877.01
System.IO.Compression.Gzip.Compress(level: Fastest, file: "sum") 1.58 429229.83 271587.44
System.IO.Compression.Deflate.Compress(level: Fastest, file: "sum") 1.57 427370.64 271989.91
System.IO.Compression.ZLib.Decompress(level: Fastest, file: "sum") 1.34 139550.56 104525.50
System.IO.Compression.Gzip.Decompress(level: Fastest, file: "sum") 1.32 139688.85 105731.13
System.IO.Compression.Deflate.Decompress(level: Fastest, file: "sum") 1.30 136345.50 104808.51
System.IO.Compression.ZLib.Decompress(level: Optimal, file: "sum") 1.30 127682.56 98520.50
System.IO.Compression.Gzip.Decompress(level: Optimal, file: "sum") 1.27 127627.84 100837.98
System.IO.Compression.Deflate.Decompress(level: Optimal, file: "sum") 1.26 125285.66 99570.05
System.IO.Compression.ZLib.Decompress(level: Optimal, file: "alice29.txt") 1.13 525257.97 466485.81
System.IO.Compression.Gzip.Decompress(level: Optimal, file: "alice29.txt") 1.11 526093.01 473394.17
System.IO.Compression.Deflate.Decompress(level: Optimal, file: "alice29.txt") 1.11 516434.36 466673.98
System.IO.Compression.Gzip.Compress(level: Optimal, file: "TestDocument.pdf") 1.10 2598676.80 2355396.19
System.IO.Compression.ZLib.Compress(level: Optimal, file: "TestDocument.pdf") 1.10 2594676.63 2355279.80
System.IO.Compression.Deflate.Compress(level: Optimal, file: "TestDocument.pdf") 1.10 2585309.39 2353737.32
System.IO.Compression.ZLib.Decompress(level: Optimal, file: "TestDocument.pdf") 1.05 351785.59 334371.83
System.IO.Compression.Gzip.Decompress(level: Optimal, file: "TestDocument.pdf") 1.04 353616.17 339535.04
System.IO.Compression.Deflate.Decompress(level: Optimal, file: "TestDocument.pdf 1.03 342057.75 332249.63
System.IO.Compression.Brotli.Compress_WithoutState(level: Optimal, file: "TestDo 1.01 508737051.00 501369100.00
System.IO.Compression.Brotli.Compress(level: Optimal, file: "TestDocument.pdf") 1.01 331332.68 327121.47
System.IO.Compression.Brotli.Decompress(level: Optimal, file: "sum") 1.01 154442.29 152562.15
System.IO.Compression.ZLib.Decompress(level: Fastest, file: "TestDocument.pdf") 1.01 357752.79 353440.56
System.IO.Compression.Brotli.Decompress(level: Optimal, file: "alice29.txt") 1.01 481975.35 478448.91
System.IO.Compression.Brotli.Compress_WithState(level: Optimal, file: "TestDocum 1.01 504233430.50 501565277.50
System.IO.Compression.Brotli.Decompress_WithState(level: Optimal, file: "TestDoc 1.00 383464.41 381716.51

@carlossanlop carlossanlop self-assigned this May 17, 2024
@jkotas
Copy link
Member

jkotas commented May 17, 2024

To make this easier to review, it would be best to split this into 3 PRs:

  • Copied latest zlib-ng release and updated license-related file
  • All the other steps
  • Deleted zlib and zlib-intel references

@carlossanlop
Copy link
Member Author

@jkotas yes, I can do that. I don't want you to keep a fire extinguisher next to your overworked processor.

@carlossanlop
Copy link
Member Author

To make this easier to review, it would be best to split this into 3 PRs:

@jkotas I force-pushed 3 commits to make this easier to review. This is the commit with all the changes that coupled zlib-ng with runtime: c65dad0

@jkotas
Copy link
Member

jkotas commented May 21, 2024

The test binaries like src/native/external/zlib-ng/test/CVE-2002-0059/test.gz are going to pain to deal with for source build.

Can we exclude the test subdirectory from the vendored copy? I do not expect that we are going to run the native zlib-ng tests in this repo.

@jkotas
Copy link
Member

jkotas commented May 21, 2024

To make this easier to review, it would be best to split this into 3 PRs:

I force-pushed 3 commits to make this easier to review. This is the commit with all the changes that coupled zlib-ng with runtime: c65dad0

This is better than nothing, but it is still pretty clunky experience.

FWIW, I am regularly splitting my changes into multiple PRs when they are mix of large mechanical delta and smaller actual code delta. For example, I have done it a few days ago in #102422. I could have folded this mechanical change into #102392 that it is needed for, but I have intentionally split it into multiple PRs to get it signed-off and merged faster.

@build-analysis build-analysis bot mentioned this pull request May 21, 2024
Closed
@carlossanlop carlossanlop mentioned this pull request May 21, 2024
Merged
@carlossanlop
Copy link
Member Author

As requested, I submitted #102520 to only bring in the zlib-ng code into our repo (and its licensing files), but without consuming it anywhere yet.

I'll update this PR to only include the code that enables zlib-ng, and will revert the changes that remove zlib and zlib-intel (I'll remove those in a third PR).

@carlossanlop carlossanlop changed the title Migrate to zlib-ng Migrate to zlib-ng, part 2: consume it in runtime May 21, 2024
@carlossanlop
Copy link
Member Author

Big difference now: 23 files instead of 388.

jkoritzinsky
jkoritzinsky reviewed May 22, 2024
View reviewed changes
src/native/external/zlib-intel.cmake Outdated Show resolved Hide resolved
src/native/external/zlib-ng.cmake Outdated Show resolved Hide resolved
src/native/libs/System.IO.Compression.Native/CMakeLists.txt Outdated Show resolved Hide resolved
src/native/libs/System.IO.Compression.Native/zlib_allocator_win.c Show resolved Hide resolved
src/native/libs/System.IO.Compression.Native/zlib_allocator_win.c Outdated Show resolved Hide resolved
eng/native/configurecompiler.cmake Outdated Show resolved Hide resolved
@carlossanlop
Copy link
Member Author

I updated the PR description to include some initial perf comparisons in Linux, Windows and MacOS.

akoeplinger
akoeplinger reviewed May 23, 2024
View reviewed changes
src/mono/mono/mini/CMakeLists.txt Outdated Show resolved Hide resolved
src/mono/mono/mini/CMakeLists.txt Outdated Show resolved Hide resolved
src/mono/mono/mini/CMakeLists.txt Outdated Show resolved Hide resolved
src/native/libs/System.IO.Compression.Native/pal_zlib.c Show resolved Hide resolved
@akoeplinger
Copy link
Member

The intention is that we don't use system zlib anymore anywhere right? is that OK with distro maintainers?

I wonder if we should set ZLIB_SYMBOL_PREFIX to e.g. dotnet_ so we don't run into conflicting symbols like what happened with ICU on iOS recently (#98941)

@carlossanlop
Copy link
Member Author

I wonder if we should set ZLIB_SYMBOL_PREFIX to e.g. dotnet_ so we don't run into conflicting symbols like what happened with ICU on iOS recently (#98941)

The linked issue indicates that the repro steps require creating a new iOS project. What that tells me is that the issue was found after merging, and it was caught in servicing (8.0, back in February). I also don't see a PR that fixed the issue, it looks like a workaround was given instead. I'd like to avoid getting at that point to find out about this problem, and also avoid a workaround. Do you know if I can repro this before merging this PR? Maybe even write a test than can confirm this does not happen?

@build-analysis build-analysis bot mentioned this pull request May 24, 2024
Open
@build-analysis build-analysis bot mentioned this pull request May 31, 2024
Open
@akoeplinger
Copy link
Member

Do you know if I can repro this before merging this PR? Maybe even write a test than can confirm this does not happen?

No, the problem was in fact caused by an iOS update after we shipped so there was no way to catch it beforehand, which is why I'd like to make sure we don't have the same potential conflict.

This was referenced Jun 15, 2024
Open
Open
Open
This was referenced Jun 18, 2024
Open
Open
This was referenced Jun 19, 2024
Closed
Closed
Closed
Open
Open
jkotas
jkotas reviewed Jun 19, 2024
View reviewed changes
src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Unix.targets
@@ -129,6 +129,7 @@ The .NET Foundation licenses this file to you under the MIT license.
<NativeLibrary Condition="'$(_targetArchitecture)' == 'x64'" Include="$(IlcSdkPath)$(VxSortSupportName)$(LibFileExt)" />
<NativeLibrary Include="$(IlcSdkPath)$(StandaloneGCSupportName)$(LibFileExt)" />
<NativeLibrary Condition="'$(LinkStandardCPlusPlusLibrary)' != 'true' and '$(StaticICULinking)' != 'true'" Include="$(IlcSdkPath)libstdc++compat.a" />
<NativeLibrary Include="$(IlcSdkPath)libz.a" />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native AOT is supported for mobile. We should use the same default zlib strategy for all runtime flavors targeting given platform. I think you should introduce a property like UseSystemZLib and use it in these target files.

(You will also want to run outer loop native aot tests to make sure that it works.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assumed that you are planning to use system zlib for mobile and browser/wasm given the discussion in #101465 , but it looks like that you are planning to use system zlib for browser wasm only. What is the plan?

In any case, it would be useful to have a property to disable use of zlib-ng for mobile and native aot that advanced users can use to resolve the conflicts. This property is also going to be useful to switch to system zlib in source build scenarios.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the moment of writing my comment in the main issue, I was testing using our copy of zlib-ng everywhere. Last night I was able to get it to work for all platforms (just one CI leg failing but I think I got it fixed just now).

Since this won't make it into preview6, I was thinking I could do the check for an installed system zlib in a separate PR, as we had originally discussed it initially when I created the PR:

image

Thanks for your suggestions, I will make sure to keep them in mind for when I work on that.

This was referenced Jun 19, 2024
Closed
Closed
@build-analysis build-analysis bot mentioned this pull request Jun 20, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lambdageek lambdageek lambdageek left review comments

@akoeplinger akoeplinger akoeplinger left review comments

@jkoritzinsky jkoritzinsky jkoritzinsky left review comments

@stephentoub stephentoub stephentoub left review comments

@jkotas jkotas jkotas left review comments

@ericstj ericstj ericstj left review comments

@lewing lewing Awaiting requested review from lewing lewing will be requested when the pull request is marked ready for review lewing is a code owner

@pavelsavara pavelsavara Awaiting requested review from pavelsavara pavelsavara will be requested when the pull request is marked ready for review pavelsavara is a code owner

@BrzVlad BrzVlad Awaiting requested review from BrzVlad BrzVlad will be requested when the pull request is marked ready for review BrzVlad is a code owner

@steveisok steveisok Awaiting requested review from steveisok steveisok will be requested when the pull request is marked ready for review steveisok is a code owner

@lateralusX lateralusX Awaiting requested review from lateralusX lateralusX will be requested when the pull request is marked ready for review lateralusX is a code owner

@thaystg thaystg Awaiting requested review from thaystg thaystg will be requested when the pull request is marked ready for review thaystg is a code owner

@marek-safar marek-safar Awaiting requested review from marek-safar marek-safar will be requested when the pull request is marked ready for review marek-safar is a code owner

@MichalStrehovsky MichalStrehovsky Awaiting requested review from MichalStrehovsky MichalStrehovsky will be requested when the pull request is marked ready for review MichalStrehovsky is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@carlossanlop carlossanlop

Labels
area-System.IO.Compression
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@carlossanlop @jkotas @akoeplinger @lambdageek @jkoritzinsky @stephentoub @ericstj