Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libz - Looks like we've been duplicating effort #3

Closed
sortie opened this issue Apr 19, 2015 · 2 comments
Closed

libz - Looks like we've been duplicating effort #3

sortie opened this issue Apr 19, 2015 · 2 comments

Comments

@sortie
Copy link

sortie commented Apr 19, 2015

Hi!

I'm Jonas. Looks like I'm not the only one that feels zlib needs a cleanup. I've been working on a cleaner and safer edition of zlib during the past year as part of my operating system. I think we should cooperate.

Our focus has been on cleaning it up thoroughly, correctness, security, proper portability, and so on. I can see we've made a lot of the same changes independently. I can see zlib-ng has done several optimizations, some of which looks interesting. We've done very little work in that area, we offer the same performance as zlib.

You can download my libz from https://sortix.org/libz/. Development is discussed in #libz on freenode, feel free to drop by. :)

@Dead2
Copy link
Member

Dead2 commented Apr 22, 2015

I had a good look through all your changesets, and it seems we have indeed made a lot of similar changes. There is also a pretty big delta due to different goals though, but I would like to pull a lot of your changes for sure. I likely will not pull all the feature-removal patches, since some of those actually make sense in embedded or statically linked setups. (disabling building gzip+crc code for example)

Too bad you started out with a mega-patch, since that makes it pretty hard to pull or audit.

At the moment I have three work-items planned:

  • Finish rewriting from K&R to Ansi C (exported functions have not been touched yet)
  • Rewrite to use uint types, and consider uint_fast in hotpaths where this actually helps performance (this varies a lot depending on architecture and the code itself, so needs to be done with cross-arch benchmarks and not blindly).
  • Find a solution to a buffer size problem (a new exported function possibly, needs further investigation and cooperation across zlib-compatible libraries)

@mtl1979
Copy link
Collaborator

mtl1979 commented May 5, 2015

For all developers working for zlib-ng, it would really require making copy of sortie's libz repository and reverting all of the cosmetic changes that are unnecessary and then making clean diff against official zlib 1.2.8.
Then we could isolate the different parts that zlib-ng could benefit from and make partial diffs on separate branches. Only then we can audit the code on each branch and pull if it passes.

That's pretty big task to do in my opinion.

mtl1979 pushed a commit to mtl1979/zlib-ng that referenced this issue Nov 21, 2015
mtl1979 pushed a commit to mtl1979/zlib-ng that referenced this issue Nov 21, 2015
mtl1979 pushed a commit to mtl1979/zlib-ng that referenced this issue Nov 23, 2015
mtl1979 pushed a commit to mtl1979/zlib-ng that referenced this issue Nov 24, 2015
@Dead2 Dead2 closed this as completed Mar 17, 2017
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Aug 17, 2018
==397==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x4e8193 in fill_window_sse /src/zlib-ng/arch/x86/fill_window_sse.c:52:30
    zlib-ng#1 0x4e9254 in deflate_quick /src/zlib-ng/arch/x86/deflate_quick.c:220:13
    zlib-ng#2 0x4ad375 in zng_deflate /src/zlib-ng/deflate.c:951:18
    zlib-ng#3 0x49fef1 in test_large_deflate /src/zlib-ng/test/example.c:264:11
    zlib-ng#4 0x4a3232 in main /src/zlib-ng/test/example.c:536:5
    zlib-ng#5 0x7f42277fa82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    zlib-ng#6 0x41b3a8 in _start (/src/zlib-ng/example+0x41b3a8)

  Uninitialized value was created by a heap allocation
    #0 0x45b9e0 in malloc /src/llvm/projects/compiler-rt/lib/msan/msan_interceptors.cc:910
    zlib-ng#1 0x4a47b2 in zng_deflateInit2_ /src/zlib-ng/deflate.c:284:27
    zlib-ng#2 0x4a3f6a in zng_deflateInit_ /src/zlib-ng/deflate.c:224:12
    zlib-ng#3 0x49fdd5 in test_large_deflate /src/zlib-ng/test/example.c:253:11
    zlib-ng#4 0x4a3232 in main /src/zlib-ng/test/example.c:536:5
    zlib-ng#5 0x7f42277fa82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Aug 20, 2018
==397==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x4e8193 in fill_window_sse /src/zlib-ng/arch/x86/fill_window_sse.c:52:30
    zlib-ng#1 0x4e9254 in deflate_quick /src/zlib-ng/arch/x86/deflate_quick.c:220:13
    zlib-ng#2 0x4ad375 in zng_deflate /src/zlib-ng/deflate.c:951:18
    zlib-ng#3 0x49fef1 in test_large_deflate /src/zlib-ng/test/example.c:264:11
    zlib-ng#4 0x4a3232 in main /src/zlib-ng/test/example.c:536:5
    zlib-ng#5 0x7f927f42d82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    zlib-ng#6 0x41b3a8 in _start (/src/zlib-ng/example+0x41b3a8)
  Uninitialized value was created by a heap allocation
    #0 0x45b9e0 in malloc /src/llvm/projects/compiler-rt/lib/msan/msan_interceptors.cc:910
    zlib-ng#1 0x4a47b2 in zng_deflateInit2_ /src/zlib-ng/deflate.c:284:27
    zlib-ng#2 0x4a3f6a in zng_deflateInit_ /src/zlib-ng/deflate.c:224:12
    zlib-ng#3 0x49fdd5 in test_large_deflate /src/zlib-ng/test/example.c:253:11
    zlib-ng#4 0x4a3232 in main /src/zlib-ng/test/example.c:536:5
    zlib-ng#5 0x7f927f42d82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Aug 21, 2018
==1==WARNING: MemorySanitizer: use-of-uninitialized-value
  #0 0x59fa93 in deflate_medium zlib-ng/deflate_medium.c:259:21
  zlib-ng#1 0x590905 in zng_deflate zlib-ng/deflate.c:951:18
  zlib-ng#2 0x587095 in zng_compress2 zlib-ng/compress.c:59:15
  zlib-ng#3 0x5866e3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:18:3
  zlib-ng#4 0x5862fd in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:38:3
  zlib-ng#5 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  zlib-ng#6 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  zlib-ng#7 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  zlib-ng#8 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  zlib-ng#9 0x7fea2fea482f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
  zlib-ng#10 0x41ec68 in _start
Uninitialized value was created by a heap allocation
  #0 0x45f2a0 in malloc /src/llvm/projects/compiler-rt/lib/msan/msan_interceptors.cc:910
  zlib-ng#1 0x587d42 in zng_deflateInit2_ zlib-ng/deflate.c:284:27
  zlib-ng#2 0x5874fa in zng_deflateInit_ zlib-ng/deflate.c:224:12
  zlib-ng#3 0x586c95 in zng_compress2 zlib-ng/compress.c:41:11
  zlib-ng#4 0x5866e3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:18:3
  zlib-ng#5 0x5862fd in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:38:3
  zlib-ng#6 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  zlib-ng#7 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  zlib-ng#8 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  zlib-ng#9 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  zlib-ng#10 0x7fea2fea482f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Aug 21, 2018
ninja test shows the following total running times:

2/7 Test zlib-ng#2: compress_fuzzer ..................   Passed    0.67 sec
3/7 Test zlib-ng#3: example_small_fuzzer .............   Passed    0.55 sec
4/7 Test zlib-ng#4: example_large_fuzzer .............   Passed    0.12 sec
5/7 Test zlib-ng#5: example_flush_fuzzer .............   Passed    0.20 sec
6/7 Test zlib-ng#6: example_dict_fuzzer ..............   Passed    1.16 sec

The patch also restricts all the fuzzers to process less than 1Mb input.
We will probably need to restrict the dict_fuzzer to less than 1Mb.
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Aug 24, 2018
in quick_send_bits zlib-ng/arch/x86/deflate_quick.c:134 we are writing 4 bytes:

  *(unsigned *)(s->pending_buf + s->pending) = out;

that runs over the allocated buffer by 1 byte.  The memory sanitizer crashes
like this:

MemorySanitizer:DEADLYSIGNAL
--
  | ==1==ERROR: MemorySanitizer: SEGV on unknown address 0x730000020000 (pc 0x0000005b6ce4 bp 0x7fff59adb5e0 sp 0x7fff59adb570 T1)
  | ==1==The signal is caused by a WRITE memory access.
  | #0 0x5b6ce3 in quick_send_bits zlib-ng/arch/x86/deflate_quick.c:134:48
  | zlib-ng#1 0x5b5752 in deflate_quick zlib-ng/arch/x86/deflate_quick.c:243:21
  | zlib-ng#2 0x590a15 in zng_deflate zlib-ng/deflate.c:952:18
  | zlib-ng#3 0x587165 in zng_compress2 zlib-ng/compress.c:59:15
  | zlib-ng#4 0x5866d3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:22:3
  | zlib-ng#5 0x5862d8 in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:74:3
  | zlib-ng#6 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  | zlib-ng#7 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  | zlib-ng#8 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  | zlib-ng#9 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  | zlib-ng#10 0x7fb8919b082f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
  | zlib-ng#11 0x41ec68 in _start
  | MemorySanitizer can not provide additional info.
  | SUMMARY: MemorySanitizer: SEGV (/mnt/scratch0/clusterfuzz/slave-bot/builds/clusterfuzz-builds_zlib-ng_7ead0a3e4980f024583384fd355b6e3ddd4b2ca2/revisions/compress_fuzzer+0x5b6ce3)

This patch allocates 4 extra bytes in s->pending_buf such that the writes are
still falling in the allocated memory.
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Aug 24, 2018
…ing_buf_size

to avoid runing over the allocated buffer:

MemorySanitizer:DEADLYSIGNAL
--
  | ==1==ERROR: MemorySanitizer: SEGV on unknown address 0x730000020000 (pc 0x0000005b6ce4 bp 0x7fff59adb5e0 sp 0x7fff59adb570 T1)
  | ==1==The signal is caused by a WRITE memory access.
  | #0 0x5b6ce3 in quick_send_bits zlib-ng/arch/x86/deflate_quick.c:134:48
  | zlib-ng#1 0x5b5752 in deflate_quick zlib-ng/arch/x86/deflate_quick.c:243:21
  | zlib-ng#2 0x590a15 in zng_deflate zlib-ng/deflate.c:952:18
  | zlib-ng#3 0x587165 in zng_compress2 zlib-ng/compress.c:59:15
  | zlib-ng#4 0x5866d3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:22:3
  | zlib-ng#5 0x5862d8 in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:74:3
  | zlib-ng#6 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  | zlib-ng#7 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  | zlib-ng#8 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  | zlib-ng#9 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  | zlib-ng#10 0x7fb8919b082f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
  | zlib-ng#11 0x41ec68 in _start
  | MemorySanitizer can not provide additional info.
  | SUMMARY: MemorySanitizer: SEGV (/mnt/scratch0/clusterfuzz/slave-bot/builds/clusterfuzz-builds_zlib-ng_7ead0a3e4980f024583384fd355b6e3ddd4b2ca2/revisions/compress_fuzzer+0x5b6ce3)

6 bytes before the end of the buffer is safe as each write to s->pending_buf is
4 bytes long, and there are two writes.  After the first write the index
s->pending is incremented by a maximum 2:

  length = quick_len_codes[lc] & 0xFF;

Max length by reading through all values in quick_len_codes[] is 13.

  s->bi_valid = width - (bytes_out * 8);

Max s->bi_valid is 7.

  width = s->bi_valid + length;

Max width is 20.

  bytes_out = width / 8;

Max bytes_out is 2.

  s->pending += bytes_out;

Max increment for s->pending is 2.
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Aug 25, 2018
…e end of s->pending_buf

to avoid runing over the allocated buffer:

MemorySanitizer:DEADLYSIGNAL
--
  | ==1==ERROR: MemorySanitizer: SEGV on unknown address 0x730000020000 (pc 0x0000005b6ce4 bp 0x7fff59adb5e0 sp 0x7fff59adb570 T1)
  | ==1==The signal is caused by a WRITE memory access.
  | #0 0x5b6ce3 in quick_send_bits zlib-ng/arch/x86/deflate_quick.c:134:48
  | zlib-ng#1 0x5b5752 in deflate_quick zlib-ng/arch/x86/deflate_quick.c:243:21
  | zlib-ng#2 0x590a15 in zng_deflate zlib-ng/deflate.c:952:18
  | zlib-ng#3 0x587165 in zng_compress2 zlib-ng/compress.c:59:15
  | zlib-ng#4 0x5866d3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:22:3
  | zlib-ng#5 0x5862d8 in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:74:3
  | zlib-ng#6 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  | zlib-ng#7 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  | zlib-ng#8 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  | zlib-ng#9 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  | zlib-ng#10 0x7fb8919b082f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
  | zlib-ng#11 0x41ec68 in _start
  | MemorySanitizer can not provide additional info.
  | SUMMARY: MemorySanitizer: SEGV (/mnt/scratch0/clusterfuzz/slave-bot/builds/clusterfuzz-builds_zlib-ng_7ead0a3e4980f024583384fd355b6e3ddd4b2ca2/revisions/compress_fuzzer+0x5b6ce3)

6 bytes before the end of the buffer is safe as each write to s->pending_buf is
4 bytes long, and there are two writes.  After the first write the index
s->pending is incremented by a maximum 2:

  length = quick_len_codes[lc] & 0xFF;

Max length by reading through all values in quick_len_codes[] is 13.

  s->bi_valid = width - (bytes_out * 8);

Max s->bi_valid is 7.

  width = s->bi_valid + length;

Max width is 20.

  bytes_out = width / 8;

Max bytes_out is 2.

  s->pending += bytes_out;

Max increment for s->pending is 2.
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Aug 25, 2018
…ized-value

==1==WARNING: MemorySanitizer: use-of-uninitialized-value
  #0 0x59fa93 in deflate_medium zlib-ng/deflate_medium.c:259:21
  zlib-ng#1 0x590905 in zng_deflate zlib-ng/deflate.c:951:18
  zlib-ng#2 0x587095 in zng_compress2 zlib-ng/compress.c:59:15
  zlib-ng#3 0x5866e3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:18:3
  zlib-ng#4 0x5862fd in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:38:3
  zlib-ng#5 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  zlib-ng#6 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  zlib-ng#7 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  zlib-ng#8 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  zlib-ng#9 0x7fea2fea482f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
  zlib-ng#10 0x41ec68 in _start
Uninitialized value was created by a heap allocation
  #0 0x45f2a0 in malloc /src/llvm/projects/compiler-rt/lib/msan/msan_interceptors.cc:910
  zlib-ng#1 0x587d42 in zng_deflateInit2_ zlib-ng/deflate.c:284:27
  zlib-ng#2 0x5874fa in zng_deflateInit_ zlib-ng/deflate.c:224:12
  zlib-ng#3 0x586c95 in zng_compress2 zlib-ng/compress.c:41:11
  zlib-ng#4 0x5866e3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:18:3
  zlib-ng#5 0x5862fd in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:38:3
  zlib-ng#6 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  zlib-ng#7 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  zlib-ng#8 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  zlib-ng#9 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  zlib-ng#10 0x7fea2fea482f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Aug 25, 2018
==4908==ERROR: MemorySanitizer: SEGV on unknown address 0x730fffffffff (pc 0x0000004b1b97 bp 0x7ffd4bf59a00 sp 0x7ffd4bf598a0 T4908)
==4908==The signal is caused by a READ memory access.
  #0 0x5a0599 in fizzle_matches zlib-ng/deflate_medium.c:168:12
  zlib-ng#1 0x59ea27 in deflate_medium zlib-ng/deflate_medium.c:296:21
  zlib-ng#2 0x5901c5 in zng_deflate zlib-ng/deflate.c:951:18
  zlib-ng#3 0x586955 in zng_compress2 zlib-ng/compress.c:59:15
  zlib-ng#4 0x5861eb in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:18:3
  zlib-ng#5 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  zlib-ng#6 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  zlib-ng#7 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  zlib-ng#8 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  zlib-ng#9 0x7fa3d7ff582f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
  zlib-ng#10 0x41ec68 in _start
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Aug 26, 2018
…ized-value

==1==WARNING: MemorySanitizer: use-of-uninitialized-value
  #0 0x59fa93 in deflate_medium zlib-ng/deflate_medium.c:259:21
  zlib-ng#1 0x590905 in zng_deflate zlib-ng/deflate.c:951:18
  zlib-ng#2 0x587095 in zng_compress2 zlib-ng/compress.c:59:15
  zlib-ng#3 0x5866e3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:18:3
  zlib-ng#4 0x5862fd in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:38:3
  zlib-ng#5 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  zlib-ng#6 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  zlib-ng#7 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  zlib-ng#8 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  zlib-ng#9 0x7fea2fea482f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
  zlib-ng#10 0x41ec68 in _start
Uninitialized value was created by a heap allocation
  #0 0x45f2a0 in malloc /src/llvm/projects/compiler-rt/lib/msan/msan_interceptors.cc:910
  zlib-ng#1 0x587d42 in zng_deflateInit2_ zlib-ng/deflate.c:284:27
  zlib-ng#2 0x5874fa in zng_deflateInit_ zlib-ng/deflate.c:224:12
  zlib-ng#3 0x586c95 in zng_compress2 zlib-ng/compress.c:41:11
  zlib-ng#4 0x5866e3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:18:3
  zlib-ng#5 0x5862fd in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:38:3
  zlib-ng#6 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  zlib-ng#7 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  zlib-ng#8 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  zlib-ng#9 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  zlib-ng#10 0x7fea2fea482f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Aug 26, 2018
…deflate_quick

by aggregating the two consecutive values to be written by static_emit_ptr to
s->pending_buf and writing the two values at once in a 4 byte store, we avoid
running out of the allocated buffer. We used to call quick_send_bits twice and
bumped the counter s->pending in the first call, which made the second call
write to memory beyond the safe 4 bytes that were guaranteed by the following
condition in the enclosing loop in deflate_quick:

  if (s->pending + 4 >= s->pending_buf_size) {
    flush_pending(s->strm);

The bug was exposed by the memory sanitizer like so:

MemorySanitizer:DEADLYSIGNAL
--
  | ==1==ERROR: MemorySanitizer: SEGV on unknown address 0x730000020000 (pc 0x0000005b6ce4 bp 0x7fff59adb5e0 sp 0x7fff59adb570 T1)
  | ==1==The signal is caused by a WRITE memory access.
  | #0 0x5b6ce3 in quick_send_bits zlib-ng/arch/x86/deflate_quick.c:134:48
  | zlib-ng#1 0x5b5752 in deflate_quick zlib-ng/arch/x86/deflate_quick.c:243:21
  | zlib-ng#2 0x590a15 in zng_deflate zlib-ng/deflate.c:952:18
  | zlib-ng#3 0x587165 in zng_compress2 zlib-ng/compress.c:59:15
  | zlib-ng#4 0x5866d3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:22:3
  | zlib-ng#5 0x5862d8 in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:74:3
  | zlib-ng#6 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  | zlib-ng#7 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  | zlib-ng#8 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  | zlib-ng#9 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  | zlib-ng#10 0x7fb8919b082f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
  | zlib-ng#11 0x41ec68 in _start
  | MemorySanitizer can not provide additional info.
  | SUMMARY: MemorySanitizer: SEGV (/mnt/scratch0/clusterfuzz/slave-bot/builds/clusterfuzz-builds_zlib-ng_7ead0a3e4980f024583384fd355b6e3ddd4b2ca2/revisions/compress_fuzzer+0x5b6ce3)
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Aug 26, 2018
…deflate_quick

by aggregating the two consecutive values to be written by static_emit_ptr to
s->pending_buf and writing the two values at once in a 4 byte store, we avoid
running out of the allocated buffer. We used to call quick_send_bits twice and
bumped the counter s->pending in the first call, which made the second call
write to memory beyond the safe 4 bytes that were guaranteed by the following
condition in the enclosing loop in deflate_quick:

  if (s->pending + 4 >= s->pending_buf_size) {
    flush_pending(s->strm);

The bug was exposed by the memory sanitizer like so:

MemorySanitizer:DEADLYSIGNAL
--
  | ==1==ERROR: MemorySanitizer: SEGV on unknown address 0x730000020000 (pc 0x0000005b6ce4 bp 0x7fff59adb5e0 sp 0x7fff59adb570 T1)
  | ==1==The signal is caused by a WRITE memory access.
  | #0 0x5b6ce3 in quick_send_bits zlib-ng/arch/x86/deflate_quick.c:134:48
  | zlib-ng#1 0x5b5752 in deflate_quick zlib-ng/arch/x86/deflate_quick.c:243:21
  | zlib-ng#2 0x590a15 in zng_deflate zlib-ng/deflate.c:952:18
  | zlib-ng#3 0x587165 in zng_compress2 zlib-ng/compress.c:59:15
  | zlib-ng#4 0x5866d3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:22:3
  | zlib-ng#5 0x5862d8 in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:74:3
  | zlib-ng#6 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  | zlib-ng#7 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  | zlib-ng#8 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  | zlib-ng#9 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  | zlib-ng#10 0x7fb8919b082f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
  | zlib-ng#11 0x41ec68 in _start
  | MemorySanitizer can not provide additional info.
  | SUMMARY: MemorySanitizer: SEGV (/mnt/scratch0/clusterfuzz/slave-bot/builds/clusterfuzz-builds_zlib-ng_7ead0a3e4980f024583384fd355b6e3ddd4b2ca2/revisions/compress_fuzzer+0x5b6ce3)
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Aug 27, 2018
to avoid the following ubsan error:
/src/zlib-ng/arch/x86/deflate_quick.c:134:66: runtime error: left shift of 2049 by 20 places cannot be represented in type 'int'
#0 0x4759d4 in quick_send_bits zlib-ng/arch/x86/deflate_quick.c:134:66
zlib-ng#1 0x474d7a in deflate_quick zlib-ng/arch/x86/deflate_quick.c:246:21
zlib-ng#2 0x46683c in zng_deflate zlib-ng/deflate.c:952:18
zlib-ng#3 0x4635b5 in zng_compress2 zlib-ng/compress.c:59:15
zlib-ng#4 0x46325c in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:22:3
zlib-ng#5 0x463168 in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:74:3
zlib-ng#6 0x44015c in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
zlib-ng#7 0x430e9d in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
zlib-ng#8 0x434dbb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
zlib-ng#9 0x430b88 in main /src/libfuzzer/FuzzerMain.cpp:20:10
zlib-ng#10 0x7f4cacc7c82f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
zlib-ng#11 0x405cb8 in _start
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Aug 27, 2018
…deflate_quick

by aggregating the two consecutive values to be written by static_emit_ptr to
s->pending_buf and writing the two values at once in a 4 byte store, we avoid
running out of the allocated buffer. We used to call quick_send_bits twice and
bumped the counter s->pending in the first call, which made the second call
write to memory beyond the safe 4 bytes that were guaranteed by the following
condition in the enclosing loop in deflate_quick:

  if (s->pending + 4 >= s->pending_buf_size) {
    flush_pending(s->strm);

The bug was exposed by the memory sanitizer like so:

MemorySanitizer:DEADLYSIGNAL
--
  | ==1==ERROR: MemorySanitizer: SEGV on unknown address 0x730000020000 (pc 0x0000005b6ce4 bp 0x7fff59adb5e0 sp 0x7fff59adb570 T1)
  | ==1==The signal is caused by a WRITE memory access.
  | #0 0x5b6ce3 in quick_send_bits zlib-ng/arch/x86/deflate_quick.c:134:48
  | zlib-ng#1 0x5b5752 in deflate_quick zlib-ng/arch/x86/deflate_quick.c:243:21
  | zlib-ng#2 0x590a15 in zng_deflate zlib-ng/deflate.c:952:18
  | zlib-ng#3 0x587165 in zng_compress2 zlib-ng/compress.c:59:15
  | zlib-ng#4 0x5866d3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:22:3
  | zlib-ng#5 0x5862d8 in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:74:3
  | zlib-ng#6 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  | zlib-ng#7 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  | zlib-ng#8 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  | zlib-ng#9 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  | zlib-ng#10 0x7fb8919b082f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
  | zlib-ng#11 0x41ec68 in _start
  | MemorySanitizer can not provide additional info.
  | SUMMARY: MemorySanitizer: SEGV (/mnt/scratch0/clusterfuzz/slave-bot/builds/clusterfuzz-builds_zlib-ng_7ead0a3e4980f024583384fd355b6e3ddd4b2ca2/revisions/compress_fuzzer+0x5b6ce3)
Dead2 pushed a commit that referenced this issue Sep 17, 2018
…_quick

by aggregating the two consecutive values to be written by static_emit_ptr to
s->pending_buf and writing the two values at once in a 4 byte store, we avoid
running out of the allocated buffer. We used to call quick_send_bits twice and
bumped the counter s->pending in the first call, which made the second call
write to memory beyond the safe 4 bytes that were guaranteed by the following
condition in the enclosing loop in deflate_quick:

  if (s->pending + 4 >= s->pending_buf_size) {
    flush_pending(s->strm);

The bug was exposed by the memory sanitizer like so:

MemorySanitizer:DEADLYSIGNAL
--
  | ==1==ERROR: MemorySanitizer: SEGV on unknown address 0x730000020000 (pc 0x0000005b6ce4 bp 0x7fff59adb5e0 sp 0x7fff59adb570 T1)
  | ==1==The signal is caused by a WRITE memory access.
  | #0 0x5b6ce3 in quick_send_bits zlib-ng/arch/x86/deflate_quick.c:134:48
  | #1 0x5b5752 in deflate_quick zlib-ng/arch/x86/deflate_quick.c:243:21
  | #2 0x590a15 in zng_deflate zlib-ng/deflate.c:952:18
  | #3 0x587165 in zng_compress2 zlib-ng/compress.c:59:15
  | #4 0x5866d3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:22:3
  | #5 0x5862d8 in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:74:3
  | #6 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  | #7 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  | #8 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  | #9 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  | #10 0x7fb8919b082f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
  | #11 0x41ec68 in _start
  | MemorySanitizer can not provide additional info.
  | SUMMARY: MemorySanitizer: SEGV (/mnt/scratch0/clusterfuzz/slave-bot/builds/clusterfuzz-builds_zlib-ng_7ead0a3e4980f024583384fd355b6e3ddd4b2ca2/revisions/compress_fuzzer+0x5b6ce3)
Dead2 pushed a commit that referenced this issue Sep 17, 2018
==1==WARNING: MemorySanitizer: use-of-uninitialized-value
  #0 0x59fa93 in deflate_medium zlib-ng/deflate_medium.c:259:21
  #1 0x590905 in zng_deflate zlib-ng/deflate.c:951:18
  #2 0x587095 in zng_compress2 zlib-ng/compress.c:59:15
  #3 0x5866e3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:18:3
  #4 0x5862fd in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:38:3
  #5 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  #6 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  #7 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  #8 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  #9 0x7fea2fea482f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
  #10 0x41ec68 in _start
Uninitialized value was created by a heap allocation
  #0 0x45f2a0 in malloc /src/llvm/projects/compiler-rt/lib/msan/msan_interceptors.cc:910
  #1 0x587d42 in zng_deflateInit2_ zlib-ng/deflate.c:284:27
  #2 0x5874fa in zng_deflateInit_ zlib-ng/deflate.c:224:12
  #3 0x586c95 in zng_compress2 zlib-ng/compress.c:41:11
  #4 0x5866e3 in check_compress_level zlib-ng/test/fuzz/compress_fuzzer.c:18:3
  #5 0x5862fd in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:38:3
  #6 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  #7 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  #8 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  #9 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  #10 0x7fea2fea482f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
Dead2 pushed a commit that referenced this issue Sep 17, 2018
==4908==ERROR: MemorySanitizer: SEGV on unknown address 0x730fffffffff (pc 0x0000004b1b97 bp 0x7ffd4bf59a00 sp 0x7ffd4bf598a0 T4908)
==4908==The signal is caused by a READ memory access.
  #0 0x5a0599 in fizzle_matches zlib-ng/deflate_medium.c:168:12
  #1 0x59ea27 in deflate_medium zlib-ng/deflate_medium.c:296:21
  #2 0x5901c5 in zng_deflate zlib-ng/deflate.c:951:18
  #3 0x586955 in zng_compress2 zlib-ng/compress.c:59:15
  #4 0x5861eb in LLVMFuzzerTestOneInput zlib-ng/test/fuzz/compress_fuzzer.c:18:3
  #5 0x4e9b48 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  #6 0x4a2f66 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  #7 0x4b3adb in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  #8 0x4a2091 in main /src/libfuzzer/FuzzerMain.cpp:20:10
  #9 0x7fa3d7ff582f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
  #10 0x41ec68 in _start
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Oct 31, 2018
Before this patch

cmake -DWITH_SANITIZERS=1
make
make test

used to fail with:

Running tests...
Test project /home/hansr/github/zlib/zlib-ng
    Start 1: example
1/2 Test zlib-ng#1: example ..........................***Failed    0.14 sec
    Start 2: example64
2/2 Test zlib-ng#2: example64 ........................***Failed    0.13 sec

==11605==ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges [0x62e000000595,0x62e0000053b5) and [0x62e000000400, 0x62e000005220) overlap
    #0 0x7fab3bcc9662 in __asan_memcpy (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x8c662)
    zlib-ng#1 0x40f936 in memcpy /usr/include/x86_64-linux-gnu/bits/string3.h:53
    zlib-ng#2 0x40f936 in read_buf /home/spop/s/zlib-ng/deflate.c:1122
    zlib-ng#3 0x410458 in deflate_stored /home/spop/s/zlib-ng/deflate.c:1394
    zlib-ng#4 0x4133d7 in zng_deflate /home/spop/s/zlib-ng/deflate.c:945
    zlib-ng#5 0x402253 in test_large_deflate /home/spop/s/zlib-ng/test/example.c:275
    zlib-ng#6 0x4014e8 in main /home/spop/s/zlib-ng/test/example.c:536
    zlib-ng#7 0x7fab3b89382f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    zlib-ng#8 0x4018e8 in _start (/work/spop/zlib-ng/example+0x4018e8)

0x62e000000595 is located 405 bytes inside of 40000-byte region [0x62e000000400,0x62e00000a040)
allocated by thread T0 here:
    #0 0x7fab3bcd579a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
    zlib-ng#1 0x40147a in main /home/spop/s/zlib-ng/test/example.c:516

0x62e000000400 is located 0 bytes inside of 40000-byte region [0x62e000000400,0x62e00000a040)
allocated by thread T0 here:
    #0 0x7fab3bcd579a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
    zlib-ng#1 0x40147a in main /home/spop/s/zlib-ng/test/example.c:516

SUMMARY: AddressSanitizer: memcpy-param-overlap ??:0 __asan_memcpy
==11605==ABORTING

The abort is on the following call to deflate():

    c_stream.next_out = compr;
    c_stream.avail_out = (unsigned int)comprLen;
    [...]

    /* Feed in already compressed data and switch to no compression: */
    PREFIX(deflateParams)(&c_stream, Z_NO_COMPRESSION, Z_DEFAULT_STRATEGY);
    c_stream.next_in = compr;
    c_stream.avail_in = (unsigned int)comprLen/2;
    err = PREFIX(deflate)(&c_stream, Z_NO_FLUSH);

with c_stream.next_out and c_stream.next_in pointing to the same memory array.
The patch reduces the size of the array for in and out to not overlap.

fix bug zlib-ng#183 following recommendations of Mika Lindqvist

 > the problem is in line c_stream.avail_in = (unsigned int)comprLen/2;
 > which feeds it too much data ... it should cap it to
 > c_stream.next_out - compr instead.
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Oct 31, 2018
Before this patch

cmake -DWITH_SANITIZERS=1
make
make test

used to fail with:

Running tests...
Test project /home/hansr/github/zlib/zlib-ng
    Start 1: example
1/2 Test zlib-ng#1: example ..........................***Failed    0.14 sec
    Start 2: example64
2/2 Test zlib-ng#2: example64 ........................***Failed    0.13 sec

==11605==ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges [0x62e000000595,0x62e0000053b5) and [0x62e000000400, 0x62e000005220) overlap
    #0 0x7fab3bcc9662 in __asan_memcpy (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x8c662)
    zlib-ng#1 0x40f936 in memcpy /usr/include/x86_64-linux-gnu/bits/string3.h:53
    zlib-ng#2 0x40f936 in read_buf /home/spop/s/zlib-ng/deflate.c:1122
    zlib-ng#3 0x410458 in deflate_stored /home/spop/s/zlib-ng/deflate.c:1394
    zlib-ng#4 0x4133d7 in zng_deflate /home/spop/s/zlib-ng/deflate.c:945
    zlib-ng#5 0x402253 in test_large_deflate /home/spop/s/zlib-ng/test/example.c:275
    zlib-ng#6 0x4014e8 in main /home/spop/s/zlib-ng/test/example.c:536
    zlib-ng#7 0x7fab3b89382f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    zlib-ng#8 0x4018e8 in _start (/work/spop/zlib-ng/example+0x4018e8)

0x62e000000595 is located 405 bytes inside of 40000-byte region [0x62e000000400,0x62e00000a040)
allocated by thread T0 here:
    #0 0x7fab3bcd579a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
    zlib-ng#1 0x40147a in main /home/spop/s/zlib-ng/test/example.c:516

0x62e000000400 is located 0 bytes inside of 40000-byte region [0x62e000000400,0x62e00000a040)
allocated by thread T0 here:
    #0 0x7fab3bcd579a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
    zlib-ng#1 0x40147a in main /home/spop/s/zlib-ng/test/example.c:516

SUMMARY: AddressSanitizer: memcpy-param-overlap ??:0 __asan_memcpy
==11605==ABORTING

fix bug zlib-ng#183 following recommendations of Mika Lindqvist

 > the problem is in line c_stream.avail_in = (unsigned int)comprLen/2;
 > which feeds it too much data ... it should cap it to
 > c_stream.next_out - compr instead.
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Oct 31, 2018
Before this patch, when configuring with address sanitizer:

./configure --with-sanitizers
make
make test

used to fail with the following error:

$ echo hello world | ./minigzip
ASAN:SIGSEGV
=================================================================
==17466==ERROR: AddressSanitizer: SEGV on unknown address 0x00000000fc80 (pc 0x7fcacddd46f8 bp 0x7ffd01ceb310 sp 0x7ffd01ceb290 T0)
    #0 0x7fcacddd46f7 in _IO_fwrite (/lib/x86_64-linux-gnu/libc.so.6+0x6e6f7)
    zlib-ng#1 0x402602 in zng_gzwrite /home/spop/s/zlib-ng/test/minigzip.c:180
    zlib-ng#2 0x403445 in gz_compress /home/spop/s/zlib-ng/test/minigzip.c:305
    zlib-ng#3 0x404724 in main /home/spop/s/zlib-ng/test/minigzip.c:509
    zlib-ng#4 0x7fcacdd8682f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    zlib-ng#5 0x4018d8 in _start (/work/spop/zlib-ng/minigzip+0x4018d8)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV ??:0 _IO_fwrite
==17466==ABORTING

During compilation the following warnings point to a missing definition:

/home/spop/s/zlib-ng/test/minigzip.c:154:31: warning: implicit declaration of function 'fdopen' is invalid in C99 [-Wimplicit-function-declaration]
    gz->file = path == NULL ? fdopen(fd, gz->write ? "wb" : "rb") :
                              ^
/home/spop/s/zlib-ng/test/minigzip.c:154:29: warning: pointer/integer type mismatch in conditional expression ('int' and 'FILE *' (aka 'struct _IO_FILE *')) [-Wconditional-type-mismatch]
    gz->file = path == NULL ? fdopen(fd, gz->write ? "wb" : "rb") :
                            ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/spop/s/zlib-ng/test/minigzip.c:504:36: warning: implicit declaration of function 'fileno' is invalid in C99 [-Wimplicit-function-declaration]
            file = PREFIX(gzdopen)(fileno(stdin), "rb");
                                   ^
/home/spop/s/zlib-ng/test/minigzip.c:508:36: warning: implicit declaration of function 'fileno' is invalid in C99 [-Wimplicit-function-declaration]
            file = PREFIX(gzdopen)(fileno(stdout), outmode);
                                   ^
/home/spop/s/zlib-ng/test/minigzip.c:534:48: warning: implicit declaration of function 'fileno' is invalid in C99 [-Wimplicit-function-declaration]
                        file = PREFIX(gzdopen)(fileno(stdout), outmode);
                                               ^
5 warnings generated.

and looking at stdio.h that defines fdopen we see that it is only defined under
__USE_POSIX:

 #ifdef	__USE_POSIX
/* Create a new stream that refers to an existing system file descriptor.  */
extern FILE *fdopen (int __fd, const char *__modes) __THROW __wur;
 #endif

This patch fixes the compiler warnings and the runtime ASAN error.
Dead2 pushed a commit that referenced this issue Nov 2, 2018
Before this patch, when configuring with address sanitizer:

./configure --with-sanitizers
make
make test

used to fail with the following error:

$ echo hello world | ./minigzip
ASAN:SIGSEGV
=================================================================
==17466==ERROR: AddressSanitizer: SEGV on unknown address 0x00000000fc80 (pc 0x7fcacddd46f8 bp 0x7ffd01ceb310 sp 0x7ffd01ceb290 T0)
    #0 0x7fcacddd46f7 in _IO_fwrite (/lib/x86_64-linux-gnu/libc.so.6+0x6e6f7)
    #1 0x402602 in zng_gzwrite /home/spop/s/zlib-ng/test/minigzip.c:180
    #2 0x403445 in gz_compress /home/spop/s/zlib-ng/test/minigzip.c:305
    #3 0x404724 in main /home/spop/s/zlib-ng/test/minigzip.c:509
    #4 0x7fcacdd8682f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #5 0x4018d8 in _start (/work/spop/zlib-ng/minigzip+0x4018d8)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV ??:0 _IO_fwrite
==17466==ABORTING

During compilation the following warnings point to a missing definition:

/home/spop/s/zlib-ng/test/minigzip.c:154:31: warning: implicit declaration of function 'fdopen' is invalid in C99 [-Wimplicit-function-declaration]
    gz->file = path == NULL ? fdopen(fd, gz->write ? "wb" : "rb") :
                              ^
/home/spop/s/zlib-ng/test/minigzip.c:154:29: warning: pointer/integer type mismatch in conditional expression ('int' and 'FILE *' (aka 'struct _IO_FILE *')) [-Wconditional-type-mismatch]
    gz->file = path == NULL ? fdopen(fd, gz->write ? "wb" : "rb") :
                            ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/spop/s/zlib-ng/test/minigzip.c:504:36: warning: implicit declaration of function 'fileno' is invalid in C99 [-Wimplicit-function-declaration]
            file = PREFIX(gzdopen)(fileno(stdin), "rb");
                                   ^
/home/spop/s/zlib-ng/test/minigzip.c:508:36: warning: implicit declaration of function 'fileno' is invalid in C99 [-Wimplicit-function-declaration]
            file = PREFIX(gzdopen)(fileno(stdout), outmode);
                                   ^
/home/spop/s/zlib-ng/test/minigzip.c:534:48: warning: implicit declaration of function 'fileno' is invalid in C99 [-Wimplicit-function-declaration]
                        file = PREFIX(gzdopen)(fileno(stdout), outmode);
                                               ^
5 warnings generated.

and looking at stdio.h that defines fdopen we see that it is only defined under
__USE_POSIX:

 #ifdef	__USE_POSIX
/* Create a new stream that refers to an existing system file descriptor.  */
extern FILE *fdopen (int __fd, const char *__modes) __THROW __wur;
 #endif

This patch fixes the compiler warnings and the runtime ASAN error.
Dead2 pushed a commit that referenced this issue Nov 7, 2018
Before this patch

cmake -DWITH_SANITIZERS=1
make
make test

used to fail with:

Running tests...
Test project /home/hansr/github/zlib/zlib-ng
    Start 1: example
1/2 Test #1: example ..........................***Failed    0.14 sec
    Start 2: example64
2/2 Test #2: example64 ........................***Failed    0.13 sec

==11605==ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges [0x62e000000595,0x62e0000053b5) and [0x62e000000400, 0x62e000005220) overlap
    #0 0x7fab3bcc9662 in __asan_memcpy (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x8c662)
    #1 0x40f936 in memcpy /usr/include/x86_64-linux-gnu/bits/string3.h:53
    #2 0x40f936 in read_buf /home/spop/s/zlib-ng/deflate.c:1122
    #3 0x410458 in deflate_stored /home/spop/s/zlib-ng/deflate.c:1394
    #4 0x4133d7 in zng_deflate /home/spop/s/zlib-ng/deflate.c:945
    #5 0x402253 in test_large_deflate /home/spop/s/zlib-ng/test/example.c:275
    #6 0x4014e8 in main /home/spop/s/zlib-ng/test/example.c:536
    #7 0x7fab3b89382f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #8 0x4018e8 in _start (/work/spop/zlib-ng/example+0x4018e8)

0x62e000000595 is located 405 bytes inside of 40000-byte region [0x62e000000400,0x62e00000a040)
allocated by thread T0 here:
    #0 0x7fab3bcd579a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
    #1 0x40147a in main /home/spop/s/zlib-ng/test/example.c:516

0x62e000000400 is located 0 bytes inside of 40000-byte region [0x62e000000400,0x62e00000a040)
allocated by thread T0 here:
    #0 0x7fab3bcd579a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
    #1 0x40147a in main /home/spop/s/zlib-ng/test/example.c:516

SUMMARY: AddressSanitizer: memcpy-param-overlap ??:0 __asan_memcpy
==11605==ABORTING

fix bug #183 following recommendations of Mika Lindqvist

 > the problem is in line c_stream.avail_in = (unsigned int)comprLen/2;
 > which feeds it too much data ... it should cap it to
 > c_stream.next_out - compr instead.
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Nov 10, 2018
==397==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x4e8193 in fill_window_sse /src/zlib-ng/arch/x86/fill_window_sse.c:52:30
    zlib-ng#1 0x4e9254 in deflate_quick /src/zlib-ng/arch/x86/deflate_quick.c:220:13
    zlib-ng#2 0x4ad375 in zng_deflate /src/zlib-ng/deflate.c:951:18
    zlib-ng#3 0x49fef1 in test_large_deflate /src/zlib-ng/test/example.c:264:11
    zlib-ng#4 0x4a3232 in main /src/zlib-ng/test/example.c:536:5
    zlib-ng#5 0x7f927f42d82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    zlib-ng#6 0x41b3a8 in _start (/src/zlib-ng/example+0x41b3a8)
  Uninitialized value was created by a heap allocation
    #0 0x45b9e0 in malloc /src/llvm/projects/compiler-rt/lib/msan/msan_interceptors.cc:910
    zlib-ng#1 0x4a47b2 in zng_deflateInit2_ /src/zlib-ng/deflate.c:284:27
    zlib-ng#2 0x4a3f6a in zng_deflateInit_ /src/zlib-ng/deflate.c:224:12
    zlib-ng#3 0x49fdd5 in test_large_deflate /src/zlib-ng/test/example.c:253:11
    zlib-ng#4 0x4a3232 in main /src/zlib-ng/test/example.c:536:5
    zlib-ng#5 0x7f927f42d82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Nov 10, 2018
zlib-ng compiled with MSAN used to fail with:

SUMMARY: MemorySanitizer: use-of-uninitialized-value /src/zlib-ng/match.c:473:60 in longest_match
Exiting

  Uninitialized value was stored to memory at
    #0 0x7fcaced77645 in fill_window_sse /src/zlib-ng/arch/x86/fill_window_sse.c:84:17
    zlib-ng#1 0x7fcaced7d3d4 in deflate_quick /src/zlib-ng/arch/x86/deflate_quick.c:230:13
    zlib-ng#2 0x7fcaced2f54b in zng_deflate /src/zlib-ng/deflate.c:951:18
    zlib-ng#3 0x4a04e9 in test_large_deflate /src/zlib-ng/test/example.c:266:11
    zlib-ng#4 0x4a38d2 in main /src/zlib-ng/test/example.c:539:5
    zlib-ng#5 0x7fcace96a82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

  Uninitialized value was created by a heap allocation
    #0 0x45bf70 in malloc /src/llvm/projects/compiler-rt/lib/msan/msan_interceptors.cc:910
    zlib-ng#1 0x7fcaced26cd9 in zng_deflateInit2_ /src/zlib-ng/deflate.c:315:26
    zlib-ng#2 0x7fcaced2605a in zng_deflateInit_ /src/zlib-ng/deflate.c:224:12
    zlib-ng#3 0x4a03c5 in test_large_deflate /src/zlib-ng/test/example.c:255:11
    zlib-ng#4 0x4a38d2 in main /src/zlib-ng/test/example.c:539:5
    zlib-ng#5 0x7fcace96a82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Nov 10, 2018
zlib-ng compiled with MSAN used to fail with:

SUMMARY: MemorySanitizer: use-of-uninitialized-value /src/zlib-ng/match.c:473:60 in longest_match
Exiting

  Uninitialized value was stored to memory at
    #0 0x7fcaced77645 in fill_window_sse /src/zlib-ng/arch/x86/fill_window_sse.c:84:17
    zlib-ng#1 0x7fcaced7d3d4 in deflate_quick /src/zlib-ng/arch/x86/deflate_quick.c:230:13
    zlib-ng#2 0x7fcaced2f54b in zng_deflate /src/zlib-ng/deflate.c:951:18
    zlib-ng#3 0x4a04e9 in test_large_deflate /src/zlib-ng/test/example.c:266:11
    zlib-ng#4 0x4a38d2 in main /src/zlib-ng/test/example.c:539:5
    zlib-ng#5 0x7fcace96a82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

  Uninitialized value was created by a heap allocation
    #0 0x45bf70 in malloc /src/llvm/projects/compiler-rt/lib/msan/msan_interceptors.cc:910
    zlib-ng#1 0x7fcaced26cd9 in zng_deflateInit2_ /src/zlib-ng/deflate.c:315:26
    zlib-ng#2 0x7fcaced2605a in zng_deflateInit_ /src/zlib-ng/deflate.c:224:12
    zlib-ng#3 0x4a03c5 in test_large_deflate /src/zlib-ng/test/example.c:255:11
    zlib-ng#4 0x4a38d2 in main /src/zlib-ng/test/example.c:539:5
    zlib-ng#5 0x7fcace96a82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
Dead2 pushed a commit that referenced this issue Nov 14, 2018
zlib-ng compiled with MSAN used to fail with:

SUMMARY: MemorySanitizer: use-of-uninitialized-value /src/zlib-ng/match.c:473:60 in longest_match
Exiting

  Uninitialized value was stored to memory at
    #0 0x7fcaced77645 in fill_window_sse /src/zlib-ng/arch/x86/fill_window_sse.c:84:17
    #1 0x7fcaced7d3d4 in deflate_quick /src/zlib-ng/arch/x86/deflate_quick.c:230:13
    #2 0x7fcaced2f54b in zng_deflate /src/zlib-ng/deflate.c:951:18
    #3 0x4a04e9 in test_large_deflate /src/zlib-ng/test/example.c:266:11
    #4 0x4a38d2 in main /src/zlib-ng/test/example.c:539:5
    #5 0x7fcace96a82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

  Uninitialized value was created by a heap allocation
    #0 0x45bf70 in malloc /src/llvm/projects/compiler-rt/lib/msan/msan_interceptors.cc:910
    #1 0x7fcaced26cd9 in zng_deflateInit2_ /src/zlib-ng/deflate.c:315:26
    #2 0x7fcaced2605a in zng_deflateInit_ /src/zlib-ng/deflate.c:224:12
    #3 0x4a03c5 in test_large_deflate /src/zlib-ng/test/example.c:255:11
    #4 0x4a38d2 in main /src/zlib-ng/test/example.c:539:5
    #5 0x7fcace96a82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Dec 17, 2018
configure --with-msan; make test fails on x86_64 with:

Uninitialized bytes in __interceptor_fwrite at offset 14 inside [0x72e000000000, 49152)
==706==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x498eba in zng_gzclose /home/spop/s/zlib-ng/test/minigzip.c:240:13
    zlib-ng#1 0x499982 in gz_compress /home/spop/s/zlib-ng/test/minigzip.c:313:9
    zlib-ng#2 0x49c352 in main /home/spop/s/zlib-ng/test/minigzip.c:514:13
    zlib-ng#3 0x7f8b954ae82f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/../csu/libc-start.c:291
    zlib-ng#4 0x41a9c8 in _start (/work/spop/zlib-ng/minigzip+0x41a9c8)

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/spop/s/zlib-ng/test/minigzip.c:240:13 in zng_gzclose

I think this is a bug in the memory sanitizer that may be confused by the use of
intrinsics in crc_fold_init. By moving the initialization of strm->adler out of
the crc_reset function, the memory sanitizer stops warning.
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Jan 16, 2019
configure --with-msan; make test fails on x86_64 with:

Uninitialized bytes in __interceptor_fwrite at offset 14 inside [0x72e000000000, 49152)
==706==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x498eba in zng_gzclose /home/spop/s/zlib-ng/test/minigzip.c:240:13
    zlib-ng#1 0x499982 in gz_compress /home/spop/s/zlib-ng/test/minigzip.c:313:9
    zlib-ng#2 0x49c352 in main /home/spop/s/zlib-ng/test/minigzip.c:514:13
    zlib-ng#3 0x7f8b954ae82f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/../csu/libc-start.c:291
    zlib-ng#4 0x41a9c8 in _start (/work/spop/zlib-ng/minigzip+0x41a9c8)

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/spop/s/zlib-ng/test/minigzip.c:240:13 in zng_gzclose

I think this is a bug in the memory sanitizer that may be confused by the use of
intrinsics in crc_fold_init. By moving the initialization of strm->adler out of
crc_fold_init and inlining crc_reset the memory sanitizer stops warning.
sebpop pushed a commit to sebpop/zlib-ng that referenced this issue Jan 16, 2019
configure --with-msan; make test fails on x86_64 with:

Uninitialized bytes in __interceptor_fwrite at offset 14 inside [0x72e000000000, 49152)
==706==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x498eba in zng_gzclose /home/spop/s/zlib-ng/test/minigzip.c:240:13
    zlib-ng#1 0x499982 in gz_compress /home/spop/s/zlib-ng/test/minigzip.c:313:9
    zlib-ng#2 0x49c352 in main /home/spop/s/zlib-ng/test/minigzip.c:514:13
    zlib-ng#3 0x7f8b954ae82f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/../csu/libc-start.c:291
    zlib-ng#4 0x41a9c8 in _start (/work/spop/zlib-ng/minigzip+0x41a9c8)

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/spop/s/zlib-ng/test/minigzip.c:240:13 in zng_gzclose

I think this is a bug in the memory sanitizer that may be confused by the use of
intrinsics in crc_fold_init. By moving the initialization of strm->adler out of
crc_fold_init and inlining crc_reset the memory sanitizer stops warning.
nmoinvaz added a commit to nmoinvaz/zlib-ng that referenced this issue Oct 31, 2020
When there are no symbols in the tree we skip build_tree calculations and emit a block using static tree with no codes.

trees.c:357:19: runtime error: unsigned integer overflow: 0 - 1 cannot be represented in type 'unsigned long'
    #0 0x1000ed79b in build_tree trees.c:357
    #1 0x1000ea3f5 in zng_tr_flush_block trees.c:649
    #2 0x100090ab0 in deflate_slow deflate_slow.c:131
    zlib-ng#3 0x1000572bc in zng_deflate deflate.c:990
    zlib-ng#4 0x1000aecd3 in gz_comp gzwrite.c:125
    zlib-ng#5 0x1000b05df in zng_gzclose_w gzwrite.c:511
    zlib-ng#6 0x1000967a4 in zng_gzclose gzlib.c:253
    zlib-ng#7 0x100004f70 in test_gzio example.c:133
    zlib-ng#8 0x100010c5b in main example.c:1034
    zlib-ng#9 0x7fff71f57cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
nmoinvaz added a commit to nmoinvaz/zlib-ng that referenced this issue Oct 31, 2020
When there are no symbols in the tree we skip build_tree calculations and emit a block using static tree with no codes.

trees.c:357:19: runtime error: unsigned integer overflow: 0 - 1 cannot be represented in type 'unsigned long'
    #0 0x1000ed79b in build_tree trees.c:357
    #1 0x1000ea3f5 in zng_tr_flush_block trees.c:649
    #2 0x100090ab0 in deflate_slow deflate_slow.c:131
    zlib-ng#3 0x1000572bc in zng_deflate deflate.c:990
    zlib-ng#4 0x1000aecd3 in gz_comp gzwrite.c:125
    zlib-ng#5 0x1000b05df in zng_gzclose_w gzwrite.c:511
    zlib-ng#6 0x1000967a4 in zng_gzclose gzlib.c:253
    zlib-ng#7 0x100004f70 in test_gzio example.c:133
    zlib-ng#8 0x100010c5b in main example.c:1034
    zlib-ng#9 0x7fff71f57cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
nmoinvaz added a commit to nmoinvaz/zlib-ng that referenced this issue Oct 31, 2020
When there are no symbols in the tree we skip build_tree calculations and emit a block using static tree with no codes.

trees.c:357:19: runtime error: unsigned integer overflow: 0 - 1 cannot be represented in type 'unsigned long'
    #0 0x1000ed79b in build_tree trees.c:357
    #1 0x1000ea3f5 in zng_tr_flush_block trees.c:649
    #2 0x100090ab0 in deflate_slow deflate_slow.c:131
    zlib-ng#3 0x1000572bc in zng_deflate deflate.c:990
    zlib-ng#4 0x1000aecd3 in gz_comp gzwrite.c:125
    zlib-ng#5 0x1000b05df in zng_gzclose_w gzwrite.c:511
    zlib-ng#6 0x1000967a4 in zng_gzclose gzlib.c:253
    zlib-ng#7 0x100004f70 in test_gzio example.c:133
    zlib-ng#8 0x100010c5b in main example.c:1034
    zlib-ng#9 0x7fff71f57cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
nmoinvaz added a commit to nmoinvaz/zlib-ng that referenced this issue Oct 31, 2020
When there are no symbols in the tree we skip build_tree calculations and emit a block using static tree with no codes.

trees.c:357:19: runtime error: unsigned integer overflow: 0 - 1 cannot be represented in type 'unsigned long'
    #0 0x1000ed79b in build_tree trees.c:357
    #1 0x1000ea3f5 in zng_tr_flush_block trees.c:649
    #2 0x100090ab0 in deflate_slow deflate_slow.c:131
    zlib-ng#3 0x1000572bc in zng_deflate deflate.c:990
    zlib-ng#4 0x1000aecd3 in gz_comp gzwrite.c:125
    zlib-ng#5 0x1000b05df in zng_gzclose_w gzwrite.c:511
    zlib-ng#6 0x1000967a4 in zng_gzclose gzlib.c:253
    zlib-ng#7 0x100004f70 in test_gzio example.c:133
    zlib-ng#8 0x100010c5b in main example.c:1034
    zlib-ng#9 0x7fff71f57cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
nmoinvaz added a commit to nmoinvaz/zlib-ng that referenced this issue Oct 31, 2020
When there are no symbols in the tree we skip build_tree calculations and emit a block using static tree with no codes.

trees.c:357:19: runtime error: unsigned integer overflow: 0 - 1 cannot be represented in type 'unsigned long'
    #0 0x1000ed79b in build_tree trees.c:357
    #1 0x1000ea3f5 in zng_tr_flush_block trees.c:649
    #2 0x100090ab0 in deflate_slow deflate_slow.c:131
    zlib-ng#3 0x1000572bc in zng_deflate deflate.c:990
    zlib-ng#4 0x1000aecd3 in gz_comp gzwrite.c:125
    zlib-ng#5 0x1000b05df in zng_gzclose_w gzwrite.c:511
    zlib-ng#6 0x1000967a4 in zng_gzclose gzlib.c:253
    zlib-ng#7 0x100004f70 in test_gzio example.c:133
    zlib-ng#8 0x100010c5b in main example.c:1034
    zlib-ng#9 0x7fff71f57cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
nmoinvaz added a commit to nmoinvaz/zlib-ng that referenced this issue Oct 31, 2020
gzread.c:398:18: runtime error: unsigned integer overflow: 2 * 18446744073709551615 cannot be represented in type 'unsigned long'
    #0 0x10009d31e in zng_gzfread gzread.c:398
    #1 0x100005b1a in test_gzio example.c:213
    #2 0x10001093b in main example.c:1034
    zlib-ng#3 0x7fff71f57cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
Dead2 pushed a commit that referenced this issue Nov 2, 2020
gzread.c:398:18: runtime error: unsigned integer overflow: 2 * 18446744073709551615 cannot be represented in type 'unsigned long'
    #0 0x10009d31e in zng_gzfread gzread.c:398
    #1 0x100005b1a in test_gzio example.c:213
    #2 0x10001093b in main example.c:1034
    #3 0x7fff71f57cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
nmoinvaz added a commit to nmoinvaz/zlib-ng that referenced this issue Nov 2, 2020
When there are no symbols in the tree we skip build_tree calculations and emit a block using static tree with no codes.

trees.c:357:19: runtime error: unsigned integer overflow: 0 - 1 cannot be represented in type 'unsigned long'
    #0 0x1000ed79b in build_tree trees.c:357
    #1 0x1000ea3f5 in zng_tr_flush_block trees.c:649
    #2 0x100090ab0 in deflate_slow deflate_slow.c:131
    zlib-ng#3 0x1000572bc in zng_deflate deflate.c:990
    zlib-ng#4 0x1000aecd3 in gz_comp gzwrite.c:125
    zlib-ng#5 0x1000b05df in zng_gzclose_w gzwrite.c:511
    zlib-ng#6 0x1000967a4 in zng_gzclose gzlib.c:253
    zlib-ng#7 0x100004f70 in test_gzio example.c:133
    zlib-ng#8 0x100010c5b in main example.c:1034
    zlib-ng#9 0x7fff71f57cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
Dead2 pushed a commit that referenced this issue Nov 2, 2020
When there are no symbols in the tree we skip build_tree calculations and emit a block using static tree with no codes.

trees.c:357:19: runtime error: unsigned integer overflow: 0 - 1 cannot be represented in type 'unsigned long'
    #0 0x1000ed79b in build_tree trees.c:357
    #1 0x1000ea3f5 in zng_tr_flush_block trees.c:649
    #2 0x100090ab0 in deflate_slow deflate_slow.c:131
    #3 0x1000572bc in zng_deflate deflate.c:990
    #4 0x1000aecd3 in gz_comp gzwrite.c:125
    #5 0x1000b05df in zng_gzclose_w gzwrite.c:511
    #6 0x1000967a4 in zng_gzclose gzlib.c:253
    #7 0x100004f70 in test_gzio example.c:133
    #8 0x100010c5b in main example.c:1034
    #9 0x7fff71f57cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
Gelbpunkt pushed a commit to DavinciCodeOS/zlib-ng that referenced this issue Apr 5, 2022
We always build zlib-ng with support for hardware-accelerated CRC32
because it checks for CRC32 instruction availability at runtime and
changes the active implementation accordingly. However, building for
devices where CRC32 support has not been declared (i.e. armv8-a or
generic architecture and CPU targets respectively) fails with the
following error:

fatal error: error in backend: Cannot select: intrinsic %llvm.aarch64.crc32w
PLEASE submit a bug report to https://github.com/android-ndk/ndk/issues and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: prebuilts/clang/host/linux-x86/clang-r416183/bin/clang.real -c -Wno-enum-compare -Wno-enum-compare-switch -Wno-null-pointer-arithmetic -Wno-null-dereference -Wno-pointer-compare -Wno-xor-used-as-pow -Wno-final-dtor-non-final-class -Wno-psabi -Werror=implicit-function-declaration -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -no-canonical-prefixes -DNDEBUG -UDEBUG -fno-exceptions -Wno-multichar -O2 -g -fno-strict-aliasing -Werror=date-time -Werror=pragma-pack -Werror=pragma-pack-suspicious-include -fdebug-prefix-map=/proc/self/cwd= -D__compiler_offsetof=__builtin_offsetof -faddrsig -fcommon -Werror=int-conversion -fexperimental-new-pass-manager -Wno-reserved-id-macro -Wno-unused-command-line-argument -fcolor-diagnostics -Wno-sign-compare -Wno-defaulted-function-deleted -Wno-inconsistent-missing-override -Wno-c99-designator -Wno-gnu-folding-constant -Wno-compound-token-split-by-macro -ftrivial-auto-var-init=zero -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang -ffunction-sections -fdata-sections -fno-short-enums -funwind-tables -fstack-protector-strong -Wa,--noexecstack -D_FORTIFY_SOURCE=2 -Wstrict-aliasing=2 -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=format-security -nostdlibinc -march=armv8-a -target aarch64-linux-android10000 -Bprebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/aarch64-linux-android/bin -fPIC -Iexternal/zlib-ng -Iexternal/zlib-ng -D__LIBC_API__=10000 -D__LIBM_API__=10000 -D__LIBDL_API__=10000 -Ibionic/libc/include -DHAVE_BUILTIN_CTZ -DHAVE_BUILTIN_CTZLL -DHAVE_VISIBILITY_HIDDEN -DHAVE_VISIBILITY_INTERNAL -DZLIB_CONST -DWITH_GZFILEOP -DZLIB_COMPAT -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -D__USE_LARGEFILE64 -Wall -Werror -Wno-implicit-fallthrough -O3 -DNDEBUG -fno-semantic-interposition -std=c99 -DUNALIGNED_OK -DARM_ACLE_CRC_HASH -DARM_FEATURES -DARM_NEON_ADLER32 -DARM_NEON_CHUNKSET -DARM_NEON_SLIDEHASH -DUNALIGNED64_OK -std=gnu99 -Isystem/core/include -Isystem/media/audio/include -Ihardware/libhardware/include -Ihardware/libhardware_legacy/include -Ihardware/ril/include -Iframeworks/native/include -Iframeworks/native/opengl/include -Iframeworks/av/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/uapi/asm-arm64 -isystem bionic/libc/kernel/android/scsi -isystem bionic/libc/kernel/android/uapi -Ilibnativehelper/include_jni -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Wno-void-pointer-to-enum-cast -Wno-void-pointer-to-int-cast -Wno-pointer-to-int-cast -Werror=fortify-source -Werror=address-of-temporary -Werror=return-type -Wno-tautological-constant-compare -Wno-tautological-type-limit-compare -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-int-in-bool-context -Wno-sizeof-array-div -Wno-tautological-overlap-compare -Wno-deprecated-copy -Wno-range-loop-construct -Wno-misleading-indentation -Wno-zero-as-null-pointer-constant -Wno-deprecated-anon-enum-enum-conversion -Wno-deprecated-enum-enum-conversion -Wno-string-compare -Wno-enum-enum-conversion -Wno-enum-float-conversion -Wno-pessimizing-move -Wno-non-c-typedef-for-linkage -Wno-string-concatenation -MD -MF out/soong/.intermediates/external/zlib-ng/libz/android_arm64_armv8-a_static/obj/external/zlib-ng/arch/arm/insert_string_acle.o.d -o out/soong/.intermediates/external/zlib-ng/libz/android_arm64_armv8-a_static/obj/external/zlib-ng/arch/arm/insert_string_acle.o external/zlib-ng/arch/arm/insert_string_acle.c
1.	<eof> parser at end of file
2.	Code generation
3.	Running pass 'Function Pass Manager' on module 'external/zlib-ng/arch/arm/insert_string_acle.c'.
4.	Running pass 'AArch64 Instruction Selection' on function '@quick_insert_string_acle'
 #0 0x0000000003fb9628 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (prebuilts/clang/host/linux-x86/clang-r416183/bin/clang.real+0x3fb9628)
 zlib-ng#1 0x0000000003fb94b0 llvm::sys::RunSignalHandlers() (prebuilts/clang/host/linux-x86/clang-r416183/bin/clang.real+0x3fb94b0)
 zlib-ng#2 0x0000000003f86193 (prebuilts/clang/host/linux-x86/clang-r416183/bin/clang.real+0x3f86193)
 zlib-ng#3 0x0000000003f8614b (prebuilts/clang/host/linux-x86/clang-r416183/bin/clang.real+0x3f8614b)
 zlib-ng#4 0x00000000038a8147 llvm::sys::Process::Exit(int, bool) (prebuilts/clang/host/linux-x86/clang-r416183/bin/clang.real+0x38a8147)
 zlib-ng#5 0x00000000038a80f0 (prebuilts/clang/host/linux-x86/clang-r416183/bin/clang.real+0x38a80f0)
 zlib-ng#6 0x0000000002489052 llvm::report_fatal_error(llvm::Twine const&, bool) (prebuilts/clang/host/linux-x86/clang-r416183/bin/clang.real+0x2489052)
 zlib-ng#7 0x00000000032d7948 (prebuilts/clang/host/linux-x86/clang-r416183/bin/clang.real+0x32d7948)
 zlib-ng#8 0x000000000247d157 llvm::SelectionDAGISel::CannotYetSelect(llvm::SDNode*) (prebuilts/clang/host/linux-x86/clang-r416183/bin/clang.real+0x247d157)
 zlib-ng#9 0x000000000245d006 llvm::SelectionDAGISel::SelectCodeCommon(llvm::SDNode*, unsigned char const*, unsigned int) (prebuilts/clang/host/linux-x86/clang-r416183/bin/clang.real+0x245d006)
clang: error: clang frontend command failed with exit code 70 (use -v to see invocation)

This is caused by forcing Clang to emit CRC32 instructions for a target
CPU that doesn't support them. Fix the issue by using
TARGET_ARMV8_WITH_CRC correctly in insert_string_acle.c, similar to how
it's used in crc32_acle.c.

Fixes: android/ndk#1568
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants