Skip to content

oss-fuzz: OpenMP-aware fuzzing#640

Merged
LebedevRI merged 13 commits into
darktable-org:developfrom
LebedevRI:oss-fuzz-bundled-omp
Feb 3, 2024
Merged

oss-fuzz: OpenMP-aware fuzzing#640
LebedevRI merged 13 commits into
darktable-org:developfrom
LebedevRI:oss-fuzz-bundled-omp

Conversation

@LebedevRI
Copy link
Copy Markdown
Member

@LebedevRI LebedevRI commented Feb 2, 2024

No description provided.

@LebedevRI LebedevRI force-pushed the oss-fuzz-bundled-omp branch 8 times, most recently from 95aefe6 to 13d0c04 Compare February 2, 2024 22:42
@LebedevRI
oss-fuzz: do build with OpenMP
9488ed7 
OpenMP structured blocks can't throw,
and that invariant is generally not reflected in the code
(unless done manually) when compiled without OpenMP,
and that hides bugs (787695f e.g.)

`rawspeed_get_number_of_processor_cores()` returns `1`
for fuzzers, so there should not be any actual multi-threading done.
@LebedevRI
oss-fuzz: just build our own libomp
28f6ea8
@LebedevRI
LLVMOpenMP.cmake: disable libomptarget
222d11a
@LebedevRI
Actually allow using bundled libomp when no system libomp exists
9db5982 
`find_package(OpenMP)` doesn't work then.
@LebedevRI
CMake: rawspeed_parsers (transitively) needs OpenMP
7c69ce6
@LebedevRI
Try to update bundled libomp to 17
96d8515
@LebedevRI
LLVMOpenMP.cmake: disable all sanitizers
3284e52 
ASan, UBSan and MSan are all complaining.
@LebedevRI
oss-fuzz: don't enable omp for MSan builds
55cd984
@LebedevRI
VC5Decompressor::createWaveletBandDecodingTasks(): iterate over rever…
0f1fec8 
…se bands
@LebedevRI
VC5Decompressor: avoid OpenMP-specific heap-use-after-free
bd60554 
This is rather convoluted.

```
=================================================================
==360017==ERROR: AddressSanitizer: heap-use-after-free on address 0x5110000020c8 at pc 0x556f25741bed bp 0x7feae20da010 sp 0x7feae20da008
READ of size 8 at 0x5110000020c8 thread T27
    #0 0x556f25741bec in .omp_outlined..49 /home/lebedevri/rawspeed/src/librawspeed/decompressors/VC5Decompressor.cpp:349:7
    #1 0x556f25741bec in .omp_task_entry..50 /home/lebedevri/rawspeed/src/librawspeed/decompressors/VC5Decompressor.cpp:344:1
    #2 0x7feb04f37452  (/lib/x86_64-linux-gnu/libomp.so.5+0x68452) (BuildId: 49f63176f1578200567d2439fd1ca076a4a2c2b3)
    #3 0x7feb04f3b704  (/lib/x86_64-linux-gnu/libomp.so.5+0x6c704) (BuildId: 49f63176f1578200567d2439fd1ca076a4a2c2b3)
    #4 0x7feb04f4d174  (/lib/x86_64-linux-gnu/libomp.so.5+0x7e174) (BuildId: 49f63176f1578200567d2439fd1ca076a4a2c2b3)
    #5 0x7feb04f4775e  (/lib/x86_64-linux-gnu/libomp.so.5+0x7875e) (BuildId: 49f63176f1578200567d2439fd1ca076a4a2c2b3)
    darktable-org#6 0x7feb04f45adf  (/lib/x86_64-linux-gnu/libomp.so.5+0x76adf) (BuildId: 49f63176f1578200567d2439fd1ca076a4a2c2b3)
    darktable-org#7 0x7feb04efcd3f in __kmpc_barrier (/lib/x86_64-linux-gnu/libomp.so.5+0x2dd3f) (BuildId: 49f63176f1578200567d2439fd1ca076a4a2c2b3)
    darktable-org#8 0x556f25754f5b in rawspeed::VC5Decompressor::decodeThread(bool&) const /home/lebedevri/rawspeed/src/librawspeed/decompressors/VC5Decompressor.cpp:838:19
    darktable-org#9 0x7feb04faf002 in __kmp_invoke_microtask (/lib/x86_64-linux-gnu/libomp.so.5+0xe0002) (BuildId: 49f63176f1578200567d2439fd1ca076a4a2c2b3)
    darktable-org#10 0x7feb04f183c8  (/lib/x86_64-linux-gnu/libomp.so.5+0x493c8) (BuildId: 49f63176f1578200567d2439fd1ca076a4a2c2b3)
    darktable-org#11 0x7feb04f16955  (/lib/x86_64-linux-gnu/libomp.so.5+0x47955) (BuildId: 49f63176f1578200567d2439fd1ca076a4a2c2b3)
    darktable-org#12 0x7feb04f88137  (/lib/x86_64-linux-gnu/libomp.so.5+0xb9137) (BuildId: 49f63176f1578200567d2439fd1ca076a4a2c2b3)
    darktable-org#13 0x556f256e281e in asan_thread_start(void*) crtfastmath.c
    darktable-org#14 0x7feb04c6945b in start_thread nptl/pthread_create.c:444:8
    darktable-org#15 0x7feb04ce9bbb in clone3 misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

0x5110000020c8 is located 8 bytes inside of 240-byte region [0x5110000020c0,0x5110000021b0)
freed by thread T16 here:
    #0 0x556f25718a61 in operator delete(void*) (/home/lebedevri/rawspeed/build-Clang17-FUZZ/fuzz/librawspeed/decompressors/VC5DecompressorFuzzer+0x1b8a61) (BuildId: dd91d53719a432f2)
    #1 0x556f25741b3e in std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>::operator()(rawspeed::VC5Decompressor::Wavelet::AbstractBand*) const /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/unique_ptr.h:93:2
    #2 0x556f25741b3e in std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>::~unique_ptr() /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/unique_ptr.h:398:4
    #3 0x556f25741b3e in void std::destroy_at<std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>>(std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>*) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_construct.h:88:15
    #4 0x556f25741b3e in void std::_Destroy<std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>>(std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>*) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_construct.h:149:7
    #5 0x556f25741b3e in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>*>(std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>*, std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>*) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_construct.h:163:6
    darktable-org#6 0x556f25741b3e in void std::_Destroy<std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>*>(std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>*, std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>*) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_construct.h:195:7
    darktable-org#7 0x556f25741b3e in void std::_Destroy<std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>*, std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>>(std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>*, std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>*, std::allocator<std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>>&) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/alloc_traits.h:941:7
    darktable-org#8 0x556f25741b3e in std::vector<std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>, std::allocator<std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>>>::_M_erase_at_end(std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>*) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_vector.h:1944:6
    darktable-org#9 0x556f25741b3e in std::vector<std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>, std::allocator<std::unique_ptr<rawspeed::VC5Decompressor::Wavelet::AbstractBand, std::default_delete<rawspeed::VC5Decompressor::Wavelet::AbstractBand>>>>::clear() /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_vector.h:1605:9
    darktable-org#10 0x556f25741b3e in .omp_outlined..49 /home/lebedevri/rawspeed/src/librawspeed/decompressors/VC5Decompressor.cpp:349:21
    darktable-org#11 0x556f25741b3e in .omp_task_entry..50 /home/lebedevri/rawspeed/src/librawspeed/decompressors/VC5Decompressor.cpp:344:1
    darktable-org#12 0x7feb04f37452  (/lib/x86_64-linux-gnu/libomp.so.5+0x68452) (BuildId: 49f63176f1578200567d2439fd1ca076a4a2c2b3)
    darktable-org#13 0x7feb04f3b704  (/lib/x86_64-linux-gnu/libomp.so.5+0x6c704) (BuildId: 49f63176f1578200567d2439fd1ca076a4a2c2b3)

previously allocated by thread T0 here:
    #0 0x556f257181e1 in operator new(unsigned long) (/home/lebedevri/rawspeed/build-Clang17-FUZZ/fuzz/librawspeed/decompressors/VC5DecompressorFuzzer+0x1b81e1) (BuildId: dd91d53719a432f2)
    #1 0x556f25750462 in std::__detail::_MakeUniq<rawspeed::VC5Decompressor::Wavelet::ReconstructableBand>::__single_object std::make_unique<rawspeed::VC5Decompressor::Wavelet::ReconstructableBand, rawspeed::VC5Decompressor::Wavelet&, bool&, bool&>(rawspeed::VC5Decompressor::Wavelet&, bool&, bool&) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/unique_ptr.h:1076:30
    #2 0x556f25750462 in rawspeed::VC5Decompressor::parseLargeCodeblock(rawspeed::ByteStream) /home/lebedevri/rawspeed/src/librawspeed/decompressors/VC5Decompressor.cpp:810:28
    #3 0x556f25745535 in rawspeed::VC5Decompressor::parseVC5() /home/lebedevri/rawspeed/src/librawspeed/decompressors/VC5Decompressor.cpp:588:9
    #4 0x556f2574348c in rawspeed::VC5Decompressor::VC5Decompressor(rawspeed::ByteStream, rawspeed::RawImage const&) /home/lebedevri/rawspeed/src/librawspeed/decompressors/VC5Decompressor.cpp:431:3
    #5 0x556f2571af96 in LLVMFuzzerTestOneInput /home/lebedevri/rawspeed/fuzz/librawspeed/decompressors/VC5Decompressor.cpp:55:31
    darktable-org#6 0x556f2562fe84 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) crtfastmath.c
    darktable-org#7 0x556f25630849 in fuzzer::Fuzzer::TryDetectingAMemoryLeak(unsigned char const*, unsigned long, bool) crtfastmath.c
    darktable-org#8 0x556f2561927e in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) crtfastmath.c
    darktable-org#9 0x556f2561edd6 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) crtfastmath.c
    darktable-org#10 0x556f25648596 in main (/home/lebedevri/rawspeed/build-Clang17-FUZZ/fuzz/librawspeed/decompressors/VC5DecompressorFuzzer+0xe8596) (BuildId: dd91d53719a432f2)
    darktable-org#11 0x7feb04c086c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

Thread T27 created by T0 here:
    #0 0x556f256ca4c1 in pthread_create (/home/lebedevri/rawspeed/build-Clang17-FUZZ/fuzz/librawspeed/decompressors/VC5DecompressorFuzzer+0x16a4c1) (BuildId: dd91d53719a432f2)
    #1 0x7feb04f87737  (/lib/x86_64-linux-gnu/libomp.so.5+0xb8737) (BuildId: 49f63176f1578200567d2439fd1ca076a4a2c2b3)

Thread T16 created by T0 here:
    #0 0x556f256ca4c1 in pthread_create (/home/lebedevri/rawspeed/build-Clang17-FUZZ/fuzz/librawspeed/decompressors/VC5DecompressorFuzzer+0x16a4c1) (BuildId: dd91d53719a432f2)
    #1 0x7feb04f87737  (/lib/x86_64-linux-gnu/libomp.so.5+0xb8737) (BuildId: 49f63176f1578200567d2439fd1ca076a4a2c2b3)

SUMMARY: AddressSanitizer: heap-use-after-free /home/lebedevri/rawspeed/src/librawspeed/decompressors/VC5Decompressor.cpp:349:7 in .omp_outlined..49
Shadow bytes around the buggy address:
  0x511000001e00: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x511000001e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x511000001f00: 00 00 00 00 00 00 fa fa fa fa fa fa fa fa fa fa
  0x511000001f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x511000002000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa fa
=>0x511000002080: fa fa fa fa fa fa fa fa fd[fd]fd fd fd fd fd fd
  0x511000002100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x511000002180: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
  0x511000002200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x511000002280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa fa
  0x511000002300: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==360017==ABORTING

```
@LebedevRI LebedevRI force-pushed the oss-fuzz-bundled-omp branch from 1da1010 to 826e067 Compare February 3, 2024 01:00
@LebedevRI LebedevRI changed the title oss-fuzz: just build our own libomp oss-fuzz: OpenMP-aware fuzzing Feb 3, 2024
@LebedevRI LebedevRI force-pushed the oss-fuzz-bundled-omp branch from 826e067 to f852acd Compare February 3, 2024 01:53
@codecov
Copy link
Copy Markdown

codecov Bot commented Feb 3, 2024

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (dd50b78) 60.78% compared to head (f852acd) 60.77%.

Files Patch % Lines
src/librawspeed/decompressors/VC5Decompressor.cpp 33.33% 3 Missing and 1 partial ⚠️
src/librawspeed/common/RawImage.cpp 50.00% 1 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff             @@
##           develop     #640      +/-   ##
===========================================
- Coverage    60.78%   60.77%   -0.02%     
===========================================
  Files          266      266              
  Lines        15949    15952       +3     
  Branches      2047     2051       +4     
===========================================
  Hits          9695     9695              
- Misses        6125     6127       +2     
- Partials       129      130       +1
Flag Coverage Δ
benchmarks 10.53% <0.00%> (-0.01%) ⬇️
integration 46.01% <50.00%> (-0.01%) ⬇️
linux 57.25% <50.00%> (-0.01%) ⬇️
macOS 24.28% <0.00%> (-0.02%) ⬇️
rpu_u 46.01% <50.00%> (-0.01%) ⬇️
unittests 21.43% <0.00%> (-0.01%) ⬇️
windows ∅ <ø> (∅)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@LebedevRI LebedevRI force-pushed the oss-fuzz-bundled-omp branch from f852acd to 6e31dbf Compare February 3, 2024 02:27
@LebedevRI
Copy link
Copy Markdown
Member Author

LebedevRI commented Feb 3, 2024

Alright, that is good enough for now.

@LebedevRI LebedevRI merged commit 1487e4d into darktable-org:develop Feb 3, 2024
@LebedevRI LebedevRI deleted the oss-fuzz-bundled-omp branch February 3, 2024 03:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@LebedevRI