Fuzz targets seem to have started failing under MSan on CIFuzz #11886

evverx · 2024-05-02T12:56:30Z

It should probably reach OSS-Fuzz a bit later.

/github/workspace/build-out/fuzz-journald-native -timeout=25 -rss_limit_mb=2560 -len_control=0 -seed=1337 -artifact_prefix=/tmp/tmprl9fex94/ -max_total_time=25 -print_final_stats=1 /github/workspace/cifuzz-corpus/fuzz-journald-native >fuzz-0.log 2>&1
================== Job 1 exited with exit code 0 ============
Uninitialized bytes in fputs at offset 22 inside [0x71e000000000, 2774)
==37==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x564d5b17a1cf in Puts /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerIO.cpp:155:3
    #1 0x564d5b17a1cf in fuzzer::CopyFileToErr(std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>> const&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerIO.cpp:68:3
    #2 0x564d5b16fbaf in fuzzer::WorkerThread(fuzzer::Command const&, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:249:5
    #3 0x564d5b16ff31 in __invoke<void (*)(const fuzzer::Command &, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__type_traits/invoke.h:340:25
    #4 0x564d5b16ff31 in __thread_execute<std::__Fuzzer::unique_ptr<std::__Fuzzer::__thread_struct, std::__Fuzzer::default_delete<std::__Fuzzer::__thread_struct> >, void (*)(const fuzzer::Command &, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *, 2UL, 3UL, 4UL, 5UL> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__thread/thread.h:221:5
    #5 0x564d5b16ff31 in void* std::__Fuzzer::__thread_proxy[abi:v180000]<std::__Fuzzer::tuple<std::__Fuzzer::unique_ptr<std::__Fuzzer::__thread_struct, std::__Fuzzer::default_delete<std::__Fuzzer::__thread_struct>>, void (*)(fuzzer::Command const&, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*>>(void*) /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__thread/thread.h:232:5
    #6 0x7ff843509608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8608) (BuildId: c6d0d79d906d62bb768421fc6dada0d5e729f177)
    #7 0x7ff8425de352 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f352) (BuildId: 87b331c034a6458c64ce09c03939e947212e18ce)

DEDUP_TOKEN: Puts--fuzzer::CopyFileToErr(std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>> const&)--fuzzer::WorkerThread(fuzzer::Command const&, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*)
  Uninitialized value was stored to memory at
    #0 0x564d5b0a2dd2 in __msan_memmove /src/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1752:3
    #1 0x564d5b1acffd in std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char) cxa_noexception.cpp

DEDUP_TOKEN: __msan_memmove--std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char)
  Uninitialized value was stored to memory at
    #0 0x564d5b0a2dd2 in __msan_memmove /src/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1752:3
    #1 0x564d5b1acffd in std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char) cxa_noexception.cpp

DEDUP_TOKEN: __msan_memmove--std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char)
  Uninitialized value was stored to memory at
    #0 0x564d5b0a2dd2 in __msan_memmove /src/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1752:3
    #1 0x564d5b1acffd in std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char) cxa_noexception.cpp

DEDUP_TOKEN: __msan_memmove--std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char)
  Uninitialized value was stored to memory at
    #0 0x564d5b0a2dd2 in __msan_memmove /src/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1752:3
    #1 0x564d5b1acffd in std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char) cxa_noexception.cpp

DEDUP_TOKEN: __msan_memmove--std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char)
  Uninitialized value was stored to memory at
    #0 0x564d5b0a2dd2 in __msan_memmove /src/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1752:3
    #1 0x564d5b1acffd in std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char) cxa_noexception.cpp

DEDUP_TOKEN: __msan_memmove--std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char)
  Uninitialized value was stored to memory at
    #0 0x564d5b0a2dd2 in __msan_memmove /src/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1752:3
    #1 0x564d5b1acffd in std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char) cxa_noexception.cpp

DEDUP_TOKEN: __msan_memmove--std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char)
  Uninitialized value was created by a heap allocation
    #0 0x564d5b0abd72 in malloc /src/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1007:3
    #1 0x564d5b19c6e6 in operator new(unsigned long) cxa_noexception.cpp
    #2 0x564d5b17a1b3 in fuzzer::CopyFileToErr(std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>> const&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerIO.cpp:68:8
    #3 0x564d5b16fbaf in fuzzer::WorkerThread(fuzzer::Command const&, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:249:5
    #4 0x564d5b16ff31 in __invoke<void (*)(const fuzzer::Command &, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__type_traits/invoke.h:340:25
    #5 0x564d5b16ff31 in __thread_execute<std::__Fuzzer::unique_ptr<std::__Fuzzer::__thread_struct, std::__Fuzzer::default_delete<std::__Fuzzer::__thread_struct> >, void (*)(const fuzzer::Command &, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *, 2UL, 3UL, 4UL, 5UL> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__thread/thread.h:221:5
    #6 0x564d5b16ff31 in void* std::__Fuzzer::__thread_proxy[abi:v180000]<std::__Fuzzer::tuple<std::__Fuzzer::unique_ptr<std::__Fuzzer::__thread_struct, std::__Fuzzer::default_delete<std::__Fuzzer::__thread_struct>>, void (*)(fuzzer::Command const&, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*>>(void*) /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__thread/thread.h:232:5
    #7 0x7ff843509608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8608) (BuildId: c6d0d79d906d62bb768421fc6dada0d5e729f177)

DEDUP_TOKEN: __interceptor_malloc--operator new(unsigned long)--fuzzer::CopyFileToErr(std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>> const&)
SUMMARY: MemorySanitizer: use-of-uninitialized-value /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerIO.cpp:155:3 in Puts
Exiting

The text was updated successfully, but these errors were encountered:

maflcko · 2024-05-02T13:00:45Z

This should be fixed by #11887

evverx · 2024-05-02T13:14:27Z

Looking at #11714 it seems it was a bumpy ride :-) It was prudent to turn off the "function" sanitizer there too: systemd/systemd#29972, avahi/avahi#584

maflcko · 2024-05-02T13:22:41Z

Yes, the bump was more involved, because it skipped over several clang releases. Be aware that the function sanitizer may in the future be enabled. See #11778 for the tracking issue for that.

Fixes #11886 --------- Co-authored-by: MarcoFalke <6399679+MarcoFalke@users.noreply.github.com> Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com>

evverx · 2024-05-06T17:49:56Z

To judge from https://github.com/systemd/systemd/actions/runs/8972741048/job/24641404945 the fuzz targets keep failing on CIFuzz. Could it be that the base images haven't been updated?

jonathanmetzman · 2024-05-06T19:47:36Z

Are we sure this is a CIFuzz issue? I think CIFuzz says this happens with the oss-fuzz builds too. So it seems to be an issue with oss-fuzz. Do you agree?

evverx · 2024-05-06T19:50:34Z

I'm not sure. I haven't seen those backtraces there. They don't seem to have been reported on Monorail either. Let me double-check.

evverx · 2024-05-06T20:44:30Z

Looks like there are no those backtraces on OSS-Fuzz. I can't reproduce it locally with helper.py with the latest images either so it seems CIFuzz is the only place where they keep popping up for some reason.

jonathanmetzman · 2024-05-06T20:47:18Z

The cifuzz images are building: https://pantheon.corp.google.com/gcr/images/oss-fuzz-base/GLOBAL/cifuzz-base

maflcko · 2024-05-07T06:03:05Z

From https://github.com/systemd/systemd/actions/runs/8972741048/job/24641404945#step:4:41 it looks recent enough already (from yesterday). As I bumped the cmake version in commit f5e1a6d at the same time, this can be checked via:

$ podman image inspect 'gcr.io/oss-fuzz-base/base-builder@sha256:03b27bdf148b65659da08c4ee221d1c970b2f5e3f9eb7cb943a69e9ec5003a8c' | grep 'ENV CMAKE_VERSION'
                    "created_by": "/bin/sh -c #(nop)  ENV CMAKE_VERSION=3.29.2",

Or is there another component in CIFuzz that is not yet updated?

jonathanmetzman · 2024-05-07T13:39:38Z

Yeah something odd happening here. CIFuzz should always be using the latest builder. The latest runner, although it should be used, is a more complicated story.

evverx · 2024-05-07T16:10:12Z

Looking at the backtrace it could be that it has something to do with the runtime environment in the sense that

==37==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x564d5b17a1cf in Puts /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerIO.cpp:155:3
    #1 0x564d5b17a1cf in fuzzer::CopyFileToErr(std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>> const&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerIO.cpp:68:3
    #2 0x564d5b16fbaf in fuzzer::WorkerThread(fuzzer::Command const&, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:249:5
    #3 0x564d5b16ff31 in __invoke<void (*)(const fuzzer::Command &, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__type_traits/invoke.h:340:25
    #4 0x564d5b16ff31 in __thread_execute<std::__Fuzzer::unique_ptr<std::__Fuzzer::__thread_struct, std::__Fuzzer::default_delete<std::__Fuzzer::__thread_struct> >, void (*)(const fuzzer::Command &, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *, 2UL, 3UL, 4UL, 5UL> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__thread/thread.h:221:5
    #5 0x564d5b16ff31 in void* std::__Fuzzer::__thread_proxy[abi:v180000]<std::__Fuzzer::tuple<std::__Fuzzer::unique_ptr<std::__Fuzzer::__thread_struct, std::__Fuzzer::default_delete<std::__Fuzzer::__thread_struct>>, void (*)(fuzzer::Command const&, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*>>(void*) /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__thread/thread.h:232:5
    #6 0x7ff843509608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8608) (BuildId: c6d0d79d906d62bb768421fc6dada0d5e729f177)
    #7 0x7ff8425de352 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f352) (BuildId: 87b331c034a6458c64ce09c03939e947212e18ce)

seems to indicate that some file is copied somewhere and maybe it doesn't happen anywhere else.

jonathanmetzman · 2024-05-07T16:28:22Z

OK I can reproduce this now:

/tmp/fuzz-efi-printf -jobs=2 -timeout=25 -rss_limit_mb=2560 -len_control=0 -seed=1337 -artifact_prefix=/tmp/a -max_total_time=12 -print_final_stats=1

jonathanmetzman · 2024-05-07T16:28:31Z

I guess libFuzzer is no longer MSAN safe.

jonathanmetzman · 2024-05-07T16:59:12Z

I'm going to look at fixing this.

maflcko · 2024-05-07T20:05:54Z

Ref #11922

evverx · 2024-05-08T20:44:35Z

I restarted CFLite in systemd/systemd-stable#396 and I can no longer see those backtraces. Looks like this issue can be closed. Thanks!

This was referenced May 2, 2024

infra: Fix msan build #11880

Closed

infra: Fix msan build #11887

Merged

evverx mentioned this issue May 2, 2024

Check added to PcapNg processing secdev/scapy#4373

Merged

jonathanmetzman closed this as completed in #11887 May 3, 2024

jonathanmetzman added a commit that referenced this issue May 3, 2024

infra: Fix msan build (#11887)

f5e1a6d

Fixes #11886 --------- Co-authored-by: MarcoFalke <6399679+MarcoFalke@users.noreply.github.com> Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com>

nagdahimanshu mentioned this issue May 3, 2024

Add support for staking contract LiskHQ/lisk-app-plugin#12

Merged

11 tasks

evverx mentioned this issue May 3, 2024

build(deps): bump github/super-linter from 5.0.0 to 6 systemd/systemd#32609

Merged

jonathanmetzman reopened this May 6, 2024

jonathanmetzman mentioned this issue May 6, 2024

Squash MSAN false positives #11915

Open

evverx closed this as completed May 8, 2024

davidben mentioned this issue May 10, 2024

Better pre-commit monitoring for MSan false positives #11941

Open

CarlosEduR mentioned this issue May 17, 2024

add memory sanitizer to cifuzz workflow ada-url/ada#667

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fuzz targets seem to have started failing under MSan on CIFuzz #11886

Fuzz targets seem to have started failing under MSan on CIFuzz #11886

evverx commented May 2, 2024

maflcko commented May 2, 2024 •

edited

Loading

evverx commented May 2, 2024

maflcko commented May 2, 2024

evverx commented May 6, 2024 •

edited

Loading

jonathanmetzman commented May 6, 2024

evverx commented May 6, 2024

evverx commented May 6, 2024

jonathanmetzman commented May 6, 2024

maflcko commented May 7, 2024

jonathanmetzman commented May 7, 2024

evverx commented May 7, 2024

jonathanmetzman commented May 7, 2024

jonathanmetzman commented May 7, 2024

jonathanmetzman commented May 7, 2024

maflcko commented May 7, 2024

evverx commented May 8, 2024

Fuzz targets seem to have started failing under MSan on CIFuzz #11886

Fuzz targets seem to have started failing under MSan on CIFuzz #11886

Comments

evverx commented May 2, 2024

maflcko commented May 2, 2024 • edited Loading

evverx commented May 2, 2024

maflcko commented May 2, 2024

evverx commented May 6, 2024 • edited Loading

jonathanmetzman commented May 6, 2024

evverx commented May 6, 2024

evverx commented May 6, 2024

jonathanmetzman commented May 6, 2024

maflcko commented May 7, 2024

jonathanmetzman commented May 7, 2024

evverx commented May 7, 2024

jonathanmetzman commented May 7, 2024

jonathanmetzman commented May 7, 2024

jonathanmetzman commented May 7, 2024

maflcko commented May 7, 2024

evverx commented May 8, 2024

maflcko commented May 2, 2024 •

edited

Loading

evverx commented May 6, 2024 •

edited

Loading