Need instrumented versions for system libs to make MSAN work

[inferno-chromium]

This is needed for several projects that depend on system libraries. E.g see #607

[kcc]

I've looked at the existing warnings from libpng, freetype, libarchive,
and they are clearly caused by linking in non-instrumented system libs like -lz, lzma, or -larchive

So, I think any further msan roll up is blocked until we can provide msan-instrumented libs in the docker image for msan.

I suggest this order of steps:

• add -lz and -larchive
• observe at least png and freetype targets get clean, enable them
• add -lbz2 -llzo2 -lxml2 -llzma -lcrypto -llz4 -licuuc

[oliverchang]

Progress update:

We now have a set of scripts in https://github.com/google/oss-fuzz/tree/master/infra/base-images/msan-builder that attempt to build MSan instrumented libraries by hooking into debian's package build system.

This should hopefully work for most packages in a debian-based distro with some minor overrides.

At at high level this:

• Install a compiler wrapper and sets up various other wrappers and environment variables.

• Figure out package dependencies that are c/c++ libraries (and build those too)

• Use apt-get source to get the source archive for them.

• Do dpkg-buildpackage to do the build and create .deb files

• Extract shared libraries from the .deb files, fixing symlinks and rpaths to be relative.

Our infra will build a set of libraries specified here

Some remaining work:

• Make sure openssl gets --no-asm during configure
• Some basic tests to make sure the .so files are actually msan instrumented
• Enable more libraries to be built by default.
• Include these libraries in MSan project builds.

[oliverchang]

Also have a question for @eugenis :

To keep our build process simple, we're building libraries with -fsanitize-recover=memory. Are there any downsides to this? And can we use a binary compiled without -fsanitize-recover=memory with these libraries?

[eugenis]

Having just one module built with -fsanitize-recover=memory will flip the default setting of halt_on_error in MSAN_OPTIONS to false, which will make all reports in interceptors non-terminating. Even when the interceptor is called from a module that is built w/o fsanitize-recover. You can compensate for that by setting MSAN_OPTIONS=halt_on_error=true.

…

On Thu, Dec 7, 2017 at 10:05 AM, Oliver Chang ***@***.***> wrote: Also have a question for @eugenis <https://github.com/eugenis> : To keep our build process simple, we're building libraries with -fsanitize-recover=memory. Are there any downsides to this? And can we mix a binary compiled *without* -fsanitize-recover=memory with these libraries? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#608 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZuSoEQ8FrbNeWPFzjr2hCMjLomTxSBks5s-CkHgaJpZM4NaotU> .

[oliverchang]

Progress update. MSan builds will now use static libraries available in this list. Builds will also have their rpaths patched to use any instrumented dynamic libraries available too.

2 projects I looked at:

• libarchive. Many false positives closed, but 9 reports remain open. These all seem related to lz4, so either it's not instrumented properly or these are legitimate reports.

• libpng had a libz dependency. https://oss-fuzz.com/v2/testcase-detail/5439055517974528 was closed after it started running with the instrumented libz.

[inferno-chromium]

Done!