LeakSanitizer has encountered a fatal error #723
Comments
|
Looks like a SEGV during leak detection. |
|
@sitsofe Could you run with LSAN_OPTIONS=verbosity=1:log_threads=1:log_pointers=1? Also, do you use x86_64? |
|
@chefmax Yes I'm using x86_64. Here's a run + log_pointers=1 If there's any more information I can provide, please let me know. |
|
does this happen with clang top-of-tree? |
|
FWIW, I've seen a strange segfault some time ago on Aarch64. I didn't spend much time on investigation, but the issue seemed to appear in PointsIntoChunk function (I used LSan w/o ASan): In my case, the chunk value actually pointed into metadata chunk, not to user chunk, thus all subsequent logic became invalid. But please note, that x86_64 and AArch64 use different allocation strategies, thus I'm not sure this is the case. I can attach a draft patch here I'm preparing to post upstream if desired. |
|
Reproducing the issue turned out to be simple in the end - the program had protected its memory: #include <stdlib.h>
#include <sys/mman.h>
int main(void) {
int *protected;
posix_memalign((void **) &protected, 4096, 32 * 1024);
mprotect(protected, 1024, PROT_NONE);
//*protected = 1;
return 0;
}I guess the leak sanitizer should explicitly mark memory as accessible before trying to access it after the program finishes... |
Or just not access protected pages to avoid false negatives. |
|
@yugr So if a memory region is protected against reading in any form you simply assume it doesn't contain pointers to any other memory (since you couldn't read it to check anyway...)? If so perhaps it's the only reasonable thing that can be done (plus a documentation note) - if the program is running and someone asks for a leak check it's probably risky to start fiddling with memory mappings. Further, not fiddling the mapping will just make the program crash. Given this it sounds like a potential false positive is the lesser of the evils... |
|
@sitsofe Yes, that's what I had in mind (just a suggestion though - I'm not a maintainer). |
|
Apparently, this is some kind of user-implemented guard page (aka electric fence), I am not convinced that implementing support for this use case is worth the extra complexity in lsan. |
|
The granularity mistake was introduced by me and wasn't in the original code. I believe the purpose for it stems from a desire to protect some memory that should only be read/written at specific times and later I see it being used with mmap. If you want to close this feel free (it's worth noting that tcmalloc's leak detection hits the same issue so I guess this type of situation is rare). |
|
Let me close this. If more users complain we'll reconsider. |
|
May I offer a simple fix for this? |
|
Is it possible that this issue is related to the memalign, not just the mprotect? We have seen some failures in LSAN related to memaligned memory, and the "patch" here apparently resolves that, but I'm not fully across the issue. |
|
We're also hitting this in what is likely a very similar condition. It'd be nice to get a warning instead of a failure (even if a flag), as right now we have to disable ASAN :( |
|
You could also do ASAN_OPTIONS=detect_leaks=0, i.e. disable leak detection
but not the rest of ASan.
…On Thu, Oct 17, 2019 at 5:11 PM Ben Vanik ***@***.***> wrote:
We're also hitting this in what is likely a very similar condition. It'd
be nice to get a warning instead of a failure (even if a flag), as right
now we have to disable ASAN :(
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#723?email_source=notifications&email_token=AADG4SSY4N4PVPB3XNSMOVLQPD5KLA5CNFSM4CPOZL72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBR6ZVQ#issuecomment-543419606>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADG4STCMXKCCIEJE2PZS5LQPD5KLANCNFSM4CPOZL7Q>
.
|
|
We started seeing this recently, and might have found a culprit. I'm attempting to build a smaller reproduction. We have a library that is dynamically loading an lttng provider, and it seems that unconditionally if we enable this, ASAN results in this crash. Thats a very common use case, so a bit scary that those 2 interfere with each other. |
|
Yup. I can reproduce it with a small test project. Tested on ubuntu 18.04. |
|
@ThadHouse While your part of the example is small it is not self contained because you need the external lttng project to be installed before you can try it. You might wish to contact the LTTng folks and get their take on things first... |
|
ASAN shouldn't be crashing, no matter what the libraries it's being applied to do. In addition, lttng is extremely popular as a logging library, and it's dependency is only required to build. I'll add tomorrow morning exactly what package needs to be installed to get the build to work. |
|
I was able to work around this issue by specifying the following runtime options With this option in effect, LeakSanitizer no longer crashes while reporting leaks in my particular case. |
|
mprotect() issue doesn’t get resolve with the asan option "ASAN_OPTIONS=fast_unwind_on_malloc=0" for my case. Any suggestions on this? |
|
Please go through below issue. It might be useful and if you have similar problem workaround has been given on it. |
When using -fanitize=address in CI, we consistently get spurious test failures due to some issue with the leak detection pass at the end of execution: [==========] tests: Running 1 test(s). [ RUN ] call_rtas_set_debug [ OK ] call_rtas_set_debug [==========] tests: 1 test(s) run. [ PASSED ] 1 test(s). ==2492==LeakSanitizer has encountered a fatal error. ==2492==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1 ==2492==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc) FAIL tests/link_librtas (exit status: 1) Perhaps some interaction with cmocka? It does not happen for me locally using the same build-with command line. Possibly related issue: google/sanitizers#723 Signed-off-by: Nathan Lynch <nathanl@linux.ibm.com>
When using -fanitize=address in CI, we consistently get spurious test failures due to some issue with the leak detection pass at the end of execution: [==========] tests: Running 1 test(s). [ RUN ] call_rtas_set_debug [ OK ] call_rtas_set_debug [==========] tests: 1 test(s) run. [ PASSED ] 1 test(s). ==2492==LeakSanitizer has encountered a fatal error. ==2492==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1 ==2492==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc) FAIL tests/link_librtas (exit status: 1) Perhaps some interaction with cmocka? It does not happen for me locally using the same build-with command line. Possibly related issue: google/sanitizers#723 Signed-off-by: Nathan Lynch <nathanl@linux.ibm.com>
When using `-fanitize=address`, we consistently get spurious build failures on ARM64 due to some issue with the leak detection pass at the end of execution: ``` ==2413==AddressSanitizer: failed to intercept '__isoc99_printf' ==2413==AddressSanitizer: failed to intercept '__isoc99_sprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_snprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_fprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_vprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_vsprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_vsnprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_vfprintf' ==2413==AddressSanitizer: failed to intercept 'xdr_destroy' ==2413==AddressSanitizer: failed to intercept 'crypt' ==2413==AddressSanitizer: failed to intercept 'crypt_r' ==2413==AddressSanitizer: failed to intercept '__cxa_throw' ==2413==AddressSanitizer: failed to intercept '__cxa_rethrow_primary_exception' ==2413==AddressSanitizer: libc interceptors initialized || `[0x002000000000, 0x007fffffffff]` || HighMem || || `[0x001400000000, 0x001fffffffff]` || HighShadow || || `[0x001200000000, 0x0013ffffffff]` || ShadowGap || || `[0x001000000000, 0x0011ffffffff]` || LowShadow || || `[0x000000000000, 0x000fffffffff]` || LowMem || MemToShadow(shadow): 0x001200000000 0x00123fffffff 0x001280000000 0x0013ffffffff redzone=16 max_redzone=2048 quarantine_size_mb=256M thread_local_quarantine_size_kb=1024K ``` Note, it does not happen for me locally and on AWS Graviton 2 using the same build-with command line. Related to google#11798 Related to google/sanitizers#723
When using `-fanitize=address`, we consistently get spurious build failures on ARM64 due to some issue with the leak detection pass at the end of execution: ``` ==2413==AddressSanitizer: failed to intercept '__isoc99_printf' ==2413==AddressSanitizer: failed to intercept '__isoc99_sprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_snprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_fprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_vprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_vsprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_vsnprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_vfprintf' ==2413==AddressSanitizer: failed to intercept 'xdr_destroy' ==2413==AddressSanitizer: failed to intercept 'crypt' ==2413==AddressSanitizer: failed to intercept 'crypt_r' ==2413==AddressSanitizer: failed to intercept '__cxa_throw' ==2413==AddressSanitizer: failed to intercept '__cxa_rethrow_primary_exception' ==2413==AddressSanitizer: libc interceptors initialized || `[0x002000000000, 0x007fffffffff]` || HighMem || || `[0x001400000000, 0x001fffffffff]` || HighShadow || || `[0x001200000000, 0x0013ffffffff]` || ShadowGap || || `[0x001000000000, 0x0011ffffffff]` || LowShadow || || `[0x000000000000, 0x000fffffffff]` || LowMem || MemToShadow(shadow): 0x001200000000 0x00123fffffff 0x001280000000 0x0013ffffffff redzone=16 max_redzone=2048 quarantine_size_mb=256M thread_local_quarantine_size_kb=1024K ``` Note, it does not happen for me locally and on AWS Graviton 2 using the same build-with command line. Related to google#11798 Related to google/sanitizers#723 Related to ligurio/lua-c-api-tests#72
When using `-fanitize=address`, we consistently get spurious build failures on ARM64 due to some issue with the leak detection pass at the end of execution: ``` ==2413==AddressSanitizer: failed to intercept '__isoc99_printf' ==2413==AddressSanitizer: failed to intercept '__isoc99_sprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_snprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_fprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_vprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_vsprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_vsnprintf' ==2413==AddressSanitizer: failed to intercept '__isoc99_vfprintf' ==2413==AddressSanitizer: failed to intercept 'xdr_destroy' ==2413==AddressSanitizer: failed to intercept 'crypt' ==2413==AddressSanitizer: failed to intercept 'crypt_r' ==2413==AddressSanitizer: failed to intercept '__cxa_throw' ==2413==AddressSanitizer: failed to intercept '__cxa_rethrow_primary_exception' ==2413==AddressSanitizer: libc interceptors initialized || `[0x002000000000, 0x007fffffffff]` || HighMem || || `[0x001400000000, 0x001fffffffff]` || HighShadow || || `[0x001200000000, 0x0013ffffffff]` || ShadowGap || || `[0x001000000000, 0x0011ffffffff]` || LowShadow || || `[0x000000000000, 0x000fffffffff]` || LowMem || MemToShadow(shadow): 0x001200000000 0x00123fffffff 0x001280000000 0x0013ffffffff redzone=16 max_redzone=2048 quarantine_size_mb=256M thread_local_quarantine_size_kb=1024K ``` Note, it does not happen for me locally and on AWS Graviton 2 using the same build-with command line. Related to #11798 Related to google/sanitizers#723 Related to ligurio/lua-c-api-tests#72
When statically compiling a closed source piece of software LeakSanitizer regularly encounters a fatal error:
The program is mostly compiled statically but still links against a few pieces:
However if the program takes an early exit no sanitizer warning is thrown:
When running fully the program makes use of shared memory.
The text was updated successfully, but these errors were encountered: