Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AddressSanitizer CHECK failed #1171

Open
jmssil opened this issue Dec 2, 2019 · 17 comments
Open

AddressSanitizer CHECK failed #1171

jmssil opened this issue Dec 2, 2019 · 17 comments

Comments

@jmssil
Copy link

jmssil commented Dec 2, 2019

Hi,

I get this error:

==26081==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179 "((0)) == ((sigaltstack(&altstack, &oldstack)))" (0x0, 0xffffffffffffffff)
    #0 0x7f375645c445 in AsanCheckFailed ../../../../src/libsanitizer/asan/asan_rtl.cc:66
    #1 0x7f3756479145 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) ../../../../src/libsanitizer/sanitizer_common/sanitizer_termination.cc:77
    #2 0x7f3756471be9 in __sanitizer::UnsetAlternateSignalStack() ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179
    #3 0x7f375645fe24 in __asan::AsanThread::Destroy() ../../../../src/libsanitizer/asan/asan_thread.cc:108
    #4 0x7f375482ebd1 in __nptl_deallocate_tsd (/lib64/libpthread.so.0+0x7bd1)
    #5 0x7f375482ede2 in start_thread (/lib64/libpthread.so.0+0x7de2)
    #6 0x7f37538aaeac in __clone (/lib64/libc.so.6+0xfdeac)

but absolutely no clues where to look.

Google seems to know nothing about this specific error.

How can I understand what's wrong and fix it?
@eugenis
Copy link
Contributor

eugenis commented Dec 2, 2019 via email

@jmssil
Copy link
Author

jmssil commented Dec 2, 2019 via email

@jmssil
Copy link
Author

jmssil commented Dec 2, 2019 via email

@eugenis
Copy link
Contributor

eugenis commented Dec 2, 2019 via email

@jmssil
Copy link
Author

jmssil commented Dec 6, 2019

I'm sorry, I starred the e-mail but then completely missed it.

Here is the output with -f:

$ ASAN_OPTIONS=halt_on_error=0 ASAN_OPTIONS=verbosity=1 LD_PRELOAD=/opt/gnat/lib64/libasan.so strace -f ./FCS/SAR_Logger/pc_sar_data_logger.eab|& grep sigaltstack
sigaltstack(NULL, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0
sigaltstack({ss_sp=0x7fad63b22000, ss_flags=0, ss_size=32768}, NULL) = 0
sigaltstack(NULL, {ss_sp=0x7fad63b22000, ss_flags=0, ss_size=32768}) = 0
sigaltstack({ss_sp=0x7fad615c8f60, ss_flags=0, ss_size=16384}, NULL) = 0
sigaltstack({ss_sp=0x7fad615c8f60, ss_flags=0, ss_size=16384}, NULL) = 0
[pid 32175] sigaltstack(NULL, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0
[pid 32175] sigaltstack({ss_sp=0x7fad63a3c000, ss_flags=0, ss_size=32768}, NULL) = 0
[pid 32175] sigaltstack({ss_sp=0x7fad5c5fae60, ss_flags=0, ss_size=16384}, NULL) = 0
[pid 32176] sigaltstack(NULL, <unfinished ...>
[pid 32176] <... sigaltstack resumed> {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0
[pid 32176] sigaltstack({ss_sp=0x7fad63a25000, ss_flags=0, ss_size=32768}, <unfinished ...>
[pid 32176] <... sigaltstack resumed> NULL) = 0
[pid 32176] sigaltstack({ss_sp=0x7fad5c4f5e60, ss_flags=0, ss_size=16384}, NULL) = 0
^C

with -f it does not end, I had to kill it.

Without the grep, we can see it gets "stuck" in this:

(...)
[pid 32524] futex(0x6200000011cc, FUTEX_WAIT_BITSET_PRIVATE, 401, {3078306, 654442437}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 32524] futex(0x6200000011f8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 32524] sched_yield() = 0
[pid 32524] futex(0x6200000011cc, FUTEX_WAIT_BITSET_PRIVATE, 403, {3078306, 754763901}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 32524] futex(0x6200000011f8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 32524] sched_yield() = 0
[pid 32524] futex(0x6200000011cc, FUTEX_WAIT_BITSET_PRIVATE, 405, {3078306, 855042858}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 32524] futex(0x6200000011f8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 32524] sched_yield() = 0
[pid 32524] futex(0x6200000011cc, FUTEX_WAIT_BITSET_PRIVATE, 407, {3078306, 955362834}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 32524] futex(0x6200000011f8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 32524] sched_yield() = 0
[pid 32524] futex(0x6200000011cc, FUTEX_WAIT_BITSET_PRIVATE, 409, {3078307, 55636245}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 32524] futex(0x6200000011f8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 32524] sched_yield() = 0
[pid 32524] futex(0x6200000011cc, FUTEX_WAIT_BITSET_PRIVATE, 411, {3078307, 155922526}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 32524] futex(0x6200000011f8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 32524] sched_yield() = 0
[pid 32524] futex(0x6200000011cc, FUTEX_WAIT_BITSET_PRIVATE, 413, {3078307, 256291074}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 32524] futex(0x6200000011f8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 32524] sched_yield() = 0
[pid 32524] futex(0x6200000011cc, FUTEX_WAIT_BITSET_PRIVATE, 415, {3078307, 356546839}, ffffffff^Cstrace: Process 32523 detached
strace: Process 32524 detached
<detached ...>
strace: Process 32525 detached

I don't think our application sets its own sigaltstack (I've never heard about this) and I ran a grep and this word "sigaltstack" only occurs in the executables.

@jmssil
Copy link
Author

jmssil commented Dec 6, 2019

It seems to be working now after recompiling. Maybe something outdated, maybe a glitch.

@jmssil
Copy link
Author

jmssil commented Dec 10, 2019

It seems to have happened again with a colleague:

==20==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179 "((0)) == ((sigaltstack(&altstack, &oldstack)))" (0x0, 0xffffffffffffffff)
#0 0x7f1502cba445 in AsanCheckFailed ../../../../src/libsanitizer/asan/asan_rtl.cc:66
#1 0x7f1502cd7145 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) ../../../../src/libsanitizer/sanitizer_common/sanitizer_termination.cc:77
#2 0x7f1502ccfbe9 in __sanitizer::UnsetAlternateSignalStack() ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179
#3 0x7f1502cbde24 in __asan::AsanThread::Destroy() ../../../../src/libsanitizer/asan/asan_thread.cc:108
#4 0x7f150108bc61 in __nptl_deallocate_tsd (/lib64/libpthread.so.0+0x7c61)
#5 0x7f150108be72 in start_thread (/lib64/libpthread.so.0+0x7e72)
#6 0x7f150010788c in clone (/lib64/libc.so.6+0xfe88c)

And still related to sigaltstack.

What would cause this undeterminism since we don't mess with sigaltstack (don't even know what it is)?

@jmssil
Copy link
Author

jmssil commented Dec 10, 2019

One more example:

==28227==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179 "((0)) == ((sigaltstack(&altstack, &oldstack)))" (0x0, 0xffffffffffffffff)
==28227==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179 "((0)) == ((sigaltstack(&altstack, &oldstack)))" (0x0, 0xffffffffffffffff)
==28227==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179 "((0)) == ((sigaltstack(&altstack, &oldstack)))" (0x0, 0xffffffffffffffff)
==28227==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179 "((0)) == ((sigaltstack(&altstack, &oldstack)))" (0x0, 0xffffffffffffffff)
==28227==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179 "((0)) == ((sigaltstack(&altstack, &oldstack)))" (0x0, 0xffffffffffffffff)
==28227==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179 "((0)) == ((sigaltstack(&altstack, &oldstack)))" (0x0, 0xffffffffffffffff)
==28227==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179 "((0)) == ((sigaltstack(&altstack, &oldstack)))" (0x0, 0xffffffffffffffff)
==28227==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179 "((0)) == ((sigaltstack(&altstack, &oldstack)))" (0x0, 0xffffffffffffffff)
==28227==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179 "((0)) == ((sigaltstack(&altstack, &oldstack)))" (0x0, 0xffffffffffffffff)
==28227==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179 "((0)) == ((sigaltstack(&altstack, &oldstack)))" (0x0, 0xffffffffffffffff)
==28227==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179 "((0)) == ((sigaltstack(&altstack, &oldstack)))" (0x0, 0xffffffffffffffff)
#0 0x7f9a038d8445 in AsanCheckFailed ../../../../src/libsanitizer/asan/asan_rtl.cc:66
#1 0x7f9a038f5145 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) ../../../../src/libsanitizer/sanitizer_common/sanitizer_termination.cc:77
#2 0x7f9a038edbe9 in __sanitizer::UnsetAlternateSignalStack() ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:179
#3 0x7f9a038dbe24 in __asan::AsanThread::Destroy() ../../../../src/libsanitizer/asan/asan_thread.cc:108
#4 0x7f9a01caabd1 in __nptl_deallocate_tsd (/lib64/libpthread.so.0+0x7bd1)
#5 0x7f9a01caade2 in start_thread (/lib64/libpthread.so.0+0x7de2)
#6 0x7f9a00d26eac in __clone (/lib64/libc.so.6+0xfdeac)

#0 0x7f9a038d8445 in AsanCheckFailed ../../../../src/libsanitizer/asan/asan_rtl.cc:66
#0 0x7f9a038d8445 in AsanCheckFailed ../../../../src/libsanitizer/asan/asan_rtl.cc:66
#0 0x7f9a038d8445 in AsanCheckFailed ../../../../src/libsanitizer/asan/asan_rtl.cc:66
#0 0x7f9a038d8445 in AsanCheckFailed ../../../../src/libsanitizer/asan/asan_rtl.cc:66

I have absolutely no clue what this means. The program has been working correctly for years, no problems detected. There may be memory errors and leaks, but I don't understand these failed checks.

Is this a bug, a false positive, a benign symptom?

@eugenis
Copy link
Contributor

eugenis commented Dec 13, 2019 via email

@jmssil
Copy link
Author

jmssil commented Dec 13, 2019

What would sigaltstack be used for? I can't understand from a Google search. And why does it relate to ASAn?

I need LD_PRELOAD because:
==10474==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.

So I run with:
ASAN_OPTIONS=halt_on_error=0 LD_PRELOAD=/opt/gnat/lib64/libasan.so ./program.eab

I build with:
"-fsanitize=address", -- Enable AddressSanitizer, a fast memory error detector.
"-fsanitize-recover=address", -- Allows using ASAN_OPTIONS=halt_on_error=0 when running the program.
"-fno-omit-frame-pointer", -- Get nicer stack traces in error messages.
"-fPIE", -- Produce a position-independent executable.
-- This allow the kernel to load the main executable at a different base address.
-- Needed for using AddressSanitizer due to the size of the executable.

Then bind with:
"-shared", -- Link with dynamic libraries.
-- Needed for -fPIE and -pie.

And link with:
"-fsanitize=address", -- Enable AddressSanitizer, a fast memory error detector.
"-fsanitize-recover=address",
"-fno-omit-frame-pointer", -- Get nicer stack traces in error messages.
"-pie", -- Produce a position-independent executable.
-- This allow the kernel to load the main executable at a different base address.
-- Needed for using AddressSanitizer due to the size of the executable.

If I use a newer version from ASan, included in a newer version of GNAT, I get this for one program:
==16300==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:185 "((0)) == ((sigaltstack(&altstack, &oldstack)))" (0x0, 0xffffffffffffffff)
#0 0x7f30bc803c45 in AsanCheckFailed ../../../../src/libsanitizer/asan/asan_rtl.cc:67
#1 0x7f30bc81ee69 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) ../../../../src/libsanitizer/sanitizer_common/sanitizer_termination.cc:77
#2 0x7f30bc816fc9 in __sanitizer::UnsetAlternateSignalStack() ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:185
#3 0x7f30bc80796c in __asan::AsanThread::Destroy() ../../../../src/libsanitizer/asan/asan_thread.cc:103
#4 0x7f30ba9a5bd1 in __nptl_deallocate_tsd (/lib64/libpthread.so.0+0x7bd1)
#5 0x7f30ba9a5de2 in start_thread (/lib64/libpthread.so.0+0x7de2)
#6 0x7f30b9c27eac in __clone (/lib64/libc.so.6+0xfdeac)

We avoid using ASan from LLVM since we use GNAT.

@jmssil
Copy link
Author

jmssil commented Dec 13, 2019

It seems that ASAN_OPTIONS=use_sigaltstack=1 prevents this crash. Does it make sense?

@eugenis
Copy link
Contributor

eugenis commented Dec 16, 2019 via email

@stsp
Copy link

stsp commented Jan 18, 2022

I have hit the same bug.
I posted the details and the reproducer here:
https://gcc.gnu.org/bugzilla//show_bug.cgi?id=101476
The analyses are detailed, with a proposed fix.
Would be nice if this is finally fixed.

@jussihi
Copy link

jussihi commented Sep 13, 2022

I think that I've just stumbled across this bug today, it happens after I pthread_join a pthread_cancel'ed thread. ASan complains about a stack-buffer-overflow. Is this still not fixed?

    #0 0x7f0b41c5a3ca in __interceptor_sigaltstack /usr/src/debug/gcc/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:10168
    #1 0x7f0b41cdfe89 in __sanitizer::UnsetAlternateSignalStack() /usr/src/debug/gcc/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cpp:189
    #2 0x7f0b41cceb95 in __asan::AsanThread::Destroy() /usr/src/debug/gcc/libsanitizer/asan/asan_thread.cpp:110
    #3 0x7f0b41a9c900  (/usr/lib/libc.so.6+0x83900)
    #4 0x7f0b41a9f5cd  (/usr/lib/libc.so.6+0x865cd)
    #5 0x7f0b41b216ff  (/usr/lib/libc.so.6+0x1086ff)

@mid-kid
Copy link

mid-kid commented Mar 7, 2023

I'm hitting the same issue, here's a simple example to reproduce this with:
https://godbolt.org/z/f5rMWbcGq

@marxin
Copy link

marxin commented Mar 8, 2023

Please paste source files directly here and do not link it to an external service:

#include <pthread.h>

static void *thread_test(void *user)
{
    (void)user;
    struct timespec now1;
    struct timespec now2;
    struct timespec now3;
    for (;;) {
        clock_gettime(CLOCK_MONOTONIC, &now1);
        clock_gettime(CLOCK_MONOTONIC, &now2);
        clock_gettime(CLOCK_MONOTONIC, &now3);

        clock_nanosleep(CLOCK_MONOTONIC, 0,
            &(struct timespec){.tv_nsec = 1000000}, NULL);
    }
    return NULL;
}

int main()
{
    pthread_t thread;
    pthread_create(&thread, NULL, thread_test, NULL);
    clock_nanosleep(CLOCK_MONOTONIC, 0,
        &(struct timespec){.tv_nsec = 10000000}, NULL);
    pthread_cancel(thread);
    pthread_join(thread, NULL);
}

@ArnaudBienner ArnaudBienner mentioned this issue Mar 28, 2023
Open
@ArnaudBienner
Copy link

I faced the same issue (at least same error message) in some code using signals and pthread_cancel and in my case, disabling use_sigaltstack (i.e. ASAN_OPTIONS=use_sigaltstack=0 not 1 like mentioned in previous answers) fixed the issue.

spdk-bot pushed a commit to spdk/spdk that referenced this issue Sep 21, 2023
@sberbz @tomzawadzki
test/common: add asan_option use_sigaltstack=0
cc2a492 
Currently when SPDK is built with `--enable-asan`, flag is not passed
to DPDK mainly due to AsanCheckFailed that is seemingly a false
positive. Forcing ASan to use single stack fix this issue, but it's
not well documented (at all) and conversation attached below, has
different and contradiction solutions proposed.

This patch is only relevant if `-Db_sanitize=address` is
enabled for DPDK.

Mentioned conversation on ASan github and GCC bugzilla
```
google/sanitizers#1171
https://gcc.gnu.org/bugzilla//show_bug.cgi?id=101476
```

Update comment regarding librados issue.

Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Change-Id: Ic6a5743b56972ff1f9f9c0fefed34084180bb65b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/19709
Reviewed-by: Ben Walker <ben@nvidia.com>
Reviewed-by: Jim Harris <jim.harris@gmail.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
This was referenced Jan 26, 2024
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@eugenis @mid-kid @ArnaudBienner @marxin @stsp @jmssil @jussihi