test/threading_cuda random crash at exit

[WilliamTambellini]

Worrying crash in test/threading_cuda :
ArrayFire v3.7.0 (CUDA, 64-bit Linux, build 70ef198)
Platform: CUDA Toolkit 10.0, Driver: 440.33.01
[0] GeForce GTX 1080 Ti, 11179 MB, CUDA Compute 6.1

$ test/threading_cuda
Running main() from /local/nbuild/jenkins/workspace/AF-Release-Linux/test/gtest/googletest/src/gtest_main.cc
[==========] Running 9 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 9 tests from Threading
[ RUN ] Threading.SetPerThreadActiveDevice
Image IO Not Configured. Test will exit
[ OK ] Threading.SetPerThreadActiveDevice (0 ms)
[ RUN ] Threading.SimultaneousRead
[ OK ] Threading.SimultaneousRead (5594 ms)
[ RUN ] Threading.MemoryManagementScope
[ OK ] Threading.MemoryManagementScope (2008 ms)
[ RUN ] Threading.MemoryManagement_JIT_Node
[ OK ] Threading.MemoryManagement_JIT_Node (8 ms)
[ RUN ] Threading.FFT_R2C
[ OK ] Threading.FFT_R2C (687 ms)
[ RUN ] Threading.FFT_C2C
[ OK ] Threading.FFT_C2C (13 ms)
[ RUN ] Threading.FFT_ALL
[ OK ] Threading.FFT_ALL (12 ms)
[ RUN ] Threading.BLAS
[ OK ] Threading.BLAS (339 ms)
[ RUN ] Threading.Sparse
[ OK ] Threading.Sparse (12699 ms)
[----------] 9 tests from Threading (21360 ms total)

[----------] Global test environment tear-down
[==========] 9 tests from 1 test case ran. (21360 ms total)
[ PASSED ] 9 tests.

YOU HAVE 2 DISABLED TESTS

Segmentation fault

with gdb
...
[ OK ] Threading.Sparse (12035 ms)
[----------] 9 tests from Threading (20924 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 1 test case ran. (20924 ms total)
[ PASSED ] 9 tests.

YOU HAVE 2 DISABLED TESTS

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x00007fffe3f9f8ad in ?? () from /usr/local/cuda-10.0/lib64/libnvrtc.so.10.0
#2 0x00007fffe3f9f8d5 in ?? () from /usr/local/cuda-10.0/lib64/libnvrtc.so.10.0
#3 0x00007fffe5577d9d in __cxa_finalize () from /lib64/libc.so.6
#4 0x00007fffe3c26e06 in ?? () from /usr/local/cuda-10.0/lib64/libnvrtc.so.10.0
#5 0x0000000000000016 in ?? ()
#6 0x0000000000000000 in ?? ()

[9prady9]

I was unable to reproduce this using neither CUDA 10.0 nor 10.2. I am on Arch Linux, what linux-distro are you using ?

[WilliamTambellini]

Tks @9prady9 . That one runs on CentOS6 and compiled with gcc6.1. Which gcc version do you use ?

[9prady9]

I used gcc-7. I have 8 and 9, but 10.0 CUDA works with 7 only.

[WilliamTambellini]

Warning : the crash seems random.

[ncorgan]

I've been running into the same thing (identical stack trace). It happens more reliably the more ArrayFire CUDA calls are made in a multi-threaded program, but running this minimal code in quick succession will result in the same crash.

int main(int, char**)
{
    std::cout << "ArrayFire Config Info" << std::endl;
 
    const int availableBackends = af::getAvailableBackends();
    const std::vector<af::Backend> backends =
    {   
        AF_BACKEND_CUDA,
        AF_BACKEND_OPENCL,
        AF_BACKEND_CPU
    };  
    for(af::Backend backend: backends)
    {   
        if(availableBackends & backend)
        {
            std::cout << std::endl;
            af::setBackend(backend);
            af::info();
        }
    }   
}

Device configuration:

ArrayFire v3.7.0 (CUDA, 64-bit Linux, build ba46c2ed)
Platform: CUDA Toolkit 10.2, Driver: 440.33.01
[0] GeForce RTX 2070 SUPER, 7980 MB, CUDA Compute 7.5

ArrayFire v3.7.0 (OpenCL, 64-bit Linux, build ba46c2ed)
[0] NVIDIA: GeForce RTX 2070 SUPER, 7979 MB -- OpenCL 1.2 CUDA -- Device driver 440.33.01 -- FP64 Support: True -- Unified Memory (False)

ArrayFire v3.7.0 (CPU, 64-bit Linux, build ba46c2ed)
[0] Intel: Intel(R) Core(TM) i7-9700K CPU @ 3.60GHz, 15855 MB, Max threads(8) GNU Compiler Collection(GCC/G++) 7.4.0

Ubuntu 18.04.3, uname -a output:

Linux lagann 5.3.0-28-generic #30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[umar456]

@ncorgan Can you try again with the v3.7 or the master branch. That is a pretty old commit.

[ncorgan]

I've confirmed the same behavior with both the v3.7 and master branches.

[umar456]

@ncorgan Are you saying that the code you posted results in this stacktrace?

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x00007fffe3f9f8ad in ?? () from /usr/local/cuda-10.0/lib64/libnvrtc.so.10.0
#2 0x00007fffe3f9f8d5 in ?? () from /usr/local/cuda-10.0/lib64/libnvrtc.so.10.0
#3 0x00007fffe5577d9d in __cxa_finalize () from /lib64/libc.so.6
#4 0x00007fffe3c26e06 in ?? () from /usr/local/cuda-10.0/lib64/libnvrtc.so.10.0
#5 0x0000000000000016 in ?? ()
#6 0x0000000000000000 in ?? ()

[ncorgan]

In rerunning, that stacktrace appears when running more intensive code. I'm seeing this with the minimal code example, in the thread spawned when running af::info with the CPU backend:

#0  0x00007ffff762b66c in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007ffff67ab6db in start_thread (arg=0x7fff889bf700) at pthread_create.c:463
#2  0x00007ffff708688f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

This is possibly a separate issue, but I can confirm that the stacktrace you posted is still present with master. I'm looking further into it.

[umar456]

Hmm. thats in the standard library. I am using 8.3 to compile my code. Let me switch to an older version and try on Ubuntu.

[ncorgan]

I'm starting a bisect on the CUDA threading crash issue. Given build times, this will take a while, but I'll let you know what comes out of it.

[umar456]

@ncorgan can you give me the value for CUDA_architecture_build_targets from your CMakeCache.txt in your build folder?

[ncorgan]

I ran into a slightly different stacktrace in v3.5.0, but I suspect the root is similar:

#0  0x0000000000000000 in ?? ()
#1  0x00007ffee255febd in ?? () from /usr/local/cuda-10.2/targets/x86_64-linux/lib/libnvrtc.so.10.2
#2  0x00007ffee255fee5 in ?? () from /usr/local/cuda-10.2/targets/x86_64-linux/lib/libnvrtc.so.10.2
#3  0x00007ffff5edb041 in __run_exit_handlers (status=0, listp=0x7ffff6283718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#4  0x00007ffff5edb13a in __GI_exit (status=<optimized out>) at exit.c:139
#5  0x00007ffff5eb9b9e in __libc_start_main (main=0x5555555620c0 <main>, argc=2, argv=0x7fffffffe028, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe018) at ../csu/libc-start.c:344
#6  0x000055555556273a in _start ()

I'm running into build issues going further back than that, but a common thread here (for both the OP and my tests) is the CUDA driver (440.33.01). I'm not sure if it's related, but the common behavior between ArrayFire releases two years apart might be explained by this.

[ncorgan]

CUDA_architecture_build_targets:STRING='7.5 '