v0.14.0rc2 crash on shutdown

[dooglus]

I had rc2 running overnight. It was fully synced. My computer was being slow, which is sometimes caused by bitcoin-qt doing a lot of disk access. I clicked the X in the top right of the bitcoin-qt gui window to shut it down to see if that would help. Nothing happened - the gui window didn't go away. I figured I had misclicked (using a tap-to-click touchpad it happens) so I clicked the X again. bitcoin-qt crashed, like this:

(gdb) run
Starting program: /home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe9d6b700 (LWP 7353)]
[New Thread 0x7fffdd8ff700 (LWP 7356)]
[New Thread 0x7fffd5ffe700 (LWP 7357)]
[New Thread 0x7fffd57ad700 (LWP 7358)]
[New Thread 0x7fffd4f5c700 (LWP 7359)]
[New Thread 0x7fffcffff700 (LWP 7360)]
[New Thread 0x7fffcf7ae700 (LWP 7367)]
[New Thread 0x7fffcef5d700 (LWP 7368)]
[New Thread 0x7fffce70c700 (LWP 7369)]
[New Thread 0x7fffcdebb700 (LWP 7370)]
[New Thread 0x7fffcd66a700 (LWP 7371)]
[New Thread 0x7fffb972f700 (LWP 7375)]
[New Thread 0x7fff9d18f700 (LWP 7399)]
[New Thread 0x7fff9c93e700 (LWP 7400)]
[New Thread 0x7fff97fff700 (LWP 7401)]
[New Thread 0x7fff977ae700 (LWP 7402)]
[New Thread 0x7fff96f5d700 (LWP 7403)]
[New Thread 0x7fff9670c700 (LWP 7404)]
[New Thread 0x7fff95ebb700 (LWP 7405)]
[New Thread 0x7fff95033700 (LWP 7407)]
[New Thread 0x7fff77fff700 (LWP 7408)]
[Thread 0x7fff97fff700 (LWP 7401) exited]
[Thread 0x7fff9d18f700 (LWP 7399) exited]
[Thread 0x7fff95033700 (LWP 7407) exited]
[Thread 0x7fffce70c700 (LWP 7369) exited]
[Thread 0x7fffcd66a700 (LWP 7371) exited]
[Thread 0x7fffcef5d700 (LWP 7368) exited]
[Thread 0x7fffcdebb700 (LWP 7370) exited]
[Thread 0x7fffcf7ae700 (LWP 7367) exited]
[Thread 0x7fff96f5d700 (LWP 7403) exited]
[Thread 0x7fff977ae700 (LWP 7402) exited]
[Thread 0x7fff9670c700 (LWP 7404) exited]
[Thread 0x7fff9c93e700 (LWP 7400) exited]
[Thread 0x7fffd5ffe700 (LWP 7357) exited]
[Thread 0x7fff95ebb700 (LWP 7405) exited]
[Thread 0x7fffd4f5c700 (LWP 7359) exited]
[Thread 0x7fffcffff700 (LWP 7360) exited]
[Thread 0x7fffd57ad700 (LWP 7358) exited]
[Thread 0x7fffdd8ff700 (LWP 7356) exited]
[Thread 0x7fff77fff700 (LWP 7408) exited]
[Thread 0x7fffe9d6b700 (LWP 7353) exited]
bitcoin-qt: /usr/include/boost/thread/pthread/recursive_mutex.hpp:101: boost::recursive_mutex::~recursive_mutex(): Assertion `!pthread_mutex_destroy(&m)' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff3292067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) thread apply all where

Thread 13 (Thread 0x7fffb972f700 (LWP 7375)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00005555559bbb7b in BGThread (this=0x7fffd84d1990) at leveldb/util/env_posix.cc:574
#2  leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7fffd84d1990) at leveldb/util/env_posix.cc:521
#3  0x00007ffff4cc30a4 in start_thread (arg=0x7fffb972f700) at pthread_create.c:309
#4  0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7ffff7fb8780 (LWP 7340)):
#0  0x00007ffff3292067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff3293448 in __GI_abort () at abort.c:89
#2  0x00007ffff328b266 in __assert_fail_base (fmt=0x7ffff33c3f18 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x555555d9b940 "!pthread_mutex_destroy(&m)", file=file@entry=0x5555559ebc80 "/usr/include/boost/thread/pthread/recursive_mutex.hpp", 
    line=line@entry=101, 
    function=function@entry=0x555555ec1e20 <_ZZN5boost15recursive_mutexD4EvE19__PRETTY_FUNCTION__> "boost::recursive_mutex::~recursive_mutex()") at assert.c:92
#3  0x00007ffff328b312 in __GI___assert_fail (assertion=0x555555d9b940 "!pthread_mutex_destroy(&m)", 
    file=0x5555559ebc80 "/usr/include/boost/thread/pthread/recursive_mutex.hpp", line=101, 
    function=0x555555ec1e20 <_ZZN5boost15recursive_mutexD4EvE19__PRETTY_FUNCTION__> "boost::recursive_mutex::~recursive_mutex()") at assert.c:101
#4  0x00005555555c5601 in boost::recursive_mutex::~recursive_mutex (this=0x555556206220, __in_chrg=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:101
#5  0x000055555594fc78 in ~recursive_mutex (this=0x555556206220, __in_chrg=<optimized out>) at util.cpp:165
#6  ~AnnotatedMixin (this=0x555556206220, __in_chrg=<optimized out>) at sync.h:55
#7  ~CCriticalSection (this=0x555556206220, __in_chrg=<optimized out>) at sync.h:95
#8  CInit::~CInit (this=<optimized out>, __in_chrg=<optimized out>) at util.cpp:164
#9  0x00007ffff3294b29 in __run_exit_handlers (status=0, listp=0x7ffff36025a8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#10 0x00007ffff3294b75 in __GI_exit (status=<optimized out>) at exit.c:104
#11 0x00007ffff327eb4c in __libc_start_main (main=0x5555555c65c0 <main(int, char**)>, argc=1, argv=0x7fffffffe5b8, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffffffe5a8) at libc-start.c:321
#12 0x00005555555d7b08 in _start ()
(gdb) 

I don't have a lot of trust in my computer. It sometimes fails to run g++ correctly for example. See #9814 (comment) for other discussion of this. But this issue looks real to me.

[laanwj]

These are line 164 and 165 of util.cpp in 0.14.0rc2:

        for (int i = 0; i < CRYPTO_num_locks(); i++)
            delete ppmutexOpenSSL[i];

Don't see how this could fail, really :/

[dooglus]

I've installed memtest86+ ("thorough real-mode memory tester") and will run it next time I reboot the machine. Again, this could be another "ghost" bug caused by a hardware error but I thought it best to report it anyway.

Edit: memtest86+ found no memory errors, though it does hang after 2.5 minutes if I let it use all 4 cores in parallel while testing. No idea what that means.

[fanquake]

Going to close this for the same reason as #9814. If we ever get another report it can be reopened.