Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gui, wallet: random abort (segmentation fault) running master/HEAD #9683

Closed
dooglus opened this issue Feb 4, 2017 · 79 comments

Comments

Projects
None yet
8 participants
@dooglus
Copy link
Contributor

commented Feb 4, 2017

I'm running the current master branch version (1c2edd9). I recently saw it crash shortly after starting up.

$ ./bitcoin-qt
    *** Error in `./bitcoin-qt': free(): invalid next size (fast): 0x00007f6fabfb5f90 ***
    Aborted
$ 

Unfortunately I don't have a stack trace available.

Here's the full debug.log file contents:

2017-02-03 23:28:28 

[...]

2017-02-03 23:28:28 Bitcoin version v0.13.99.0-1c2edd9
2017-02-03 23:28:28 InitParameterInteraction: parameter interaction: -listen=0 -> setting -discover=0
2017-02-03 23:28:28 InitParameterInteraction: parameter interaction: -listen=0 -> setting -listenonion=0
2017-02-03 23:28:28 InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
2017-02-03 23:28:29 Assuming ancestors of block 0000000000000000030abc968e1bd635736e880b946085c93152969b9a81a6e2 have valid signatures.
2017-02-03 23:28:29 Default data directory /home/chris/.bitcoin
2017-02-03 23:28:29 Using data directory /home/chris/.bitcoin
2017-02-03 23:28:29 Using config file /home/chris/.bitcoin/bitcoin.conf
2017-02-03 23:28:29 Using at most 32 automatic connections (65536 file descriptors available)
2017-02-03 23:28:29 Using 32 MiB out of 32 requested for signature cache, able to store 1048576 elements
2017-02-03 23:28:29 Using 4 threads for script verification
2017-02-03 23:28:29 scheduler thread start
2017-02-03 23:28:29 HTTP: creating work queue of depth 16
2017-02-03 23:28:29 Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcuser for rpcauth auth generation.
2017-02-03 23:28:29 HTTP: starting 4 worker threads
2017-02-03 23:28:29 Using BerkeleyDB version Berkeley DB 5.3.28: (September  9, 2013)
2017-02-03 23:28:29 Using wallet wallet.dat
2017-02-03 23:28:29 init message: Verifying wallet...
2017-02-03 23:28:29 CDBEnv::Open: LogDir=/home/chris/.bitcoin/database ErrorFile=/home/chris/.bitcoin/db.log
2017-02-03 23:28:30 Cache configuration:
2017-02-03 23:28:30 * Using 37.5MiB for block index database
2017-02-03 23:28:30 * Using 8.0MiB for chain state database
2017-02-03 23:28:30 * Using 254.5MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space)
2017-02-03 23:28:30 init message: Loading block index...
2017-02-03 23:28:30 Opening LevelDB in /home/chris/.bitcoin/blocks/index
2017-02-03 23:28:31 Opened LevelDB successfully
2017-02-03 23:28:31 Using obfuscation key for /home/chris/.bitcoin/blocks/index: 0000000000000000
2017-02-03 23:28:31 Opening LevelDB in /home/chris/.bitcoin/chainstate
2017-02-03 23:28:32 Opened LevelDB successfully
2017-02-03 23:28:32 Using obfuscation key for /home/chris/.bitcoin/chainstate: 0000000000000000
2017-02-03 23:28:36 LoadBlockIndexDB: last block file = 724
2017-02-03 23:28:36 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=55, size=52580365, heights=451383...451437, time=2017-02-03...2017-02-03)
2017-02-03 23:28:36 Checking all blk files are present...
2017-02-03 23:28:36 LoadBlockIndexDB: transaction index enabled
2017-02-03 23:28:36 LoadBlockIndexDB: hashBestChain=00000000000000000214c910b71f60e755bec4c09905a3d25da1073fd6bfb1c5 height=451437 date=2017-02-03 23:25:50 progress=0.999997
2017-02-03 23:28:36 init message: Rewinding blocks...
2017-02-03 23:28:38 init message: Verifying blocks...
2017-02-03 23:28:38 Verifying last 1 blocks at level 3
2017-02-03 23:28:38 [0%]...[99%]...[DONE].
2017-02-03 23:28:39 No coin database inconsistencies in last 2 blocks (4519 transactions)
2017-02-03 23:28:39  block index            8807ms
2017-02-03 23:28:39 init message: Loading wallet...
2017-02-03 23:28:39 nFileVersion = 139900
2017-02-03 23:28:39 Keys: 0 plaintext, 2613 encrypted, 259 w/ metadata, 2613 total
2017-02-03 23:28:39  wallet                  609ms
2017-02-03 23:28:39 init message: Rescanning...
2017-02-03 23:28:39 Rescanning last 32 blocks (from block 451405)...
2017-02-03 23:28:42  rescan                 2454ms
2017-02-03 23:28:42 setKeyPool.size() = 79
2017-02-03 23:28:42 mapWallet.size() = 6538
2017-02-03 23:28:42 mapAddressBook.size() = 11963
2017-02-03 23:28:42 mapBlockIndex.size() = 452118
2017-02-03 23:28:42 nBestHeight = 451437
2017-02-03 23:28:42 init message: Loading addresses...
2017-02-03 23:28:42 Loaded 63143 addresses from peers.dat  209ms
2017-02-03 23:28:42 init message: Loading banlist...
2017-02-03 23:28:42 init message: Starting network threads...
2017-02-03 23:28:42 net thread start
2017-02-03 23:28:42 init message: Done loading
2017-02-03 23:28:42 dnsseed thread start
2017-02-03 23:28:42 opencon thread start
2017-02-03 23:28:42 msghand thread start
2017-02-03 23:28:42 addcon thread start
2017-02-03 23:28:42 GUI: Platform customization: "other"
2017-02-03 23:28:43 connect() to [2003:7a:4917:e400:108e:b3ea:5d0c:8ae4]:8333 failed: Network is unreachable (101)
2017-02-03 23:28:43 GUI: PaymentServer::LoadRootCAs: Loaded  169  root certificates
2017-02-03 23:28:43 receive version message: /Satoshi:0.12.0/: version 70012, blocks=451437, us=x.x.x.x:x, peer=0, peeraddr=89.184.65.85:8333
@TheBlueMatt

This comment has been minimized.

Copy link
Contributor

commented Feb 4, 2017

Hmm, there are a few races that can be seen on disconnect that need fixing (eg #9671 and one to be fixed a bit later), but its not clear that that would have caused this situation unless there were some pending writes to debug.log that were lost :/. Was there any non-standard configuration at the time I could try to reproduce in helgrind/valgrind (even strange distro or so).

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Feb 4, 2017

The configuraton is pretty standard. I pasted in it this comment.

I was restarting bitcoind and bitcoin-qt over and over to investigate the compacting issue (#9684). I didn't change anything (other than switching between bitcoind and bitcoin-qt), and only saw this crash once, shortly after a restart - it seems to have crashed after just 15 seconds.

I'm running on debian stable, with bdb 5.3.28. I don't think anything's unusual with my build other than that.

I'll rebuild from the master HEAD and report back if I see anything like this again.

Is there an option that will cause debug.log to be flushed after each write? I don't want to be losing unbuffered log messages to a crash.

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Feb 4, 2017

Is there an option that will cause debug.log to be flushed after each write? I don't want to be losing unbuffered log messages to a crash.

To answer my own question, it appears that the debug.log file is unbuffered by default:

From OpenDebugLog() in src/util.cpp:

boost::filesystem::path pathDebug = GetDataDir() / "debug.log";
fileout = fopen(pathDebug.string().c_str(), "a");
if (fileout) setbuf(fileout, NULL); // unbuffered
@laanwj

This comment has been minimized.

Copy link
Member

commented Feb 7, 2017

Crashes such as this (in libc itself, or segmentation faults) won't log anything to the debug log.

Personally I always run bitcoind in gdb: https://gist.github.com/laanwj/29bc141fb8d10608651c .

Another option to allow for post-mortem debugging is to configure your OS to save core dumps properly, but a lot of modern Linux distros and such eat these by default.

@laanwj

This comment has been minimized.

Copy link
Member

commented Aug 24, 2017

Closing, as there is no way to reproduce this and not enough information for diagnosing what caused the crash.

@laanwj laanwj closed this Aug 24, 2017

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

I still regularly get crashes like this, but cannot find a way to reproduce them reliably. I wrote a script to repeatably start and stop bitcoin-qt and it usually crashes within 10 restarts. I've not been able to get it to crash even once running bitcoind, so it looks like it's related to the GUI version only.

Is there any interest in fixing the problem? All my reports of crashes seem to get closed without being fixed, which is a little discouraging.

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

I'll rebuild from the v0.15.0rc2 tag and try to get a backtrace from a crash in gdb...

@laanwj laanwj reopened this Aug 24, 2017

@laanwj

This comment has been minimized.

Copy link
Member

commented Aug 24, 2017

Is there any interest in fixing the problem? All my reports of crashes seem to get closed without being fixed, which is a little discouraging.

I understand. But if there is no way to reproduce for us, and crash reports don't have enough information to trace the crash to a certain point in the code, at some point there's no use in keeping it open. There's just nothing we can do.

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

It took quite a lot of attempts, and I couldn't get it to happen inside gdb, but I turned on core dumping and managed to get it to crash.

I ran 'bitcoin-cli stop' in a loop in one shell, and 'bitcoin-qt' in a loop in another, and:

$ ulimit -c unlimited
$ while [[ ! -e core ]]; do echo "$(date) : starting"; ~/Programs/bitcoin.git/src/qt/bitcoin-qt; done
Thu Aug 24 10:58:50 PDT 2017 : starting
got stop request
Thu Aug 24 10:59:04 PDT 2017 : starting
got stop request
Thu Aug 24 10:59:33 PDT 2017 : starting
got stop request
Thu Aug 24 10:59:49 PDT 2017 : starting
got stop request
Thu Aug 24 11:00:25 PDT 2017 : starting
got stop request
Thu Aug 24 11:00:38 PDT 2017 : starting
got stop request
Thu Aug 24 11:01:01 PDT 2017 : starting
got stop request
Thu Aug 24 11:01:13 PDT 2017 : starting
got stop request
Thu Aug 24 11:01:26 PDT 2017 : starting
got stop request
Thu Aug 24 11:01:38 PDT 2017 : starting
got stop request
Thu Aug 24 11:01:51 PDT 2017 : starting
got stop request
Thu Aug 24 11:02:03 PDT 2017 : starting
got stop request
Thu Aug 24 11:02:18 PDT 2017 : starting
got stop request
Thu Aug 24 11:02:33 PDT 2017 : starting
got stop request
Thu Aug 24 11:02:45 PDT 2017 : starting
got stop request
Thu Aug 24 11:02:58 PDT 2017 : starting
got stop request
Thu Aug 24 11:03:10 PDT 2017 : starting
got stop request
Thu Aug 24 11:03:27 PDT 2017 : starting
got stop request
Thu Aug 24 11:03:40 PDT 2017 : starting
got stop request
Thu Aug 24 11:03:52 PDT 2017 : starting
got stop request
Thu Aug 24 11:04:03 PDT 2017 : starting
got stop request
Thu Aug 24 11:04:14 PDT 2017 : starting
got stop request
Thu Aug 24 11:04:26 PDT 2017 : starting
got stop request
Thu Aug 24 11:04:41 PDT 2017 : starting
got stop request
Thu Aug 24 11:04:53 PDT 2017 : starting
got stop request
Thu Aug 24 11:05:05 PDT 2017 : starting
got stop request
Thu Aug 24 11:05:19 PDT 2017 : starting
got stop request
Thu Aug 24 11:05:30 PDT 2017 : starting
got stop request
Thu Aug 24 11:05:46 PDT 2017 : starting
got stop request
Thu Aug 24 11:05:59 PDT 2017 : starting
got stop request
Thu Aug 24 11:06:15 PDT 2017 : starting
got stop request
Thu Aug 24 11:06:29 PDT 2017 : starting
got stop request
Thu Aug 24 11:06:44 PDT 2017 : starting
got stop request
Thu Aug 24 11:06:57 PDT 2017 : starting
got stop request
Thu Aug 24 11:07:12 PDT 2017 : starting
got stop request
Thu Aug 24 11:07:23 PDT 2017 : starting
got stop request
Thu Aug 24 11:07:39 PDT 2017 : starting
got stop request
Thu Aug 24 11:07:50 PDT 2017 : starting
got stop request
Thu Aug 24 11:08:01 PDT 2017 : starting
got stop request
Thu Aug 24 11:08:13 PDT 2017 : starting
got stop request
bitcoin-qt: /usr/include/boost/thread/pthread/recursive_mutex.hpp:101: boost::recursive_mutex::~recursive_mutex(): Assertion `!pthread_mutex_destroy(&m)' failed.
Aborted (core dumped)
$ 

The core file is 269225984 bytes long.

Loading it into gdb:

$ gdb qt/bitcoin-qt core.bak1
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from qt/bitcoin-qt...done.
[New LWP 23615]
[New LWP 23813]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f3e3161d067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) thread apply all where

Thread 2 (Thread 0x7f3dea0d7700 (LWP 23813)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055a249a17d8b in BGThread (this=0x7f3e14428e20) at leveldb/util/env_posix.cc:641
#2  leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7f3e14428e20) at leveldb/util/env_posix.cc:556
#3  0x00007f3e3304e064 in start_thread (arg=0x7f3dea0d7700) at pthread_create.c:309
#4  0x00007f3e316d062d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f3e36343780 (LWP 23615)):
#0  0x00007f3e3161d067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f3e3161e448 in __GI_abort () at abort.c:89
#2  0x00007f3e31616266 in __assert_fail_base (fmt=0x7f3e3174ef18 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x55a249e51a60 "!pthread_mutex_destroy(&m)", file=file@entry=0x55a249a4a700 "/usr/include/boost/thread/pthread/recursive_mutex.hpp", 
    line=line@entry=101, 
    function=function@entry=0x55a249f750e0 <_ZZN5boost15recursive_mutexD4EvE19__PRETTY_FUNCTION__> "boost::recursive_mutex::~recursive_mutex()") at assert.c:92
#3  0x00007f3e31616312 in __GI___assert_fail (assertion=0x55a249e51a60 "!pthread_mutex_destroy(&m)", 
    file=0x55a249a4a700 "/usr/include/boost/thread/pthread/recursive_mutex.hpp", line=101, 
    function=0x55a249f750e0 <_ZZN5boost15recursive_mutexD4EvE19__PRETTY_FUNCTION__> "boost::recursive_mutex::~recursive_mutex()") at assert.c:101
#4  0x000055a2495874a1 in boost::recursive_mutex::~recursive_mutex (this=0x55a24b41b718, __in_chrg=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:101
#5  0x000055a2499aadc4 in ~recursive_mutex (this=0x55a24b41b718, __in_chrg=<optimized out>) at util.cpp:150
#6  ~AnnotatedMixin (this=0x55a24b41b718, __in_chrg=<optimized out>) at sync.h:54
#7  ~CCriticalSection (this=0x55a24b41b718, __in_chrg=<optimized out>) at sync.h:94
#8  operator() (this=<optimized out>, __ptr=0x55a24b41b448) at /usr/include/c++/4.9/bits/unique_ptr.h:119
#9  reset (__p=0x55a24b41b448, this=0x55a24a2917a0 <ppmutexOpenSSL>) at /usr/include/c++/4.9/bits/unique_ptr.h:581
#10 CInit::~CInit (this=<optimized out>, __in_chrg=<optimized out>) at util.cpp:149
#11 0x00007f3e3161fb29 in __run_exit_handlers (status=0, listp=0x7f3e3198d5a8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#12 0x00007f3e3161fb75 in __GI_exit (status=<optimized out>) at exit.c:104
#13 0x00007f3e31609b4c in __libc_start_main (main=0x55a2495884c0 <main(int, char**)>, argc=1, argv=0x7ffe8dba7ad8, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7ffe8dba7ac8) at libc-start.c:321
#14 0x000055a249599a38 in _start ()
(gdb) 

Is there anything useful I can type there to give you the information you need to fix the error? Edit: Please tell me which core dump to load up. I'm calling this one core.bak1 for reference.

This is qt/bitcoin-qt built against the v0.15.0rc2 tag with a single line added so I can see whether the crash happened before or after the 'bitcoin-cli stop' request was received or not:

$ git diff
diff --git a/src/rpc/server.cpp b/src/rpc/server.cpp
index 9ad8d22..3ff0da1 100644
--- a/src/rpc/server.cpp
+++ b/src/rpc/server.cpp
@@ -243,6 +243,7 @@ UniValue stop(const JSONRPCRequest& jsonRequest)
             "\nStop Bitcoin server.");
     // Event loop will exit after current HTTP requests have been handled, so
     // this reply will get back to the client.
+    printf("got stop request\n");
     StartShutdown();
     return "Bitcoin server stopping";
 }
$ 
@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

I'll keep a copy of that core file and try to get another crash. I've seen different messages on the console when it crashes - sometimes I think it talks about duplicate free() calls, or something similar indicative of a memory allocation race condition.

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

I got a 2nd crash, and it did indeed complain about free():

$ while [[ ! -e core ]]; do echo "$(date) : starting"; ~/Programs/bitcoin.git/src/qt/bitcoin-qt; done
Thu Aug 24 11:27:27 PDT 2017 : starting
got stop request
Thu Aug 24 11:27:43 PDT 2017 : starting
got stop request
Thu Aug 24 11:27:59 PDT 2017 : starting
got stop request
Thu Aug 24 11:28:13 PDT 2017 : starting
got stop request
Thu Aug 24 11:28:28 PDT 2017 : starting
got stop request
Thu Aug 24 11:28:39 PDT 2017 : starting
got stop request
Thu Aug 24 11:28:52 PDT 2017 : starting
got stop request
Thu Aug 24 11:29:07 PDT 2017 : starting
got stop request
Thu Aug 24 11:29:25 PDT 2017 : starting
got stop request
Thu Aug 24 11:29:35 PDT 2017 : starting
*** Error in `/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt': free(): invalid next size (fast): 0x00005637d209d7b0 ***
Aborted (core dumped)
$ 

This time the core file was 373645312 bytes long. I was sleeping 5 seconds between each bitcoin-cli stop this time instead of 1 second, so the bitcoin-qt is likely to have had more of a chance to connect to peers than in the previous run.

Here's the backtrace:

$ gdb qt/bitcoin-qt core.bak2
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from qt/bitcoin-qt...done.
[New LWP 28788]
[New LWP 28793]
[New LWP 28790]
[New LWP 28807]
[New LWP 28796]
[New LWP 28794]
[New LWP 28797]
[New LWP 28795]
[New LWP 28798]
[New LWP 28806]
[New LWP 28808]
[New LWP 28800]
[New LWP 28804]
[New LWP 28792]
[New LWP 28810]
[New LWP 28809]
[New LWP 28805]
[New LWP 28789]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f045d1d7067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) thread apply all where

Thread 18 (Thread 0x7f0453ca6700 (LWP 28789)):
#0  0x00007f045d281aed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f045a48b252 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#2  0x00007f045a48cddf in xcb_wait_for_event () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#3  0x00007f04563f91b9 in ?? () from /usr/lib/x86_64-linux-gnu/qt5/plugins/platforms/libqxcb.so
#4  0x00007f045f400d2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007f045ec08064 in start_thread (arg=0x7f0453ca6700) at pthread_create.c:309
#6  0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 17 (Thread 0x7f0413175700 (LWP 28805)):
#0  0x00007f045d283893 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00005637cf6b6df5 in CConnman::ThreadSocketHandler (this=<optimized out>) at net.cpp:1256
#2  0x00005637cf689d9d in operator() (this=0x7f0413174e70) at /usr/include/c++/4.9/functional:2439
#3  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x5637cfe47d31 "net", func=...) at util.h:302
#4  0x00005637cf6ba9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#5  operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#6  std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#7  0x00007f045db1a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007f045ec08064 in start_thread (arg=0x7f0413175700) at pthread_create.c:309
#9  0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 16 (Thread 0x7f0410a51700 (LWP 28809)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00005637cf6b1f15 in __gthread_cond_timedwait (__abs_timeout=0x7f0410a50ca0, __mutex=<optimized out>, __cond=0x7f04405bd440)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>, __lock=..., this=0x7f04405bd440)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x7f04405bd440)
    at /usr/include/c++/4.9/condition_variable:118
#4  wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CConnman::ThreadMessageHandler()::<lambda()> > (
    __p=..., __atime=..., __lock=..., this=0x7f04405bd440) at /usr/include/c++/4.9/condition_variable:128
#5  CConnman::ThreadMessageHandler (this=0x7f044056d040) at net.cpp:2021
#6  0x00005637cf689d9d in operator() (this=0x7f0410a50e70) at /usr/include/c++/4.9/functional:2439
#7  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x5637cfe14ebd "msghand", func=...) at util.h:302
#8  0x00005637cf6ba9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#9  operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#10 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#11 0x00007f045db1a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007f045ec08064 in start_thread (arg=0x7f0410a51700) at pthread_create.c:309
#13 0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 15 (Thread 0x7f03fbfff700 (LWP 28810)):
#0  0x00007f045d281aed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f045c3c6ee4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f045c3c6ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f045f6961cc in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#4  0x00007f045f63b512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007f045f3fbcb4 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007f045f400d2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007f045ec08064 in start_thread (arg=0x7f03fbfff700) at pthread_create.c:309
#8  0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 14 (Thread 0x7f043ffff700 (LWP 28792)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00005637cf54f4cb in boost::condition_variable::wait (this=0x5637d0239e68 <scriptcheckqueue+40>, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
#2  0x00005637cf7f8bb0 in CCheckQueue<CScriptCheck>::Loop (this=0x5637d0239e40 <scriptcheckqueue>, fMaster=false) at checkqueue.h:102
#3  0x00007f046121faea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#4  0x00007f045ec08064 in start_thread (arg=0x7f043ffff700) at pthread_create.c:309
#5  0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 13 (Thread 0x7f042583b700 (LWP 28804)):
#0  0x00005637cf5255e0 in std::string::append(char const*, unsigned long)@plt ()
#1  0x00005637cf9bc003 in append (__s=0x5637cff2cdce "leveldb.BuiltinBloomFilter2", this=0x7f0425839100) at /usr/include/c++/4.9/bits/basic_string.h:1026
#2  leveldb::Table::ReadMeta (this=0x7f04180def80, footer=...) at leveldb/table/table.cc:111
#3  0x00005637cf9bc482 in leveldb::Table::Open (options=..., file=0x7f041811b500, size=size@entry=138541, table=table@entry=0x7f0425839318)
    at leveldb/table/table.cc:83
#4  0x00005637cf9a497a in leveldb::TableCache::FindTable (this=this@entry=0x7f041c940870, file_number=<optimized out>, file_size=138541, 
    handle=handle@entry=0x7f04258393a0) at leveldb/db/table_cache.cc:64
#5  0x00005637cf9a4c7f in leveldb::TableCache::Get (this=0x7f041c940870, options=..., file_number=<optimized out>, file_size=<optimized out>, k=..., 
    arg=0x7f04258394f0, saver=0x5637cf9a64d0 <leveldb::SaveValue(void*, leveldb::Slice const&, leveldb::Slice const&)>) at leveldb/db/table_cache.cc:112
#6  0x00005637cf9ae545 in leveldb::Version::Get (this=this@entry=0x7f04200569c0, options=..., k=..., value=value@entry=0x7f0425839720, 
    stats=stats@entry=0x7f04258395a0) at leveldb/db/version_set.cc:408
#7  0x00005637cf994b19 in leveldb::DBImpl::Get (this=0x7f041c940bd0, options=..., key=..., value=0x7f0425839720) at leveldb/db/db_impl.cc:1141
#8  0x00005637cf79b9eb in Read<(anonymous namespace)::CoinEntry, Coin> (value=..., key=..., this=0x7f041c94abc8) at dbwrapper.h:233
#9  CCoinsViewDB::GetCoin (this=0x7f041c94abc0, outpoint=..., coin=...) at txdb.cpp:62
#10 0x00005637cf698419 in CCoinsViewErrorCatcher::GetCoin (this=<optimized out>, outpoint=..., coin=...) at init.cpp:142
#11 0x00005637cf90dc91 in CCoinsViewCache::FetchCoin (this=0x7f041c9fc940, outpoint=...) at coins.cpp:48
#12 0x00005637cf90dd66 in CCoinsViewCache::GetCoin (this=<optimized out>, outpoint=..., coin=...) at coins.cpp:61
#13 0x00005637cf7a7e68 in CCoinsViewMemPool::GetCoin (this=0x7f042583a250, outpoint=..., coin=...) at txmempool.cpp:907
#14 0x00005637cf90dc91 in CCoinsViewCache::FetchCoin (this=this@entry=0x7f042583a400, outpoint=...) at coins.cpp:48
#15 0x00005637cf90e539 in CCoinsViewCache::HaveCoin (this=this@entry=0x7f042583a400, outpoint=...) at coins.cpp:129
#16 0x00005637cf7e0f9d in AcceptToMemoryPoolWorker (chainparams=..., pool=..., state=..., ptx=std::shared_ptr (count 1, weak 0) 0x7f0418137b30, 
    fLimitFree=fLimitFree@entry=true, pfMissingInputs=pfMissingInputs@entry=0x0, nAcceptTime=1503591673, plTxnReplaced=0x0, fOverrideMempoolLimit=false, 
    nAbsurdFee=@0x7f042583a668: 0, coins_to_uncache=std::vector of length 118, capacity 128 = {...}) at validation.cpp:541
#17 0x00005637cf7e4d5f in AcceptToMemoryPoolWithTime (chainparams=..., pool=..., state=..., tx=std::shared_ptr (count 1, weak 0) 0x7f0418137b30, 
    fLimitFree=fLimitFree@entry=true, pfMissingInputs=pfMissingInputs@entry=0x0, nAcceptTime=1503591673, plTxnReplaced=0x0, fOverrideMempoolLimit=false, 
    nAbsurdFee=0) at validation.cpp:880
#18 0x00005637cf7ec648 in LoadMempool () at validation.cpp:4302
#19 0x00005637cf664fb7 in ThreadImport (vImportFiles=std::vector of length 0, capacity 0) at init.cpp:692
#20 0x00005637cf68ecca in operator()<void (*)(std::vector<boost::filesystem::path>), boost::_bi::list0> (f=<optimized out>, a=<synthetic pointer>, 
    this=<optimized out>) at /usr/include/boost/bind/bind.hpp:253
#21 operator() (this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:20
#22 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >), boost::_bi::list1<boost::_bi::value<std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> > > > > >::run (this=<optimized out>)
    at /usr/include/boost/thread/detail/thread.hpp:117
#23 0x00007f046121faea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#24 0x00007f045ec08064 in start_thread (arg=0x7f042583b700) at pthread_create.c:309
#25 0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 12 (Thread 0x7f042d10f700 (LWP 28800)):
#0  __GI___libc_malloc (bytes=59) at malloc.c:2882
#1  0x00007f045dac32e8 in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007f045db22e99 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007f045db24815 in char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f045db248ac in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, unsigned long, std::allocator<char> const&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00005637cf9c7da4 in leveldb::InternalKeyComparator::FindShortestSeparator (this=0x7f041c940be0, start=0x7f04200535d8, limit=...) at leveldb/db/dbformat.cc:74
#6  0x00005637cf9bae17 in leveldb::TableBuilder::Add (this=0x7f0420056510, key=..., value=...) at leveldb/table/table_builder.cc:102
#7  0x00005637cf99a8e2 in leveldb::DBImpl::DoCompactionWork (this=this@entry=0x7f041c940bd0, compact=compact@entry=0x7f042000e440) at leveldb/db/db_impl.cc:995
#8  0x00005637cf99b17f in leveldb::DBImpl::BackgroundCompaction (this=this@entry=0x7f041c940bd0) at leveldb/db/db_impl.cc:737
#9  0x00005637cf99bc32 in leveldb::DBImpl::BackgroundCall (this=0x7f041c940bd0) at leveldb/db/db_impl.cc:675
#10 0x00005637cf9c2dd3 in BGThread (this=0x7f0440548590) at leveldb/util/env_posix.cc:649
#11 leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7f0440548590) at leveldb/util/env_posix.cc:556
#12 0x00007f045ec08064 in start_thread (arg=0x7f042d10f700) at pthread_create.c:309
#13 0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 11 (Thread 0x7f04112a2700 (LWP 28808)):
#0  0x00007f045d283893 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00005637cf92bf29 in ConnectSocketDirectly (addrConnect=..., hSocketRet=@0x7f04112a161c: 4294967295, nTimeout=nTimeout@entry=5000) at netbase.cpp:442
#2  0x00005637cf92da96 in ConnectSocket (addrDest=..., hSocketRet=@0x7f04112a161c: 4294967295, nTimeout=5000, 
    outProxyConnectionFailed=outProxyConnectionFailed@entry=0x7f04112a1619) at netbase.cpp:571
#3  0x00005637cf6ab91a in CConnman::ConnectNode (this=this@entry=0x7f044056d040, addrConnect=..., pszDest=0x0, fCountFailure=<optimized out>) at net.cpp:390
#4  0x00005637cf6afc42 in CConnman::OpenNetworkConnection (this=this@entry=0x7f044056d040, addrConnect=..., fCountFailure=<optimized out>, 
    grantOutbound=grantOutbound@entry=0x7f04112a1b70, pszDest=pszDest@entry=0x0, fOneShot=fOneShot@entry=false, fFeeler=false, fAddnode=false) at net.cpp:1956
#5  0x00005637cf6b0a83 in CConnman::ThreadOpenConnections (this=0x7f044056d040) at net.cpp:1844
#6  0x00005637cf689d9d in operator() (this=0x7f04112a1e70) at /usr/include/c++/4.9/functional:2439
#7  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x5637cfe14eb5 "opencon", func=...) at util.h:302
#8  0x00005637cf6ba9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#9  operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#10 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#11 0x00007f045db1a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007f045ec08064 in start_thread (arg=0x7f04112a2700) at pthread_create.c:309
#13 0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 10 (Thread 0x7f0412344700 (LWP 28806)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00005637cf9507b2 in __gthread_cond_timedwait (__abs_timeout=0x7f0412343970, __mutex=0x7f04405bd4d0, __cond=0x7f04405bd4a0)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7f04405bd4a0)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7f04405bd4a0)
    at /usr/include/c++/4.9/condition_variable:105
#4  wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>, this=0x7f04405bd4a0) at /usr/include/c++/4.9/condition_variable:128
#5  wait_for<long int, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __rtime=<synthetic pointer>, 
    __lock=<synthetic pointer>, this=0x7f04405bd4a0) at /usr/include/c++/4.9/condition_variable:144
#6  CThreadInterrupt::sleep_for (this=this@entry=0x7f04405bd4a0, rel_time=...) at threadinterrupt.cpp:30
#7  0x00005637cf950860 in CThreadInterrupt::sleep_for (this=this@entry=0x7f04405bd4a0, rel_time=..., rel_time@entry=...) at threadinterrupt.cpp:35
#8  0x00005637cf6a3647 in CConnman::ThreadDNSAddressSeed (this=0x7f044056d040) at net.cpp:1579
#9  0x00005637cf689d9d in operator() (this=0x7f0412343e70) at /usr/include/c++/4.9/functional:2439
#10 TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x5637cfdfcb33 "dnsseed", func=...) at util.h:302
#11 0x00005637cf6ba9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#12 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#13 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#14 0x00007f045db1a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007f045ec08064 in start_thread (arg=0x7f0412344700) at pthread_create.c:309
#16 0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 9 (Thread 0x7f043ce19700 (LWP 28798)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f045db16c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005637cf83c7bb in Run (this=0x7f0440046080) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7f0440046080) at httpserver.cpp:351
#4  0x00007f045db1a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f045ec08064 in start_thread (arg=0x7f043ce19700) at pthread_create.c:309
#6  0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 8 (Thread 0x7f043e70c700 (LWP 28795)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f045db16c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005637cf83c7bb in Run (this=0x7f0440046080) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7f0440046080) at httpserver.cpp:351
#4  0x00007f045db1a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f045ec08064 in start_thread (arg=0x7f043e70c700) at pthread_create.c:309
#6  0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 7 (Thread 0x7f043d66a700 (LWP 28797)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f045db16c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005637cf83c7bb in Run (this=0x7f0440046080) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7f0440046080) at httpserver.cpp:351
#4  0x00007f045db1a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f045ec08064 in start_thread (arg=0x7f043d66a700) at pthread_create.c:309
#6  0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 6 (Thread 0x7f043ef5d700 (LWP 28794)):
#0  0x00007f045d28ac03 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f045dd93b08 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#2  0x00007f045dd7ed5a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#3  0x00005637cf83a8ea in ThreadHTTP (base=0x7f04400450a0, http=<optimized out>) at httpserver.cpp:303
#4  0x00005637cf84265e in operator() (this=0x7f04300008c8) at /usr/include/c++/4.9/functional:2439
#5  operator() (this=0x7f04300008c0) at /usr/include/c++/4.9/future:1241
#6  std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<bool>, std::__future_base::_Result_base::_Deleter>, bool> >::_M_invoke(std::_Any_data const&) (__functor=...)
    at /usr/include/c++/4.9/functional:2025
#7  0x00005637cf842cf2 in operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:2439
#8  std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) (this=0x7f0440046630, __f=..., __set=@0x7f043ef5cdcf: false) at /usr/include/c++/4.9/future:485
#9  0x00007f045ec0d410 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:103
#10 0x00005637cf8437be in __gthread_once (__func=<optimized out>, __once=0x7f044004669c) at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:699
#11 call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>&, bool&), std::__future_base::_State_baseV2*, std::reference_wrapper<std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> >, std::reference_wrapper<bool> > (__f=<optimized out>, __once=...) at /usr/include/c++/4.9/mutex:746
#12 _M_set_result (__ignore_failure=false, __res=..., this=0x7f0440046630) at /usr/include/c++/4.9/future:372
#13 _M_run (__args#1=<optimized out>, __args#0=<optimized out>, this=0x7f0440046630) at /usr/include/c++/4.9/future:1319
#14 operator() (__args#1=<optimized out>, __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/4.9/future:1453
#15 _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#16 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#17 std::thread::_Impl<std::_Bind_simple<std::packaged_task<bool (event_base*, evhttp*)> (event_base*, evhttp*)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#18 0x00007f045db1a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x00007f045ec08064 in start_thread (arg=0x7f043ef5d700) at pthread_create.c:309
#20 0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7f043debb700 (LWP 28796)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f045db16c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005637cf83c7bb in Run (this=0x7f0440046080) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7f0440046080) at httpserver.cpp:351
#4  0x00007f045db1a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f045ec08064 in start_thread (arg=0x7f043debb700) at pthread_create.c:309
#6  0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7f0411af3700 (LWP 28807)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00005637cf9507b2 in __gthread_cond_timedwait (__abs_timeout=0x7f0411af2b60, __mutex=0x7f04405bd4d0, __cond=0x7f04405bd4a0)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7f04405bd4a0)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7f04405bd4a0)
    at /usr/include/c++/4.9/condition_variable:105
#4  wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>, this=0x7f04405bd4a0) at /usr/include/c++/4.9/condition_variable:128
#5  wait_for<long int, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __rtime=<synthetic pointer>, 
    __lock=<synthetic pointer>, this=0x7f04405bd4a0) at /usr/include/c++/4.9/condition_variable:144
#6  CThreadInterrupt::sleep_for (this=this@entry=0x7f04405bd4a0, rel_time=...) at threadinterrupt.cpp:30
#7  0x00005637cf950860 in CThreadInterrupt::sleep_for (this=this@entry=0x7f04405bd4a0, rel_time=...) at threadinterrupt.cpp:35
#8  0x00005637cf6b1997 in CConnman::ThreadOpenAddedConnections (this=0x7f044056d040) at net.cpp:1931
#9  0x00005637cf689d9d in operator() (this=0x7f0411af2e70) at /usr/include/c++/4.9/functional:2439
#10 TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x5637cfe14eae "addcon", func=...) at util.h:302
#11 0x00005637cf6ba9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#12 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#13 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#14 0x00007f045db1a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007f045ec08064 in start_thread (arg=0x7f0411af3700) at pthread_create.c:309
#16 0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7f0447821700 (LWP 28790)):
#0  0x00007f045d281aed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f045c3c6ee4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f045c3c6ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f045f6961cc in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#4  0x00007f045f63b512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007f045f3fbcb4 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007f045f400d2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007f045ec08064 in start_thread (arg=0x7f0447821700) at pthread_create.c:309
#8  0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7f043f7ae700 (LWP 28793)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00005637cf93170f in do_wait_until (timeout=..., m=..., this=0x5637d18663f8) at /usr/include/boost/thread/pthread/condition_variable.hpp:108
#2  wait_until (tp=..., lk=..., this=0x5637d18663f8) at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:253
#3  wait_until<boost::chrono::duration<long, boost::ratio<1l, 1000000000l> > > (t=<synthetic pointer>, lock=..., this=0x5637d18663f8)
    at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:166
#4  CScheduler::serviceQueue (this=0x5637d18663c8) at scheduler.cpp:67
#5  0x00005637cf689d9d in operator() (this=0x7f043f7ade40) at /usr/include/c++/4.9/functional:2439
#6  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x5637cff12f25 "scheduler", func=...) at util.h:302
#7  0x00005637cf688551 in operator()<void (*)(char const*, std::function<void()>), boost::_bi::list0> (f=<optimized out>, a=<synthetic pointer>, 
    this=<optimized out>) at /usr/include/boost/bind/bind.hpp:313
#8  operator() (this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:20
#9  boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > > >::run() (this=0x7f0440044050) at /usr/include/boost/thread/detail/thread.hpp:117
#10 0x00007f046121faea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#11 0x00007f045ec08064 in start_thread (arg=0x7f043f7ae700) at pthread_create.c:309
#12 0x00007f045d28a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f0461efd780 (LWP 28788)):
#0  0x00007f045d1d7067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f045d1d8448 in __GI_abort () at abort.c:89
#2  0x00007f045d2151b4 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7f045d30a210 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007f045d21a98e in malloc_printerr (action=1, str=0x7f045d30a3d0 "free(): invalid next size (fast)", ptr=<optimized out>) at malloc.c:4996
#4  0x00007f045d21b696 in _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
#5  0x00007f045f686f9c in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007f045f601cb0 in QSortFilterProxyModel::lessThan(QModelIndex const&, QModelIndex const&) const () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007f045f60fe69 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#8  0x00007f045f612d14 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#9  0x00007f045f614b84 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#10 0x00007f045f614ced in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#11 0x00007f045f6061ac in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#12 0x00007f045f60b6cd in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#13 0x00007f045f60ba95 in QSortFilterProxyModel::sort(int, Qt::SortOrder) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#14 0x00007f0460810ff7 in QTableView::setSortingEnabled(bool) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#15 0x00005637cf63ff6a in TransactionView::setModel (this=0x5637d1f39bb0, _model=_model@entry=0x5637d1baaca0) at qt/transactionview.cpp:212
#16 0x00005637cf5e0c3f in WalletView::setWalletModel (this=this@entry=0x5637d1eea430, _walletModel=0x5637d1baaca0) at qt/walletview.cpp:121
#17 0x00005637cf5cd5c5 in WalletFrame::addWallet (this=0x5637d16dd440, name=..., walletModel=<optimized out>) at qt/walletframe.cpp:49
#18 0x00005637cf5478b1 in BitcoinApplication::initializeResult (this=0x7ffe1fef1510, success=<optimized out>) at qt/bitcoin.cpp:492
#19 0x00007f045f66deda in QObject::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#20 0x00007f045fe036f2 in QGuiApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5
#21 0x00007f046059a18c in QApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#22 0x00007f046059712c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#23 0x00007f046059c600 in QApplication::notify(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#24 0x00007f045f63ddcb in QCoreApplication::notifyInternal(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#25 0x00007f045f63fdc3 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#26 0x00007f045f695da3 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#27 0x00007f045c3c6c5d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#28 0x00007f045c3c6f48 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#29 0x00007f045c3c6ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#30 0x00007f045f6961b7 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#31 0x00007f045f63b512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#32 0x00007f045f64315c in QCoreApplication::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#33 0x00005637cf53406e in main (argc=1, argv=<optimized out>) at qt/bitcoin.cpp:714
(gdb) 

Again, I'm happy to type commands at the (gdb) prompt if it would be helpful. Please tell me which core dump to load up. I'm calling this one core.bak2 for reference.

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

3rd crash, looks like the 1st one:

$ while [[ ! -e core ]]; do echo "$(date) : starting"; ~/Programs/bitcoin.git/src/qt/bitcoin-qt; done
Thu Aug 24 11:35:59 PDT 2017 : starting
got stop request
Thu Aug 24 11:36:09 PDT 2017 : starting
got stop request
Thu Aug 24 11:36:25 PDT 2017 : starting
got stop request
Thu Aug 24 11:36:39 PDT 2017 : starting
got stop request
Thu Aug 24 11:36:53 PDT 2017 : starting
got stop request
bitcoin-qt: /usr/include/boost/thread/pthread/recursive_mutex.hpp:101: boost::recursive_mutex::~recursive_mutex(): Assertion `!pthread_mutex_destroy(&m)' failed.
Aborted (core dumped)
$ cp core core.bak3
$ gdb qt/bitcoin-qt core.bak3
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from qt/bitcoin-qt...done.
[New LWP 29158]
[New LWP 29169]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f03e28f9067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) thread apply all where

Thread 2 (Thread 0x7f03b298b700 (LWP 29169)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055b001721d8b in BGThread (this=0x7f03c842a290) at leveldb/util/env_posix.cc:641
#2  leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7f03c842a290) at leveldb/util/env_posix.cc:556
#3  0x00007f03e432a064 in start_thread (arg=0x7f03b298b700) at pthread_create.c:309
#4  0x00007f03e29ac62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f03e761f780 (LWP 29158)):
#0  0x00007f03e28f9067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f03e28fa448 in __GI_abort () at abort.c:89
#2  0x00007f03e28f2266 in __assert_fail_base (fmt=0x7f03e2a2af18 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x55b001b5ba60 "!pthread_mutex_destroy(&m)", file=file@entry=0x55b001754700 "/usr/include/boost/thread/pthread/recursive_mutex.hpp", 
    line=line@entry=101, 
    function=function@entry=0x55b001c7f0e0 <_ZZN5boost15recursive_mutexD4EvE19__PRETTY_FUNCTION__> "boost::recursive_mutex::~recursive_mutex()") at assert.c:92
#3  0x00007f03e28f2312 in __GI___assert_fail (assertion=0x55b001b5ba60 "!pthread_mutex_destroy(&m)", 
    file=0x55b001754700 "/usr/include/boost/thread/pthread/recursive_mutex.hpp", line=101, 
    function=0x55b001c7f0e0 <_ZZN5boost15recursive_mutexD4EvE19__PRETTY_FUNCTION__> "boost::recursive_mutex::~recursive_mutex()") at assert.c:101
#4  0x000055b0012914a1 in boost::recursive_mutex::~recursive_mutex (this=0x55b002c0b740, __in_chrg=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:101
#5  0x000055b0016b4dc4 in ~recursive_mutex (this=0x55b002c0b740, __in_chrg=<optimized out>) at util.cpp:150
#6  ~AnnotatedMixin (this=0x55b002c0b740, __in_chrg=<optimized out>) at sync.h:54
#7  ~CCriticalSection (this=0x55b002c0b740, __in_chrg=<optimized out>) at sync.h:94
#8  operator() (this=<optimized out>, __ptr=0x55b002c0b448) at /usr/include/c++/4.9/bits/unique_ptr.h:119
#9  reset (__p=0x55b002c0b448, this=0x55b001f9b7a0 <ppmutexOpenSSL>) at /usr/include/c++/4.9/bits/unique_ptr.h:581
#10 CInit::~CInit (this=<optimized out>, __in_chrg=<optimized out>) at util.cpp:149
#11 0x00007f03e28fbb29 in __run_exit_handlers (status=0, listp=0x7f03e2c695a8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#12 0x00007f03e28fbb75 in __GI_exit (status=<optimized out>) at exit.c:104
#13 0x00007f03e28e5b4c in __libc_start_main (main=0x55b0012924c0 <main(int, char**)>, argc=1, argv=0x7fffa51fa1e8, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffa51fa1d8) at libc-start.c:321
#14 0x000055b0012a3a38 in _start ()
(gdb) 
@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

I saw a 4th crash. This one took a lot of attempts before it happened, but the error message on crash is a new one:

$ while [[ ! -e core ]]; do echo "$(date) : starting"; ~/Programs/bitcoin.git/src/qt/bitcoin-qt; done
Thu Aug 24 11:40:35 PDT 2017 : starting
got stop request
Thu Aug 24 11:40:53 PDT 2017 : starting
got stop request
[...]
Thu Aug 24 12:10:24 PDT 2017 : starting
got stop request
Thu Aug 24 12:10:39 PDT 2017 : starting
got stop request
Thu Aug 24 12:10:55 PDT 2017 : starting
*** Error in `/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt': malloc(): smallbin double linked list corrupted: 0x000055e3f85adbe0 ***
Aborted (core dumped)
$ cp core core.bak4
$ gdb qt/bitcoin-qt core.bak4
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from qt/bitcoin-qt...done.
[New LWP 12656]
[New LWP 12661]
[New LWP 12657]
[New LWP 12662]
[New LWP 12664]
[New LWP 12660]
[New LWP 12676]
[New LWP 12672]
[New LWP 12677]
[New LWP 12678]
[New LWP 12680]
[New LWP 12675]
[New LWP 12679]
[New LWP 12681]
[New LWP 12663]
[New LWP 12665]
[New LWP 12666]
[New LWP 12667]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f18ba82a067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) thread apply all where

Thread 18 (Thread 0x7f189670c700 (LWP 12667)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f18bb169c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x000055e3f4f967bb in Run (this=0x7f18a0046080) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7f18a0046080) at httpserver.cpp:351
#4  0x00007f18bb16d970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f18bc25b064 in start_thread (arg=0x7f189670c700) at pthread_create.c:309
#6  0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 17 (Thread 0x7f1896f5d700 (LWP 12666)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f18bb169c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x000055e3f4f967bb in Run (this=0x7f18a0046080) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7f18a0046080) at httpserver.cpp:351
#4  0x00007f18bb16d970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f18bc25b064 in start_thread (arg=0x7f1896f5d700) at pthread_create.c:309
#6  0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 16 (Thread 0x7f18977ae700 (LWP 12665)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f18bb169c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x000055e3f4f967bb in Run (this=0x7f18a0046080) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7f18a0046080) at httpserver.cpp:351
#4  0x00007f18bb16d970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f18bc25b064 in start_thread (arg=0x7f18977ae700) at pthread_create.c:309
#6  0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 15 (Thread 0x7f189ccb9700 (LWP 12663)):
#0  0x00007f18ba8ddc03 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f18bb3e6b08 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#2  0x00007f18bb3d1d5a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#3  0x000055e3f4f948ea in ThreadHTTP (base=0x7f18a00450a0, http=<optimized out>) at httpserver.cpp:303
#4  0x000055e3f4f9c65e in operator() (this=0x7f188c0008c8) at /usr/include/c++/4.9/functional:2439
#5  operator() (this=0x7f188c0008c0) at /usr/include/c++/4.9/future:1241
#6  std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<bool>, std::__future_base::_Result_base::_Deleter>, bool> >::_M_invoke(std::_Any_data const&) (__functor=...)
    at /usr/include/c++/4.9/functional:2025
#7  0x000055e3f4f9ccf2 in operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:2439
#8  std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) (this=0x7f18a0046630, __f=..., __set=@0x7f189ccb8dcf: false) at /usr/include/c++/4.9/future:485
#9  0x00007f18bc260410 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:103
#10 0x000055e3f4f9d7be in __gthread_once (__func=<optimized out>, __once=0x7f18a004669c) at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:699
#11 call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>&, bool&), std::__future_base::_State_baseV2*, std::reference_wrapper<std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> >, std::reference_wrapper<bool> > (__f=<optimized out>, __once=...) at /usr/include/c++/4.9/mutex:746
#12 _M_set_result (__ignore_failure=false, __res=..., this=0x7f18a0046630) at /usr/include/c++/4.9/future:372
#13 _M_run (__args#1=<optimized out>, __args#0=<optimized out>, this=0x7f18a0046630) at /usr/include/c++/4.9/future:1319
#14 operator() (__args#1=<optimized out>, __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/4.9/future:1453
#15 _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#16 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#17 std::thread::_Impl<std::_Bind_simple<std::packaged_task<bool (event_base*, evhttp*)> (event_base*, evhttp*)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#18 0x00007f18bb16d970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x00007f18bc25b064 in start_thread (arg=0x7f189ccb9700) at pthread_create.c:309
#20 0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 14 (Thread 0x7f186d02e700 (LWP 12681)):
#0  0x00007f18ba8d4aed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f18b9a19ee4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f18b9a19ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f18bcce91cc in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#4  0x00007f18bcc8e512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007f18bca4ecb4 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007f18bca53d2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007f18bc25b064 in start_thread (arg=0x7f186d02e700) at pthread_create.c:309
#8  0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 13 (Thread 0x7f186e4f8700 (LWP 12679)):
#0  0x00007f18ba8d6893 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x000055e3f5085f29 in ConnectSocketDirectly (addrConnect=..., hSocketRet=@0x7f186e4f761c: 4294967295, nTimeout=nTimeout@entry=5000) at netbase.cpp:442
#2  0x000055e3f5087a96 in ConnectSocket (addrDest=..., hSocketRet=@0x7f186e4f761c: 4294967295, nTimeout=5000, 
    outProxyConnectionFailed=outProxyConnectionFailed@entry=0x7f186e4f7619) at netbase.cpp:571
#3  0x000055e3f4e0591a in CConnman::ConnectNode (this=this@entry=0x7f18a056fff0, addrConnect=..., pszDest=0x0, fCountFailure=<optimized out>) at net.cpp:390
#4  0x000055e3f4e09c42 in CConnman::OpenNetworkConnection (this=this@entry=0x7f18a056fff0, addrConnect=..., fCountFailure=<optimized out>, 
    grantOutbound=grantOutbound@entry=0x7f186e4f7b70, pszDest=pszDest@entry=0x0, fOneShot=fOneShot@entry=false, fFeeler=false, fAddnode=false) at net.cpp:1956
#5  0x000055e3f4e0aa83 in CConnman::ThreadOpenConnections (this=0x7f18a056fff0) at net.cpp:1844
#6  0x000055e3f4de3d9d in operator() (this=0x7f186e4f7e70) at /usr/include/c++/4.9/functional:2439
#7  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55e3f556eeb5 "opencon", func=...) at util.h:302
#8  0x000055e3f4e149cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#9  operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#10 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#11 0x00007f18bb16d970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007f18bc25b064 in start_thread (arg=0x7f186e4f8700) at pthread_create.c:309
#13 0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 12 (Thread 0x7f187588a700 (LWP 12675)):
#0  Solver (scriptPubKey=..., typeRet=@0x7f187588813c: 3798923336, vSolutionsRet=std::vector of length 0, capacity 0) at script/standard.cpp:155
#1  0x000055e3f4ed89b3 in IsMine (keystore=..., scriptPubKey=..., isInvalid=isInvalid@entry=@0x7f18758883b7: false, sigversion=sigversion@entry=SIGVERSION_BASE)
    at script/ismine.cpp:51
#2  0x000055e3f4ed96a8 in IsMine (keystore=..., scriptPubKey=..., sigversion=sigversion@entry=SIGVERSION_BASE) at script/ismine.cpp:32
#3  0x000055e3f4fed987 in IsMine (txout=..., this=0x7f1878404a10) at wallet/wallet.cpp:1300
#4  CWallet::IsMine (this=this@entry=0x7f1878404a10, tx=...) at wallet/wallet.cpp:1342
#5  0x000055e3f500f00b in CWallet::AddToWalletIfInvolvingMe (this=this@entry=0x7f1878404a10, ptx=std::shared_ptr (count 2, weak 0) 0x7f18683f0020, 
    pIndex=pIndex@entry=0x0, posInBlock=posInBlock@entry=0, fUpdate=fUpdate@entry=true) at wallet/wallet.cpp:1055
#6  0x000055e3f500fa6b in CWallet::SyncTransaction (this=this@entry=0x7f1878404a10, ptx=std::shared_ptr (count 2, weak 0) 0x7f18683f0020, pindex=pindex@entry=0x0, 
    posInBlock=posInBlock@entry=0) at wallet/wallet.cpp:1219
#7  0x000055e3f500fb99 in CWallet::TransactionAddedToMempool (this=0x7f1878404a10, ptx=std::shared_ptr (count 2, weak 0) 0x7f18683f0020) at wallet/wallet.cpp:1234
#8  0x000055e3f4f74eec in operator() (a0=..., this=<optimized out>) at /usr/include/boost/function/function_template.hpp:767
#9  m_invoke<boost::function<void(const std::shared_ptr<const CTransaction>&)>, 0u, const std::shared_ptr<const CTransaction>&> (args=..., func=..., 
    this=<optimized out>) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:92
#10 operator()<boost::function<void(const std::shared_ptr<const CTransaction>&)>, const std::shared_ptr<const CTransaction>&, 1ul> (args=..., func=..., 
    this=<optimized out>) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:81
#11 m_invoke<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void(const std::shared_ptr<const CTransaction>&), boost::function<void(const std::shared_ptr<const CTransaction>&)> >, boost::signals2::mutex> > > (
    connectionBody=..., this=<optimized out>) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:117
#12 operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void(const std::shared_ptr<const CTransaction>&), boost::function<void(const std::shared_ptr<const CTransaction>&)> >, boost::signals2::mutex> > > (
    connectionBody=..., this=<optimized out>) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:110
#13 dereference (this=0x7f18758889f0) at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:82
#14 dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, const std::shared_ptr<const CTransaction>&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void(const std::shared_ptr<const CTransaction>&), boost::function<void(const std::shared_ptr<const CTransaction>&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void(const std::shared_ptr<const CTransaction>&), boost::function<void(const std::shared_ptr<const CTransaction>&)> >, boost::signals2::mutex> > > (f=...)
    at /usr/include/boost/iterator/iterator_facade.hpp:514
#15 operator* (this=0x7f18758889f0) at /usr/include/boost/iterator/iterator_facade.hpp:639
#16 operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, const std::shared_ptr<const CTransaction>&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void(const std::shared_ptr<const CTransaction>&), boost::function<void(const std::shared_ptr<const CTransaction>&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void(const std::shared_ptr<const CTransaction>&), boost::function<void(const std::shared_ptr<const CTransaction>&)> >, boost::signals2::mutex> > > (last=..., first=..., 
    this=<optimized out>) at /usr/include/boost/signals2/optional_last_value.hpp:55
#17 operator()<boost::signals2::optional_last_value<void>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, const std::shared_ptr<const CTransaction>&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void(const std::shared_ptr<const CTransaction>&), boost::function<void(const std::shared_ptr<const CTransaction>&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void(const std::shared_ptr<const CTransaction>&), boost::function<void(const std::shared_ptr<const CTransaction>&)> >, boost::signals2::mutex> > > (last=..., first=..., combiner=..., this=<optimized out>) at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:64
#18 boost::signals2::detail::signal_impl<void (std::shared_ptr<CTransaction const> const&), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (std::shared_ptr<CTransaction const> const&)>, boost::function<void (boost::signals2::connection const&, std::shared_ptr<CTransaction const> const&)>, boost::signals2::mutex>::operator()(std::shared_ptr<CTransaction const> const&) (this=<optimized out>, args#0=std::shared_ptr (count 2, weak 0) 0x7f18683f0020)
    at /usr/include/boost/signals2/detail/signal_template.hpp:246
#19 0x000055e3f4f5c109 in operator() (args#0=std::shared_ptr (count 2, weak 0) 0x7f18683f0020, this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:695
#20 CMainSignals::TransactionAddedToMempool (this=<optimized out>, ptx=std::shared_ptr (count 2, weak 0) 0x7f18683f0020) at validationinterface.cpp:97
#21 0x000055e3f4f3e09e in AcceptToMemoryPoolWorker (chainparams=..., pool=..., state=..., ptx=std::shared_ptr (count 2, weak 0) 0x7f18683f0020, 
    fLimitFree=fLimitFree@entry=true, pfMissingInputs=pfMissingInputs@entry=0x0, nAcceptTime=1503593306, plTxnReplaced=0x0, fOverrideMempoolLimit=false, 
    nAbsurdFee=@0x7f1875889668: 0, coins_to_uncache=std::vector of length 2, capacity 2 = {...}) at validation.cpp:869
#22 0x000055e3f4f3ed5f in AcceptToMemoryPoolWithTime (chainparams=..., pool=..., state=..., tx=std::shared_ptr (count 2, weak 0) 0x7f18683f0020, 
    fLimitFree=fLimitFree@entry=true, pfMissingInputs=pfMissingInputs@entry=0x0, nAcceptTime=1503593306, plTxnReplaced=0x0, fOverrideMempoolLimit=false, 
    nAbsurdFee=0) at validation.cpp:880
#23 0x000055e3f4f46648 in LoadMempool () at validation.cpp:4302
#24 0x000055e3f4dbefb7 in ThreadImport (vImportFiles=std::vector of length 0, capacity 0) at init.cpp:692
#25 0x000055e3f4de8cca in operator()<void (*)(std::vector<boost::filesystem::path>), boost::_bi::list0> (f=<optimized out>, a=<synthetic pointer>, 
    this=<optimized out>) at /usr/include/boost/bind/bind.hpp:253
#26 operator() (this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:20
#27 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >), boost::_bi::list1<boost::_bi::value<std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> > > > > >::run (this=<optimized out>)
    at /usr/include/boost/thread/detail/thread.hpp:117
#28 0x00007f18be872aea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#29 0x00007f18bc25b064 in start_thread (arg=0x7f187588a700) at pthread_create.c:309
#30 0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 11 (Thread 0x7f186dca7700 (LWP 12680)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x000055e3f4e0bf15 in __gthread_cond_timedwait (__abs_timeout=0x7f186dca6ca0, __mutex=<optimized out>, __cond=0x7f18a05c03f0)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>, __lock=..., this=0x7f18a05c03f0)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x7f18a05c03f0)
    at /usr/include/c++/4.9/condition_variable:118
#4  wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CConnman::ThreadMessageHandler()::<lambda()> > (
    __p=..., __atime=..., __lock=..., this=0x7f18a05c03f0) at /usr/include/c++/4.9/condition_variable:128
#5  CConnman::ThreadMessageHandler (this=0x7f18a056fff0) at net.cpp:2021
#6  0x000055e3f4de3d9d in operator() (this=0x7f186dca6e70) at /usr/include/c++/4.9/functional:2439
#7  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55e3f556eebd "msghand", func=...) at util.h:302
#8  0x000055e3f4e149cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#9  operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#10 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#11 0x00007f18bb16d970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007f18bc25b064 in start_thread (arg=0x7f186dca7700) at pthread_create.c:309
#13 0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 10 (Thread 0x7f186ed49700 (LWP 12678)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x000055e3f50aa7b2 in __gthread_cond_timedwait (__abs_timeout=0x7f186ed48b60, __mutex=0x7f18a05c0480, __cond=0x7f18a05c0450)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7f18a05c0450)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7f18a05c0450)
    at /usr/include/c++/4.9/condition_variable:105
#4  wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>, this=0x7f18a05c0450) at /usr/include/c++/4.9/condition_variable:128
#5  wait_for<long int, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __rtime=<synthetic pointer>, 
    __lock=<synthetic pointer>, this=0x7f18a05c0450) at /usr/include/c++/4.9/condition_variable:144
#6  CThreadInterrupt::sleep_for (this=this@entry=0x7f18a05c0450, rel_time=...) at threadinterrupt.cpp:30
#7  0x000055e3f50aa860 in CThreadInterrupt::sleep_for (this=this@entry=0x7f18a05c0450, rel_time=...) at threadinterrupt.cpp:35
#8  0x000055e3f4e0b997 in CConnman::ThreadOpenAddedConnections (this=0x7f18a056fff0) at net.cpp:1931
#9  0x000055e3f4de3d9d in operator() (this=0x7f186ed48e70) at /usr/include/c++/4.9/functional:2439
#10 TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55e3f556eeae "addcon", func=...) at util.h:302
#11 0x000055e3f4e149cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#12 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#13 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#14 0x00007f18bb16d970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007f18bc25b064 in start_thread (arg=0x7f186ed49700) at pthread_create.c:309
#16 0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 9 (Thread 0x7f186f59a700 (LWP 12677)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x000055e3f50aa7b2 in __gthread_cond_timedwait (__abs_timeout=0x7f186f599970, __mutex=0x7f18a05c0480, __cond=0x7f18a05c0450)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7f18a05c0450)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7f18a05c0450)
    at /usr/include/c++/4.9/condition_variable:105
#4  wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>, this=0x7f18a05c0450) at /usr/include/c++/4.9/condition_variable:128
#5  wait_for<long int, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __rtime=<synthetic pointer>, 
    __lock=<synthetic pointer>, this=0x7f18a05c0450) at /usr/include/c++/4.9/condition_variable:144
#6  CThreadInterrupt::sleep_for (this=this@entry=0x7f18a05c0450, rel_time=...) at threadinterrupt.cpp:30
#7  0x000055e3f50aa860 in CThreadInterrupt::sleep_for (this=this@entry=0x7f18a05c0450, rel_time=..., rel_time@entry=...) at threadinterrupt.cpp:35
#8  0x000055e3f4dfd647 in CConnman::ThreadDNSAddressSeed (this=0x7f18a056fff0) at net.cpp:1579
#9  0x000055e3f4de3d9d in operator() (this=0x7f186f599e70) at /usr/include/c++/4.9/functional:2439
#10 TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55e3f5556b33 "dnsseed", func=...) at util.h:302
#11 0x000055e3f4e149cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#12 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#13 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#14 0x00007f18bb16d970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007f18bc25b064 in start_thread (arg=0x7f186f59a700) at pthread_create.c:309
#16 0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 8 (Thread 0x7f18899ff700 (LWP 12672)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055e3f511cd8b in BGThread (this=0x7f18a03e48c0) at leveldb/util/env_posix.cc:641
#2  leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7f18a03e48c0) at leveldb/util/env_posix.cc:556
#3  0x00007f18bc25b064 in start_thread (arg=0x7f18899ff700) at pthread_create.c:309
#4  0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 7 (Thread 0x7f186fdeb700 (LWP 12676)):
#0  0x00007f18ba8d6893 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x000055e3f4e10df5 in CConnman::ThreadSocketHandler (this=<optimized out>) at net.cpp:1256
#2  0x000055e3f4de3d9d in operator() (this=0x7f186fdeae70) at /usr/include/c++/4.9/functional:2439
#3  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55e3f55a1d31 "net", func=...) at util.h:302
#4  0x000055e3f4e149cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#5  operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#6  std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#7  0x00007f18bb16d970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007f18bc25b064 in start_thread (arg=0x7f186fdeb700) at pthread_create.c:309
#9  0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 6 (Thread 0x7f18a4e96700 (LWP 12660)):
#0  0x00007f18ba8d4aed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f18b9a19ee4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f18b9a19ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f18bcce91cc in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#4  0x00007f18bcc8e512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007f18bca4ecb4 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007f18bca53d2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007f18bc25b064 in start_thread (arg=0x7f18a4e96700) at pthread_create.c:309
#8  0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7f1897fff700 (LWP 12664)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f18bb169c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x000055e3f4f967bb in Run (this=0x7f18a0046080) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7f18a0046080) at httpserver.cpp:351
#4  0x00007f18bb16d970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f18bc25b064 in start_thread (arg=0x7f1897fff700) at pthread_create.c:309
#6  0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7f189d50a700 (LWP 12662)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x000055e3f508b70f in do_wait_until (timeout=..., m=..., this=0x55e3f7d283d8) at /usr/include/boost/thread/pthread/condition_variable.hpp:108
#2  wait_until (tp=..., lk=..., this=0x55e3f7d283d8) at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:253
#3  wait_until<boost::chrono::duration<long, boost::ratio<1l, 1000000000l> > > (t=<synthetic pointer>, lock=..., this=0x55e3f7d283d8)
    at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:166
#4  CScheduler::serviceQueue (this=0x55e3f7d283a8) at scheduler.cpp:67
#5  0x000055e3f4de3d9d in operator() (this=0x7f189d509e40) at /usr/include/c++/4.9/functional:2439
#6  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55e3f566cf25 "scheduler", func=...) at util.h:302
#7  0x000055e3f4de2551 in operator()<void (*)(char const*, std::function<void()>), boost::_bi::list0> (f=<optimized out>, a=<synthetic pointer>, 
    this=<optimized out>) at /usr/include/boost/bind/bind.hpp:313
#8  operator() (this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:20
#9  boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > > >::run() (this=0x7f18a0044050) at /usr/include/boost/thread/detail/thread.hpp:117
#10 0x00007f18be872aea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#11 0x00007f18bc25b064 in start_thread (arg=0x7f189d50a700) at pthread_create.c:309
#12 0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7f18b12f9700 (LWP 12657)):
#0  0x00007f18ba8d4aed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f18b7ade252 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#2  0x00007f18b7adfddf in xcb_wait_for_event () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#3  0x00007f18b3a4c1b9 in ?? () from /usr/lib/x86_64-linux-gnu/qt5/plugins/platforms/libqxcb.so
#4  0x00007f18bca53d2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007f18bc25b064 in start_thread (arg=0x7f18b12f9700) at pthread_create.c:309
#6  0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7f189dd5b700 (LWP 12661)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055e3f4ca94cb in boost::condition_variable::wait (this=0x55e3f5993e68 <scriptcheckqueue+40>, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
#2  0x000055e3f4f52bb0 in CCheckQueue<CScriptCheck>::Loop (this=0x55e3f5993e40 <scriptcheckqueue>, fMaster=false) at checkqueue.h:102
#3  0x00007f18be872aea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#4  0x00007f18bc25b064 in start_thread (arg=0x7f189dd5b700) at pthread_create.c:309
#5  0x00007f18ba8dd62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f18bf550780 (LWP 12656)):
#0  0x00007f18ba82a067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f18ba82b448 in __GI_abort () at abort.c:89
#2  0x00007f18ba8681b4 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7f18ba95d210 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007f18ba86d98e in malloc_printerr (action=1, str=0x7f18ba95d5d8 "malloc(): smallbin double linked list corrupted", ptr=<optimized out>) at malloc.c:4996
#4  0x00007f18ba86fd2b in _int_malloc (av=0x7f18bab9a620 <main_arena>, bytes=24) at malloc.c:3359
#5  0x00007f18ba871020 in __GI___libc_malloc (bytes=24) at malloc.c:2891
#6  0x00007f18bb1162e8 in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00007f18bca5e22a in QCollator::QCollator(QLocale const&) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#8  0x00007f18bcaf1d2d in QString::localeAwareCompare_helper(QChar const*, int, QChar const*, int) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#9  0x00007f18bcc54e9e in QSortFilterProxyModel::lessThan(QModelIndex const&, QModelIndex const&) const () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#10 0x00007f18bcc62e69 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#11 0x00007f18bcc65d14 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#12 0x00007f18bcc67b84 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#13 0x00007f18bcc67ced in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#14 0x00007f18bcc591ac in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#15 0x00007f18bcc5e6cd in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#16 0x00007f18bcc5ea95 in QSortFilterProxyModel::sort(int, Qt::SortOrder) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#17 0x00007f18bde63ff7 in QTableView::setSortingEnabled(bool) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#18 0x000055e3f4d99f6a in TransactionView::setModel (this=0x55e3f83fb510, _model=_model@entry=0x55e3f80d2d70) at qt/transactionview.cpp:212
#19 0x000055e3f4d3ac3f in WalletView::setWalletModel (this=this@entry=0x55e3f83abd90, _walletModel=0x55e3f80d2d70) at qt/walletview.cpp:121
#20 0x000055e3f4d275c5 in WalletFrame::addWallet (this=0x55e3f7b9f370, name=..., walletModel=<optimized out>) at qt/walletframe.cpp:49
#21 0x000055e3f4ca18b1 in BitcoinApplication::initializeResult (this=0x7ffe61db0de0, success=<optimized out>) at qt/bitcoin.cpp:492
#22 0x00007f18bccc0eda in QObject::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#23 0x00007f18bd4566f2 in QGuiApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5
#24 0x00007f18bdbed18c in QApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#25 0x00007f18bdbea12c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#26 0x00007f18bdbef600 in QApplication::notify(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#27 0x00007f18bcc90dcb in QCoreApplication::notifyInternal(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#28 0x00007f18bcc92dc3 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#29 0x00007f18bcce8da3 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#30 0x00007f18b9a19c5d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#31 0x00007f18b9a19f48 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#32 0x00007f18b9a19ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#33 0x00007f18bcce91b7 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#34 0x00007f18bcc8e512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#35 0x00007f18bcc9615c in QCoreApplication::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#36 0x000055e3f4c8e06e in main (argc=1, argv=<optimized out>) at qt/bitcoin.cpp:714
(gdb) 
@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

So I've provided 5 sets of stack traces.

  1. bitcoin-qt: .../pthread/recursive_mutex.hpp:101: boost::recursive_mutex::~recursive_mutex(): Assertion '!pthread_mutex_destroy(&m)' failed.
  2. *** Error in '.../qt/bitcoin-qt': free(): invalid next size (fast): 0x00005637d209d7b0 ***
  3. bitcoin-qt: .../pthread/recursive_mutex.hpp:101: boost::recursive_mutex::~recursive_mutex(): Assertion '!pthread_mutex_destroy(&m)' failed.
  4. *** Error in '.../qt/bitcoin-qt': malloc(): smallbin double linked list corrupted: 0x000055e3f85adbe0 ***
  5. *** Error in '.../qt/bitcoin-qt': free(): invalid next size (fast): 0x000055fb4062dbb0 ***

The first and third appear to be exactly the same issue, as do the second and fifth. The forth different.

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

The very next time I tried running bitcoin-qt, I got a crash again. I don't know if something had been left in an inconsistent state by the previous crash, but even if so I would hope for something better than a core dump.

Here's the fifth crash. It has the same kind of error message as the 2nd one:

$ ~/Programs/bitcoin.git/src/qt/bitcoin-qt  -datadir=/home/chris/.bitcoin
*** Error in `/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt': free(): invalid next size (fast): 0x000055fb4062dbb0 ***
Aborted (core dumped)
$ cp core core.bak5
$ gdb qt/bitcoin-qt core.bak5
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from qt/bitcoin-qt...done.
[New LWP 12935]
[New LWP 12938]
[New LWP 12942]
[New LWP 12941]
[New LWP 12940]
[New LWP 12945]
[New LWP 12943]
[New LWP 12946]
[New LWP 12958]
[New LWP 12957]
[New LWP 12949]
[New LWP 12959]
[New LWP 12960]
[New LWP 12955]
[New LWP 12954]
[New LWP 12939]
[New LWP 12944]
[New LWP 12956]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt -datadir=/home/chris/.bitcoi'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007ff897444067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) thread apply all where

Thread 18 (Thread 0x7ff8477ae700 (LWP 12956)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x000055fb3e2247b2 in __gthread_cond_timedwait (__abs_timeout=0x7ff8477ad970, __mutex=0x7ff87c5c4020, __cond=0x7ff87c5c3ff0)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7ff87c5c3ff0)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7ff87c5c3ff0)
    at /usr/include/c++/4.9/condition_variable:105
#4  wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>, this=0x7ff87c5c3ff0) at /usr/include/c++/4.9/condition_variable:128
#5  wait_for<long int, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __rtime=<synthetic pointer>, 
    __lock=<synthetic pointer>, this=0x7ff87c5c3ff0) at /usr/include/c++/4.9/condition_variable:144
#6  CThreadInterrupt::sleep_for (this=this@entry=0x7ff87c5c3ff0, rel_time=...) at threadinterrupt.cpp:30
#7  0x000055fb3e224860 in CThreadInterrupt::sleep_for (this=this@entry=0x7ff87c5c3ff0, rel_time=..., rel_time@entry=...) at threadinterrupt.cpp:35
#8  0x000055fb3df77647 in CConnman::ThreadDNSAddressSeed (this=0x7ff87c573b90) at net.cpp:1579
#9  0x000055fb3df5dd9d in operator() (this=0x7ff8477ade70) at /usr/include/c++/4.9/functional:2439
#10 TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55fb3e6d0b33 "dnsseed", func=...) at util.h:302
#11 0x000055fb3df8e9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#12 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#13 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#14 0x00007ff897d87970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007ff898e75064 in start_thread (arg=0x7ff8477ae700) at pthread_create.c:309
#16 0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 17 (Thread 0x7ff86bfff700 (LWP 12944)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff897d83c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x000055fb3e1107bb in Run (this=0x7ff87c046050) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7ff87c046050) at httpserver.cpp:351
#4  0x00007ff897d87970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ff898e75064 in start_thread (arg=0x7ff86bfff700) at pthread_create.c:309
#6  0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 16 (Thread 0x7ff881a50700 (LWP 12939)):
#0  0x00007ff8974eeaed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff896633ee4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007ff896633ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ff8999031cc in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#4  0x00007ff8998a8512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ff899668cb4 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ff89966dd2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007ff898e75064 in start_thread (arg=0x7ff881a50700) at pthread_create.c:309
#8  0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 15 (Thread 0x7ff84e2ed700 (LWP 12954)):
#0  0x000055fb3e2a4bd2 in secp256k1_fe_sqr_inner (r=r@entry=0x7ff84e2ea130, a=0xbdf5ce1072cdd, a@entry=0x7ff84e2eb080) at src/field_5x52_asm_impl.h:298
#1  0x000055fb3e2b050b in secp256k1_fe_sqr (a=0x7ff84e2eb080, r=0x7ff84e2ea130) at src/field_5x52_impl.h:439
#2  secp256k1_gej_double_var (rzr=0x0, a=0x7ff84e2eb080, r=0x7ff84e2eb080) at src/group_impl.h:336
#3  secp256k1_ecmult (ctx=ctx@entry=0x55fb3fe2b4c0, r=r@entry=0x7ff84e2eb080, a=a@entry=0x7ff84e2eb000, na=na@entry=0x7ff84e2eaf50, ng=ng@entry=0x7ff84e2eaf30)
    at src/ecmult_impl.h:371
#4  0x000055fb3e2b3424 in secp256k1_ecdsa_sig_verify (message=0x7ff84e2eaed0, pubkey=0x7ff84e2eafa0, sigs=0x7ff84e2eaeb0, sigr=0x7ff84e2eae90, ctx=0x55fb3fe2b4c0)
    at src/ecdsa_impl.h:220
#5  secp256k1_ecdsa_verify (ctx=0x55fb3fe2b4c0, sig=<optimized out>, msg32=<optimized out>, pubkey=<optimized out>) at src/secp256k1.c:307
#6  0x000055fb3e23cd1c in CPubKey::Verify (this=this@entry=0x7ff84e2eb360, hash=..., vchSig=std::vector of length 70, capacity 71 = {...}) at pubkey.cpp:181
#7  0x000055fb3e23dae6 in TransactionSignatureChecker::VerifySignature (this=this@entry=0x7ff84e2eb970, vchSig=std::vector of length 70, capacity 71 = {...}, 
    pubkey=..., sighash=...) at script/interpreter.cpp:1248
#8  0x000055fb3e051255 in CachingTransactionSignatureChecker::VerifySignature (this=0x7ff84e2eb970, vchSig=std::vector of length 70, capacity 71 = {...}, 
    pubkey=..., sighash=...) at script/sigcache.cpp:89
#9  0x000055fb3e2456b2 in TransactionSignatureChecker::CheckSig (this=0x7ff84e2eb970, vchSigIn=..., vchPubKey=..., scriptCode=..., sigversion=<optimized out>)
    at script/interpreter.cpp:1266
#10 0x000055fb3e242aa8 in EvalScript (stack=std::vector of length 2, capacity 4 = {...}, script=..., flags=flags@entry=65503, checker=..., 
    sigversion=sigversion@entry=SIGVERSION_BASE, serror=serror@entry=0x7ff84e2eba9c) at script/interpreter.cpp:900
#11 0x000055fb3e2462d5 in VerifyScript (scriptSig=..., scriptPubKey=..., witness=0x7ff8482134f8, flags=65503, checker=..., serror=serror@entry=0x7ff84e2eba9c)
    at script/interpreter.cpp:1427
#12 0x000055fb3e0968d7 in CScriptCheck::operator() (this=this@entry=0x7ff84e2eba60) at validation.cpp:1202
#13 0x000055fb3e09c052 in CheckInputs (tx=..., state=..., inputs=..., fScriptChecks=fScriptChecks@entry=true, flags=flags@entry=65503, 
    cacheSigStore=cacheSigStore@entry=true, cacheFullScriptStore=false, txdata=..., pvChecks=0x0) at validation.cpp:1292
#14 0x000055fb3e0b5d1e in AcceptToMemoryPoolWorker (chainparams=..., pool=..., state=..., ptx=std::shared_ptr (count 2, weak 0) 0x7ff8483c94b0, 
    fLimitFree=fLimitFree@entry=true, pfMissingInputs=pfMissingInputs@entry=0x0, nAcceptTime=1503591734, plTxnReplaced=0x0, fOverrideMempoolLimit=false, 
    nAbsurdFee=@0x7ff84e2ec668: 0, coins_to_uncache=std::vector of length 2, capacity 2 = {...}) at validation.cpp:792
#15 0x000055fb3e0b8d5f in AcceptToMemoryPoolWithTime (chainparams=..., pool=..., state=..., tx=std::shared_ptr (count 2, weak 0) 0x7ff8483c94b0, 
    fLimitFree=fLimitFree@entry=true, pfMissingInputs=pfMissingInputs@entry=0x0, nAcceptTime=1503591734, plTxnReplaced=0x0, fOverrideMempoolLimit=false, 
    nAbsurdFee=0) at validation.cpp:880
#16 0x000055fb3e0c0648 in LoadMempool () at validation.cpp:4302
#17 0x000055fb3df38fb7 in ThreadImport (vImportFiles=std::vector of length 0, capacity 0) at init.cpp:692
#18 0x000055fb3df62cca in operator()<void (*)(std::vector<boost::filesystem::path>), boost::_bi::list0> (f=<optimized out>, a=<synthetic pointer>, 
    this=<optimized out>) at /usr/include/boost/bind/bind.hpp:253
#19 operator() (this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:20
#20 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >), boost::_bi::list1<boost::_bi::value<std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> > > > > >::run (this=<optimized out>)
    at /usr/include/boost/thread/detail/thread.hpp:117
#21 0x00007ff89b48caea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#22 0x00007ff898e75064 in start_thread (arg=0x7ff84e2ed700) at pthread_create.c:309
#23 0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 14 (Thread 0x7ff847fff700 (LWP 12955)):
#0  0x00007ff8974f0893 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x000055fb3df8adf5 in CConnman::ThreadSocketHandler (this=<optimized out>) at net.cpp:1256
#2  0x000055fb3df5dd9d in operator() (this=0x7ff847ffee70) at /usr/include/c++/4.9/functional:2439
#3  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55fb3e71bd31 "net", func=...) at util.h:302
#4  0x000055fb3df8e9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#5  operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#6  std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#7  0x00007ff897d87970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ff898e75064 in start_thread (arg=0x7ff847fff700) at pthread_create.c:309
#9  0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 13 (Thread 0x7ff84566a700 (LWP 12960)):
#0  0x00007ff8974eeaed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff896633ee4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007ff896633ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ff8999031cc in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#4  0x00007ff8998a8512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ff899668cb4 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ff89966dd2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007ff898e75064 in start_thread (arg=0x7ff84566a700) at pthread_create.c:309
#8  0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 12 (Thread 0x7ff845ebb700 (LWP 12959)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x000055fb3df85f15 in __gthread_cond_timedwait (__abs_timeout=0x7ff845ebaca0, __mutex=<optimized out>, __cond=0x7ff87c5c3f90)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>, __lock=..., this=0x7ff87c5c3f90)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x7ff87c5c3f90)
    at /usr/include/c++/4.9/condition_variable:118
#4  wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CConnman::ThreadMessageHandler()::<lambda()> > (
    __p=..., __atime=..., __lock=..., this=0x7ff87c5c3f90) at /usr/include/c++/4.9/condition_variable:128
#5  CConnman::ThreadMessageHandler (this=0x7ff87c573b90) at net.cpp:2021
#6  0x000055fb3df5dd9d in operator() (this=0x7ff845ebae70) at /usr/include/c++/4.9/functional:2439
#7  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55fb3e6e8ebd "msghand", func=...) at util.h:302
#8  0x000055fb3df8e9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#9  operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#10 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#11 0x00007ff897d87970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007ff898e75064 in start_thread (arg=0x7ff845ebb700) at pthread_create.c:309
#13 0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 11 (Thread 0x7ff862432700 (LWP 12949)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055fb3e296d8b in BGThread (this=0x7ff87c4109c0) at leveldb/util/env_posix.cc:641
#2  leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7ff87c4109c0) at leveldb/util/env_posix.cc:556
#3  0x00007ff898e75064 in start_thread (arg=0x7ff862432700) at pthread_create.c:309
#4  0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 10 (Thread 0x7ff846f5d700 (LWP 12957)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x000055fb3e2247b2 in __gthread_cond_timedwait (__abs_timeout=0x7ff846f5cb60, __mutex=0x7ff87c5c4020, __cond=0x7ff87c5c3ff0)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7ff87c5c3ff0)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7ff87c5c3ff0)
    at /usr/include/c++/4.9/condition_variable:105
#4  wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>, this=0x7ff87c5c3ff0) at /usr/include/c++/4.9/condition_variable:128
#5  wait_for<long int, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __rtime=<synthetic pointer>, 
    __lock=<synthetic pointer>, this=0x7ff87c5c3ff0) at /usr/include/c++/4.9/condition_variable:144
#6  CThreadInterrupt::sleep_for (this=this@entry=0x7ff87c5c3ff0, rel_time=...) at threadinterrupt.cpp:30
#7  0x000055fb3e224860 in CThreadInterrupt::sleep_for (this=this@entry=0x7ff87c5c3ff0, rel_time=...) at threadinterrupt.cpp:35
#8  0x000055fb3df85997 in CConnman::ThreadOpenAddedConnections (this=0x7ff87c573b90) at net.cpp:1931
#9  0x000055fb3df5dd9d in operator() (this=0x7ff846f5ce70) at /usr/include/c++/4.9/functional:2439
#10 TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55fb3e6e8eae "addcon", func=...) at util.h:302
#11 0x000055fb3df8e9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#12 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#13 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#14 0x00007ff897d87970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007ff898e75064 in start_thread (arg=0x7ff846f5d700) at pthread_create.c:309
#16 0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 9 (Thread 0x7ff84670c700 (LWP 12958)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x000055fb3e2247b2 in __gthread_cond_timedwait (__abs_timeout=0x7ff84670b960, __mutex=0x7ff87c5c4020, __cond=0x7ff87c5c3ff0)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7ff87c5c3ff0)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7ff87c5c3ff0)
    at /usr/include/c++/4.9/condition_variable:105
#4  wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>, this=0x7ff87c5c3ff0) at /usr/include/c++/4.9/condition_variable:128
#5  wait_for<long int, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __rtime=<synthetic pointer>, 
    __lock=<synthetic pointer>, this=0x7ff87c5c3ff0) at /usr/include/c++/4.9/condition_variable:144
#6  CThreadInterrupt::sleep_for (this=this@entry=0x7ff87c5c3ff0, rel_time=..., rel_time@entry=...) at threadinterrupt.cpp:30
#7  0x000055fb3df844ed in CConnman::ThreadOpenConnections (this=0x7ff87c573b90) at net.cpp:1709
#8  0x000055fb3df5dd9d in operator() (this=0x7ff84670be70) at /usr/include/c++/4.9/functional:2439
#9  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55fb3e6e8eb5 "opencon", func=...) at util.h:302
#10 0x000055fb3df8e9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#11 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#12 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#13 0x00007ff897d87970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007ff898e75064 in start_thread (arg=0x7ff84670c700) at pthread_create.c:309
#15 0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 8 (Thread 0x7ff86af5d700 (LWP 12946)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff897d83c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x000055fb3e1107bb in Run (this=0x7ff87c046050) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7ff87c046050) at httpserver.cpp:351
#4  0x00007ff897d87970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ff898e75064 in start_thread (arg=0x7ff86af5d700) at pthread_create.c:309
#6  0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 7 (Thread 0x7ff878ed4700 (LWP 12943)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff897d83c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x000055fb3e1107bb in Run (this=0x7ff87c046050) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7ff87c046050) at httpserver.cpp:351
#4  0x00007ff897d87970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ff898e75064 in start_thread (arg=0x7ff878ed4700) at pthread_create.c:309
#6  0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 6 (Thread 0x7ff86b7ae700 (LWP 12945)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff897d83c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x000055fb3e1107bb in Run (this=0x7ff87c046050) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7ff87c046050) at httpserver.cpp:351
#4  0x00007ff897d87970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ff898e75064 in start_thread (arg=0x7ff86b7ae700) at pthread_create.c:309
#6  0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7ff87a7c7700 (LWP 12940)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055fb3de234cb in boost::condition_variable::wait (this=0x55fb3eb0de68 <scriptcheckqueue+40>, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
#2  0x000055fb3e0ccbb0 in CCheckQueue<CScriptCheck>::Loop (this=0x55fb3eb0de40 <scriptcheckqueue>, fMaster=false) at checkqueue.h:102
#3  0x00007ff89b48caea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#4  0x00007ff898e75064 in start_thread (arg=0x7ff87a7c7700) at pthread_create.c:309
#5  0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7ff879f76700 (LWP 12941)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x000055fb3e20570f in do_wait_until (timeout=..., m=..., this=0x55fb3fdf74e8) at /usr/include/boost/thread/pthread/condition_variable.hpp:108
#2  wait_until (tp=..., lk=..., this=0x55fb3fdf74e8) at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:253
#3  wait_until<boost::chrono::duration<long, boost::ratio<1l, 1000000000l> > > (t=<synthetic pointer>, lock=..., this=0x55fb3fdf74e8)
    at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:166
#4  CScheduler::serviceQueue (this=0x55fb3fdf74b8) at scheduler.cpp:67
#5  0x000055fb3df5dd9d in operator() (this=0x7ff879f75e40) at /usr/include/c++/4.9/functional:2439
#6  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55fb3e7e6f25 "scheduler", func=...) at util.h:302
#7  0x000055fb3df5c551 in operator()<void (*)(char const*, std::function<void()>), boost::_bi::list0> (f=<optimized out>, a=<synthetic pointer>, 
    this=<optimized out>) at /usr/include/boost/bind/bind.hpp:313
#8  operator() (this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:20
#9  boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > > >::run() (this=0x7ff87c044000) at /usr/include/boost/thread/detail/thread.hpp:117
#10 0x00007ff89b48caea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#11 0x00007ff898e75064 in start_thread (arg=0x7ff879f76700) at pthread_create.c:309
#12 0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7ff879725700 (LWP 12942)):
#0  0x00007ff8974f7c03 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff898000b08 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#2  0x00007ff897febd5a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#3  0x000055fb3e10e8ea in ThreadHTTP (base=0x7ff87c045020, http=<optimized out>) at httpserver.cpp:303
#4  0x000055fb3e11665e in operator() (this=0x7ff8700008c8) at /usr/include/c++/4.9/functional:2439
#5  operator() (this=0x7ff8700008c0) at /usr/include/c++/4.9/future:1241
#6  std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<bool>, std::__future_base::_Result_base::_Deleter>, bool> >::_M_invoke(std::_Any_data const&) (__functor=...)
    at /usr/include/c++/4.9/functional:2025
#7  0x000055fb3e116cf2 in operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:2439
#8  std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) (this=0x7ff87c0465c0, __f=..., __set=@0x7ff879724dcf: false) at /usr/include/c++/4.9/future:485
#9  0x00007ff898e7a410 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:103
#10 0x000055fb3e1177be in __gthread_once (__func=<optimized out>, __once=0x7ff87c04662c) at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:699
#11 call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>&, bool&), std::__future_base::_State_baseV2*, std::reference_wrapper<std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> >, std::reference_wrapper<bool> > (__f=<optimized out>, __once=...) at /usr/include/c++/4.9/mutex:746
#12 _M_set_result (__ignore_failure=false, __res=..., this=0x7ff87c0465c0) at /usr/include/c++/4.9/future:372
#13 _M_run (__args#1=<optimized out>, __args#0=<optimized out>, this=0x7ff87c0465c0) at /usr/include/c++/4.9/future:1319
#14 operator() (__args#1=<optimized out>, __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/4.9/future:1453
#15 _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#16 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#17 std::thread::_Impl<std::_Bind_simple<std::packaged_task<bool (event_base*, evhttp*)> (event_base*, evhttp*)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#18 0x00007ff897d87970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x00007ff898e75064 in start_thread (arg=0x7ff879725700) at pthread_create.c:309
#20 0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7ff88df13700 (LWP 12938)):
#0  0x00007ff8974eeaed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff8946f8252 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#2  0x00007ff8946f9ddf in xcb_wait_for_event () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#3  0x00007ff8906661b9 in ?? () from /usr/lib/x86_64-linux-gnu/qt5/plugins/platforms/libqxcb.so
#4  0x00007ff89966dd2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ff898e75064 in start_thread (arg=0x7ff88df13700) at pthread_create.c:309
#6  0x00007ff8974f762d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7ff89c16a780 (LWP 12935)):
#0  0x00007ff897444067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ff897445448 in __GI_abort () at abort.c:89
#2  0x00007ff8974821b4 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7ff897577210 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007ff89748798e in malloc_printerr (action=1, str=0x7ff8975773d0 "free(): invalid next size (fast)", ptr=<optimized out>) at malloc.c:4996
#4  0x00007ff897488696 in _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
#5  0x00007ff8998f3f9c in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ff89986ecb0 in QSortFilterProxyModel::lessThan(QModelIndex const&, QModelIndex const&) const () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007ff89987ce69 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#8  0x00007ff89987fd14 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#9  0x00007ff899881b84 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#10 0x00007ff899881ced in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#11 0x00007ff8998731ac in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#12 0x00007ff8998786cd in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#13 0x00007ff899878a95 in QSortFilterProxyModel::sort(int, Qt::SortOrder) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#14 0x00007ff89aa7dff7 in QTableView::setSortingEnabled(bool) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#15 0x000055fb3df13f6a in TransactionView::setModel (this=0x55fb404ca520, _model=_model@entry=0x55fb40170660) at qt/transactionview.cpp:212
#16 0x000055fb3deb4c3f in WalletView::setWalletModel (this=this@entry=0x55fb4047ada0, _walletModel=0x55fb40170660) at qt/walletview.cpp:121
#17 0x000055fb3dea15c5 in WalletFrame::addWallet (this=0x55fb3fc6e590, name=..., walletModel=<optimized out>) at qt/walletframe.cpp:49
#18 0x000055fb3de1b8b1 in BitcoinApplication::initializeResult (this=0x7ffeec7667d0, success=<optimized out>) at qt/bitcoin.cpp:492
#19 0x00007ff8998daeda in QObject::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#20 0x00007ff89a0706f2 in QGuiApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5
#21 0x00007ff89a80718c in QApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#22 0x00007ff89a80412c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#23 0x00007ff89a809600 in QApplication::notify(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#24 0x00007ff8998aadcb in QCoreApplication::notifyInternal(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#25 0x00007ff8998acdc3 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#26 0x00007ff899902da3 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#27 0x00007ff896633c5d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#28 0x00007ff896633f48 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#29 0x00007ff896633ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#30 0x00007ff8999031b7 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#31 0x00007ff8998a8512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#32 0x00007ff8998b015c in QCoreApplication::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#33 0x000055fb3de0806e in main (argc=2, argv=<optimized out>) at qt/bitcoin.cpp:714
(gdb) 
@jonasschnelli

This comment has been minimized.

Copy link
Member

commented Aug 24, 2017

Hmm...

#0  0x00007ff897444067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ff897445448 in __GI_abort () at abort.c:89
#2  0x00007ff8974821b4 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7ff897577210 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007ff89748798e in malloc_printerr (action=1, str=0x7ff8975773d0 "free(): invalid next size (fast)", ptr=<optimized out>) at malloc.c:4996
#4  0x00007ff897488696 in _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
#5  0x00007ff8998f3f9c in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ff89986ecb0 in QSortFilterProxyModel::lessThan(QModelIndex const&, QModelIndex const&) const () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007ff89987ce69 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#8  0x00007ff89987fd14 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#9  0x00007ff899881b84 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#10 0x00007ff899881ced in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#11 0x00007ff8998731ac in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#12 0x00007ff8998786cd in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#13 0x00007ff899878a95 in QSortFilterProxyModel::sort(int, Qt::SortOrder) () from /usr/lib/x86_64-

What version of Qt are you using (I guess you have self compiled)?

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

I build bitcoin-qt myself, but installed binary packages of Qt using the debian stable repo:

$ ldd qt/bitcoin-qt | grep -i qt
	libQt5Network.so.5 => /usr/lib/x86_64-linux-gnu/libQt5Network.so.5 (0x00007f9a7b233000)
	libQt5Widgets.so.5 => /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5 (0x00007f9a7a9de000)
	libQt5Gui.so.5 => /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5 (0x00007f9a7a2a7000)
	libQt5DBus.so.5 => /usr/lib/x86_64-linux-gnu/libQt5DBus.so.5 (0x00007f9a7a025000)
	libQt5Core.so.5 => /usr/lib/x86_64-linux-gnu/libQt5Core.so.5 (0x00007f9a798e9000)

$ ldd qt/bitcoin-qt | grep -i qt | awk '{print $3}' | while read x; do dpkg -l $(dpkg -S $x | cut -d: -f1-2); done | grep ii
ii  libqt5network5:amd64                5.3.2+dfsg-4+deb8u2    amd64                  Qt 5 network module
ii  libqt5widgets5:amd64                5.3.2+dfsg-4+deb8u2    amd64                  Qt 5 widgets module
ii  libqt5gui5:amd64                    5.3.2+dfsg-4+deb8u2    amd64                  Qt 5 GUI module
ii  libqt5dbus5:amd64                   5.3.2+dfsg-4+deb8u2    amd64                  Qt 5 D-Bus module
ii  libqt5core5a:amd64                  5.3.2+dfsg-4+deb8u2    amd64                  Qt 5 core module

$ apt-cache policy libqt5core5a:amd64
libqt5core5a:
  Installed: 5.3.2+dfsg-4+deb8u2
  Candidate: 5.3.2+dfsg-4+deb8u2
  Version table:
 *** 5.3.2+dfsg-4+deb8u2 0
        500 http://debian.osuosl.org/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status
$ 
@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

It might be worth mentioning that every time I have seen these crashes I have been using a wallet.dat that I first created in 2011. I don't really use any others. My wallet.dat is 21 MB.

I tried for a few hours to reproduce with a newly created empty wallet, but it never crashed.

I have other wallets lying around, but they're either mostly empty, or giagantic, taking over a minute to load up.

For now I think I'll keep starting up and quitting with a new empty wallet and see if I can get the crash to happen that way. If not, I'll try loading up some of my other wallets.

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

I just managed to reproduce the error on a wallet I made shortly after the Bitfinex theft. I found a list of the theft transactions online somewhere and imported all the receiving addresses into a watchonly wallet to track the stolen funds. Then promptly forgot about the wallet until now.

The first time I loaded up the wallet it took a long time, saying: Rescanning last 30806 blocks (from block 451097).... Eventually it completed, and I was able to cleanly shut down the wallet.

I ran it a second time to see how long it would take to load up and it crashed almost immediately:

$ while [[ ! -e core ]]; do echo "$(date) : starting"; ~/Programs/bitcoin.git/src/qt/bitcoin-qt; done
Thu Aug 24 13:35:27 PDT 2017 : starting [this is the one which did the long resync, and didn't crash]
Thu Aug 24 13:56:37 PDT 2017 : starting [this is the first run after the resync, and it crashed]
*** Error in `/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt': free(): invalid next size (fast): 0x00005595557060b0 ***
Aborted (core dumped)
$ cp core core.bak6
$ gdb qt/bitcoin-qt core.bak6
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from qt/bitcoin-qt...done.
[New LWP 17643]
[New LWP 17655]
[New LWP 17644]
[New LWP 17645]
[New LWP 17646]
[New LWP 17654]
[New LWP 17658]
[New LWP 17648]
[New LWP 17661]
[New LWP 17650]
[New LWP 17651]
[New LWP 17652]
[New LWP 17656]
[New LWP 17657]
[New LWP 17659]
[New LWP 17660]
[New LWP 17649]
[New LWP 17647]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f7750a17067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) thread apply all where

Thread 18 (Thread 0x7f77335a2700 (LWP 17647)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000559552d9b70f in do_wait_until (timeout=..., m=..., this=0x559555295418) at /usr/include/boost/thread/pthread/condition_variable.hpp:108
#2  wait_until (tp=..., lk=..., this=0x559555295418) at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:253
#3  wait_until<boost::chrono::duration<long, boost::ratio<1l, 1000000000l> > > (t=<synthetic pointer>, lock=..., this=0x559555295418)
    at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:166
#4  CScheduler::serviceQueue (this=0x5595552953e8) at scheduler.cpp:67
#5  0x0000559552af3d9d in operator() (this=0x7f77335a1e40) at /usr/include/c++/4.9/functional:2439
#6  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55955337cf25 "scheduler", func=...) at util.h:302
#7  0x0000559552af2551 in operator()<void (*)(char const*, std::function<void()>), boost::_bi::list0> (f=<optimized out>, a=<synthetic pointer>, 
    this=<optimized out>) at /usr/include/boost/bind/bind.hpp:313
#8  operator() (this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:20
#9  boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > > >::run() (this=0x7f7734044050) at /usr/include/boost/thread/detail/thread.hpp:117
#10 0x00007f7754a5faea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#11 0x00007f7752448064 in start_thread (arg=0x7f77335a2700) at pthread_create.c:309
#12 0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 17 (Thread 0x7f7732500700 (LWP 17649)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f7751356c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x0000559552ca67bb in Run (this=0x7f7734045fc0) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7f7734045fc0) at httpserver.cpp:351
#4  0x00007f775135a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f7752448064 in start_thread (arg=0x7f7732500700) at pthread_create.c:309
#6  0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 16 (Thread 0x7f7706f5d700 (LWP 17660)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000559552b1bf15 in __gthread_cond_timedwait (__abs_timeout=0x7f7706f5cca0, __mutex=<optimized out>, __cond=0x7f77344a68b0)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>, __lock=..., this=0x7f77344a68b0)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x7f77344a68b0)
    at /usr/include/c++/4.9/condition_variable:118
#4  wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CConnman::ThreadMessageHandler()::<lambda()> > (
    __p=..., __atime=..., __lock=..., this=0x7f77344a68b0) at /usr/include/c++/4.9/condition_variable:128
#5  CConnman::ThreadMessageHandler (this=0x7f77344564b0) at net.cpp:2021
#6  0x0000559552af3d9d in operator() (this=0x7f7706f5ce70) at /usr/include/c++/4.9/functional:2439
#7  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55955327eebd "msghand", func=...) at util.h:302
#8  0x0000559552b249cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#9  operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#10 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#11 0x00007f775135a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007f7752448064 in start_thread (arg=0x7f7706f5d700) at pthread_create.c:309
#13 0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 15 (Thread 0x7f77077ae700 (LWP 17659)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000559552dba7b2 in __gthread_cond_timedwait (__abs_timeout=0x7f77077ad960, __mutex=0x7f77344a6940, __cond=0x7f77344a6910)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7f77344a6910)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7f77344a6910)
    at /usr/include/c++/4.9/condition_variable:105
#4  wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>, this=0x7f77344a6910) at /usr/include/c++/4.9/condition_variable:128
#5  wait_for<long int, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __rtime=<synthetic pointer>, 
    __lock=<synthetic pointer>, this=0x7f77344a6910) at /usr/include/c++/4.9/condition_variable:144
#6  CThreadInterrupt::sleep_for (this=this@entry=0x7f77344a6910, rel_time=..., rel_time@entry=...) at threadinterrupt.cpp:30
#7  0x0000559552b1a4ed in CConnman::ThreadOpenConnections (this=0x7f77344564b0) at net.cpp:1709
#8  0x0000559552af3d9d in operator() (this=0x7f77077ade70) at /usr/include/c++/4.9/functional:2439
#9  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55955327eeb5 "opencon", func=...) at util.h:302
#10 0x0000559552b249cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#11 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#12 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#13 0x00007f775135a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007f7752448064 in start_thread (arg=0x7f77077ae700) at pthread_create.c:309
#15 0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 14 (Thread 0x7f770ce21700 (LWP 17657)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000559552dba7b2 in __gthread_cond_timedwait (__abs_timeout=0x7f770ce20970, __mutex=0x7f77344a6940, __cond=0x7f77344a6910)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7f77344a6910)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7f77344a6910)
    at /usr/include/c++/4.9/condition_variable:105
#4  wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>, this=0x7f77344a6910) at /usr/include/c++/4.9/condition_variable:128
#5  wait_for<long int, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __rtime=<synthetic pointer>, 
    __lock=<synthetic pointer>, this=0x7f77344a6910) at /usr/include/c++/4.9/condition_variable:144
#6  CThreadInterrupt::sleep_for (this=this@entry=0x7f77344a6910, rel_time=...) at threadinterrupt.cpp:30
#7  0x0000559552dba860 in CThreadInterrupt::sleep_for (this=this@entry=0x7f77344a6910, rel_time=..., rel_time@entry=...) at threadinterrupt.cpp:35
#8  0x0000559552b0d647 in CConnman::ThreadDNSAddressSeed (this=0x7f77344564b0) at net.cpp:1579
#9  0x0000559552af3d9d in operator() (this=0x7f770ce20e70) at /usr/include/c++/4.9/functional:2439
#10 TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x559553266b33 "dnsseed", func=...) at util.h:302
#11 0x0000559552b249cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#12 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#13 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#14 0x00007f775135a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007f7752448064 in start_thread (arg=0x7f770ce21700) at pthread_create.c:309
#16 0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 13 (Thread 0x7f770d672700 (LWP 17656)):
#0  0x00007f7750ac3893 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000559552b20df5 in CConnman::ThreadSocketHandler (this=<optimized out>) at net.cpp:1256
#2  0x0000559552af3d9d in operator() (this=0x7f770d671e70) at /usr/include/c++/4.9/functional:2439
#3  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x5595532b1d31 "net", func=...) at util.h:302
#4  0x0000559552b249cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#5  operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#6  std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#7  0x00007f775135a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007f7752448064 in start_thread (arg=0x7f770d672700) at pthread_create.c:309
#9  0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 12 (Thread 0x7f7730c0d700 (LWP 17652)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f7751356c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x0000559552ca67bb in Run (this=0x7f7734045fc0) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7f7734045fc0) at httpserver.cpp:351
#4  0x00007f775135a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f7752448064 in start_thread (arg=0x7f7730c0d700) at pthread_create.c:309
#6  0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 11 (Thread 0x7f773145e700 (LWP 17651)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f7751356c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x0000559552ca67bb in Run (this=0x7f7734045fc0) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7f7734045fc0) at httpserver.cpp:351
#4  0x00007f775135a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f7752448064 in start_thread (arg=0x7f773145e700) at pthread_create.c:309
#6  0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 10 (Thread 0x7f7731caf700 (LWP 17650)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f7751356c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x0000559552ca67bb in Run (this=0x7f7734045fc0) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7f7734045fc0) at httpserver.cpp:351
#4  0x00007f775135a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f7752448064 in start_thread (arg=0x7f7731caf700) at pthread_create.c:309
#6  0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 9 (Thread 0x7f770670c700 (LWP 17661)):
#0  0x00007f7750ac1aed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f774fc06ee4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f774fc06ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f7752ed61cc in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#4  0x00007f7752e7b512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007f7752c3bcb4 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007f7752c40d2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007f7752448064 in start_thread (arg=0x7f770670c700) at pthread_create.c:309
#8  0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 8 (Thread 0x7f7732d51700 (LWP 17648)):
#0  0x00007f7750acac03 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f77515d3b08 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#2  0x00007f77515bed5a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#3  0x0000559552ca48ea in ThreadHTTP (base=0x7f77340450a0, http=<optimized out>) at httpserver.cpp:303
#4  0x0000559552cac65e in operator() (this=0x7f77280008c8) at /usr/include/c++/4.9/functional:2439
#5  operator() (this=0x7f77280008c0) at /usr/include/c++/4.9/future:1241
#6  std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<bool>, std::__future_base::_Result_base::_Deleter>, bool> >::_M_invoke(std::_Any_data const&) (__functor=...)
    at /usr/include/c++/4.9/functional:2025
#7  0x0000559552caccf2 in operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:2439
#8  std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) (this=0x7f77340465a0, __f=..., __set=@0x7f7732d50dcf: false) at /usr/include/c++/4.9/future:485
#9  0x00007f775244d410 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:103
#10 0x0000559552cad7be in __gthread_once (__func=<optimized out>, __once=0x7f773404660c) at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:699
#11 call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>&, bool&), std::__future_base::_State_baseV2*, std::reference_wrapper<std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> >, std::reference_wrapper<bool> > (__f=<optimized out>, __once=...) at /usr/include/c++/4.9/mutex:746
#12 _M_set_result (__ignore_failure=false, __res=..., this=0x7f77340465a0) at /usr/include/c++/4.9/future:372
#13 _M_run (__args#1=<optimized out>, __args#0=<optimized out>, this=0x7f77340465a0) at /usr/include/c++/4.9/future:1319
#14 operator() (__args#1=<optimized out>, __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/4.9/future:1453
#15 _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#16 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#17 std::thread::_Impl<std::_Bind_simple<std::packaged_task<bool (event_base*, evhttp*)> (event_base*, evhttp*)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#18 0x00007f775135a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x00007f7752448064 in start_thread (arg=0x7f7732d51700) at pthread_create.c:309
#20 0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 7 (Thread 0x7f7707fff700 (LWP 17658)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000559552dba7b2 in __gthread_cond_timedwait (__abs_timeout=0x7f7707ffeb60, __mutex=0x7f77344a6940, __cond=0x7f77344a6910)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7f77344a6910)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7f77344a6910)
    at /usr/include/c++/4.9/condition_variable:105
#4  wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>, this=0x7f77344a6910) at /usr/include/c++/4.9/condition_variable:128
#5  wait_for<long int, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __rtime=<synthetic pointer>, 
    __lock=<synthetic pointer>, this=0x7f77344a6910) at /usr/include/c++/4.9/condition_variable:144
#6  CThreadInterrupt::sleep_for (this=this@entry=0x7f77344a6910, rel_time=...) at threadinterrupt.cpp:30
#7  0x0000559552dba860 in CThreadInterrupt::sleep_for (this=this@entry=0x7f77344a6910, rel_time=...) at threadinterrupt.cpp:35
#8  0x0000559552b1b997 in CConnman::ThreadOpenAddedConnections (this=0x7f77344564b0) at net.cpp:1931
#9  0x0000559552af3d9d in operator() (this=0x7f7707ffee70) at /usr/include/c++/4.9/functional:2439
#10 TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x55955327eeae "addcon", func=...) at util.h:302
#11 0x0000559552b249cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#12 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#13 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#14 0x00007f775135a970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007f7752448064 in start_thread (arg=0x7f7707fff700) at pthread_create.c:309
#16 0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 6 (Thread 0x7f7723fff700 (LWP 17654)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000559552e2cd8b in BGThread (this=0x7f7734412c30) at leveldb/util/env_posix.cc:641
#2  leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7f7734412c30) at leveldb/util/env_posix.cc:556
#3  0x00007f7752448064 in start_thread (arg=0x7f7723fff700) at pthread_create.c:309
#4  0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7f7733df3700 (LWP 17646)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00005595529b94cb in boost::condition_variable::wait (this=0x5595536a3e68 <scriptcheckqueue+40>, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
#2  0x0000559552c62bb0 in CCheckQueue<CScriptCheck>::Loop (this=0x5595536a3e40 <scriptcheckqueue>, fMaster=false) at checkqueue.h:102
#3  0x00007f7754a5faea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#4  0x00007f7752448064 in start_thread (arg=0x7f7733df3700) at pthread_create.c:309
#5  0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7f773b00d700 (LWP 17645)):
#0  0x00007f7750ac1aed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f774fc06ee4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f774fc06ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f7752ed61cc in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#4  0x00007f7752e7b512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007f7752c3bcb4 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007f7752c40d2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007f7752448064 in start_thread (arg=0x7f773b00d700) at pthread_create.c:309
#8  0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7f77474e6700 (LWP 17644)):
#0  0x00007f7750ac1aed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f774dccb252 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#2  0x00007f774dcccddf in xcb_wait_for_event () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#3  0x00007f7749c391b9 in ?? () from /usr/lib/x86_64-linux-gnu/qt5/plugins/platforms/libqxcb.so
#4  0x00007f7752c40d2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007f7752448064 in start_thread (arg=0x7f77474e6700) at pthread_create.c:309
#6  0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7f770e4ff700 (LWP 17655)):
#0  0x0000559552e3a879 in secp256k1_fe_mul_inner (r=r@entry=0x7f770e4fd0d0, a=0xad98c61d9d559, a@entry=0x7f770e4fd0d0, b=0xfffffffffffff, b@entry=0x7f770e4fd0a8)
    at src/field_5x52_asm_impl.h:28
#1  0x0000559552e464ec in secp256k1_fe_mul (b=0x7f770e4fd0a8, a=0x7f770e4fd0d0, r=0x7f770e4fd0d0) at src/field_5x52_impl.h:426
#2  secp256k1_gej_double_var (rzr=0x0, a=0x7f770e4fd080, r=0x7f770e4fd080) at src/group_impl.h:334
#3  secp256k1_ecmult (ctx=ctx@entry=0x5595552c93f0, r=r@entry=0x7f770e4fd080, a=a@entry=0x7f770e4fd000, na=na@entry=0x7f770e4fcf50, ng=ng@entry=0x7f770e4fcf30)
    at src/ecmult_impl.h:371
#4  0x0000559552e49424 in secp256k1_ecdsa_sig_verify (message=0x7f770e4fced0, pubkey=0x7f770e4fcfa0, sigs=0x7f770e4fceb0, sigr=0x7f770e4fce90, ctx=0x5595552c93f0)
    at src/ecdsa_impl.h:220
#5  secp256k1_ecdsa_verify (ctx=0x5595552c93f0, sig=<optimized out>, msg32=<optimized out>, pubkey=<optimized out>) at src/secp256k1.c:307
#6  0x0000559552dd2d1c in CPubKey::Verify (this=this@entry=0x7f770e4fd360, hash=..., vchSig=std::vector of length 70, capacity 71 = {...}) at pubkey.cpp:181
#7  0x0000559552dd3ae6 in TransactionSignatureChecker::VerifySignature (this=this@entry=0x7f770e4fd970, vchSig=std::vector of length 70, capacity 71 = {...}, 
    pubkey=..., sighash=...) at script/interpreter.cpp:1248
#8  0x0000559552be7255 in CachingTransactionSignatureChecker::VerifySignature (this=0x7f770e4fd970, vchSig=std::vector of length 70, capacity 71 = {...}, 
    pubkey=..., sighash=...) at script/sigcache.cpp:89
#9  0x0000559552ddb6b2 in TransactionSignatureChecker::CheckSig (this=0x7f770e4fd970, vchSigIn=..., vchPubKey=..., scriptCode=..., sigversion=<optimized out>)
    at script/interpreter.cpp:1266
#10 0x0000559552dd8aa8 in EvalScript (stack=std::vector of length 2, capacity 4 = {...}, script=..., flags=flags@entry=65503, checker=..., 
    sigversion=sigversion@entry=SIGVERSION_BASE, serror=serror@entry=0x7f770e4fda9c) at script/interpreter.cpp:900
#11 0x0000559552ddc2d5 in VerifyScript (scriptSig=..., scriptPubKey=..., witness=0x7f77080ffd18, flags=65503, checker=..., serror=serror@entry=0x7f770e4fda9c)
    at script/interpreter.cpp:1427
#12 0x0000559552c2c8d7 in CScriptCheck::operator() (this=this@entry=0x7f770e4fda60) at validation.cpp:1202
#13 0x0000559552c32052 in CheckInputs (tx=..., state=..., inputs=..., fScriptChecks=fScriptChecks@entry=true, flags=flags@entry=65503, 
    cacheSigStore=cacheSigStore@entry=true, cacheFullScriptStore=false, txdata=..., pvChecks=0x0) at validation.cpp:1292
#14 0x0000559552c4bd1e in AcceptToMemoryPoolWorker (chainparams=..., pool=..., state=..., ptx=std::shared_ptr (count 2, weak 0) 0x7f770811df20, 
    fLimitFree=fLimitFree@entry=true, pfMissingInputs=pfMissingInputs@entry=0x0, nAcceptTime=1503591511, plTxnReplaced=0x0, fOverrideMempoolLimit=false, 
    nAbsurdFee=@0x7f770e4fe668: 0, coins_to_uncache=std::vector of length 1, capacity 1 = {...}) at validation.cpp:792
#15 0x0000559552c4ed5f in AcceptToMemoryPoolWithTime (chainparams=..., pool=..., state=..., tx=std::shared_ptr (count 2, weak 0) 0x7f770811df20, 
    fLimitFree=fLimitFree@entry=true, pfMissingInputs=pfMissingInputs@entry=0x0, nAcceptTime=1503591511, plTxnReplaced=0x0, fOverrideMempoolLimit=false, 
    nAbsurdFee=0) at validation.cpp:880
#16 0x0000559552c56648 in LoadMempool () at validation.cpp:4302
#17 0x0000559552acefb7 in ThreadImport (vImportFiles=std::vector of length 0, capacity 0) at init.cpp:692
#18 0x0000559552af8cca in operator()<void (*)(std::vector<boost::filesystem::path>), boost::_bi::list0> (f=<optimized out>, a=<synthetic pointer>, 
    this=<optimized out>) at /usr/include/boost/bind/bind.hpp:253
#19 operator() (this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:20
#20 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >), boost::_bi::list1<boost::_bi::value<std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> > > > > >::run (this=<optimized out>)
    at /usr/include/boost/thread/detail/thread.hpp:117
#21 0x00007f7754a5faea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#22 0x00007f7752448064 in start_thread (arg=0x7f770e4ff700) at pthread_create.c:309
#23 0x00007f7750aca62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f775573d780 (LWP 17643)):
#0  0x00007f7750a17067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f7750a18448 in __GI_abort () at abort.c:89
#2  0x00007f7750a551b4 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7f7750b4a210 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007f7750a5a98e in malloc_printerr (action=1, str=0x7f7750b4a3d0 "free(): invalid next size (fast)", ptr=<optimized out>) at malloc.c:4996
#4  0x00007f7750a5b696 in _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
#5  0x00007f7752ec6f9c in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007f7752e41cb0 in QSortFilterProxyModel::lessThan(QModelIndex const&, QModelIndex const&) const () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007f7752e4fe69 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#8  0x00007f7752e52d14 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#9  0x00007f7752e54b84 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#10 0x00007f7752e54d17 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#11 0x00007f7752e461ac in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#12 0x00007f7752e4b6cd in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#13 0x00007f7752e4ba95 in QSortFilterProxyModel::sort(int, Qt::SortOrder) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#14 0x00007f7754050ff7 in QTableView::setSortingEnabled(bool) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#15 0x0000559552aa9f6a in TransactionView::setModel (this=0x559554aeb510, _model=_model@entry=0x5595556169d0) at qt/transactionview.cpp:212
#16 0x0000559552a4ac3f in WalletView::setWalletModel (this=this@entry=0x559554b07290, _walletModel=0x5595556169d0) at qt/walletview.cpp:121
#17 0x0000559552a375c5 in WalletFrame::addWallet (this=0x55955510c400, name=..., walletModel=<optimized out>) at qt/walletframe.cpp:49
#18 0x00005595529b18b1 in BitcoinApplication::initializeResult (this=0x7ffdfb3a0310, success=<optimized out>) at qt/bitcoin.cpp:492
#19 0x00007f7752eadeda in QObject::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#20 0x00007f77536436f2 in QGuiApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5
#21 0x00007f7753dda18c in QApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#22 0x00007f7753dd712c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#23 0x00007f7753ddc600 in QApplication::notify(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#24 0x00007f7752e7ddcb in QCoreApplication::notifyInternal(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#25 0x00007f7752e7fdc3 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#26 0x00007f7752ed5da3 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#27 0x00007f774fc06c5d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#28 0x00007f774fc06f48 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#29 0x00007f774fc06ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#30 0x00007f7752ed61b7 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#31 0x00007f7752e7b512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#32 0x00007f7752e8315c in QCoreApplication::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#33 0x000055955299e06e in main (argc=1, argv=<optimized out>) at qt/bitcoin.cpp:714
(gdb) 

I've attached a copy of the wallet.dat for this one, since it's nothing but a bunch of watchonly addresses for the BitFinex hack.

I'm glad that I've seen the crash while loading a wallet that I can share.

I tried loading the wallet by running the client inside gdb, and it crashed again the very first time I tried (but not the 2nd):

$ gdb qt/bitcoin-qt
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from qt/bitcoin-qt...done.
(gdb) run
Starting program: /home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe9d61700 (LWP 18021)]
[New Thread 0x7fffdd8dc700 (LWP 18024)]
[New Thread 0x7fffd6731700 (LWP 18025)]
[New Thread 0x7fffd5ee0700 (LWP 18026)]
[New Thread 0x7fffd568f700 (LWP 18027)]
[New Thread 0x7fffd4e3e700 (LWP 18028)]
[New Thread 0x7fffc7fff700 (LWP 18029)]
[New Thread 0x7fffc77ae700 (LWP 18030)]
[New Thread 0x7fffc6f5d700 (LWP 18031)]
[New Thread 0x7fffbce82700 (LWP 18032)]
[New Thread 0x7fffb503a700 (LWP 18033)]
[New Thread 0x7fffabdeb700 (LWP 18034)]
[New Thread 0x7fffab59a700 (LWP 18035)]
[New Thread 0x7fffaad49700 (LWP 18036)]
[New Thread 0x7fffaa4f8700 (LWP 18037)]
[New Thread 0x7fffa9ca7700 (LWP 18038)]
[New Thread 0x7fffa89f3700 (LWP 18039)]
*** Error in `/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt': double free or corruption (!prev): 0x0000555556fc1150 ***

Program received signal SIGABRT, Aborted.
0x00007ffff3292067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) thread apply all where

Thread 18 (Thread 0x7fffa89f3700 (LWP 18039)):
#0  0x00007ffff333caed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff2481ee4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007ffff2481ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff57511cc in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#4  0x00007ffff56f6512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ffff54b6cb4 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ffff54bbd2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007ffff4cc3064 in start_thread (arg=0x7fffa89f3700) at pthread_create.c:309
#8  0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 17 (Thread 0x7fffa9ca7700 (LWP 18038)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000555555746f15 in __gthread_cond_timedwait (__abs_timeout=0x7fffa9ca6ca0, __mutex=<optimized out>, __cond=0x7fffd85816a0)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>, __lock=..., this=0x7fffd85816a0)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x7fffd85816a0)
    at /usr/include/c++/4.9/condition_variable:118
#4  wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CConnman::ThreadMessageHandler()::<lambda()> > (
    __p=..., __atime=..., __lock=..., this=0x7fffd85816a0) at /usr/include/c++/4.9/condition_variable:128
#5  CConnman::ThreadMessageHandler (this=0x7fffd85312a0) at net.cpp:2021
#6  0x000055555571ed9d in operator() (this=0x7fffa9ca6e70) at /usr/include/c++/4.9/functional:2439
#7  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555555ea9ebd "msghand", func=...) at util.h:302
#8  0x000055555574f9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#9  operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#10 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#11 0x00007ffff3bd5970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007ffff4cc3064 in start_thread (arg=0x7fffa9ca7700) at pthread_create.c:309
#13 0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 16 (Thread 0x7fffaa4f8700 (LWP 18037)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00005555559e57b2 in __gthread_cond_timedwait (__abs_timeout=0x7fffaa4f7960, __mutex=0x7fffd8581730, __cond=0x7fffd8581700)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7fffd8581700)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7fffd8581700)
    at /usr/include/c++/4.9/condition_variable:105
#4  wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>, this=0x7fffd8581700) at /usr/include/c++/4.9/condition_variable:128
#5  wait_for<long int, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __rtime=<synthetic pointer>, 
    __lock=<synthetic pointer>, this=0x7fffd8581700) at /usr/include/c++/4.9/condition_variable:144
#6  CThreadInterrupt::sleep_for (this=this@entry=0x7fffd8581700, rel_time=..., rel_time@entry=...) at threadinterrupt.cpp:30
#7  0x00005555557454ed in CConnman::ThreadOpenConnections (this=0x7fffd85312a0) at net.cpp:1709
#8  0x000055555571ed9d in operator() (this=0x7fffaa4f7e70) at /usr/include/c++/4.9/functional:2439
#9  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555555ea9eb5 "opencon", func=...) at util.h:302
#10 0x000055555574f9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#11 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#12 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#13 0x00007ffff3bd5970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007ffff4cc3064 in start_thread (arg=0x7fffaa4f8700) at pthread_create.c:309
#15 0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 15 (Thread 0x7fffaad49700 (LWP 18036)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00005555559e57b2 in __gthread_cond_timedwait (__abs_timeout=0x7fffaad48b60, __mutex=0x7fffd8581730, __cond=0x7fffd8581700)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7fffd8581700)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7fffd8581700)
    at /usr/include/c++/4.9/condition_variable:105
#4  wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>, this=0x7fffd8581700) at /usr/include/c++/4.9/condition_variable:128
#5  wait_for<long int, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __rtime=<synthetic pointer>, 
    __lock=<synthetic pointer>, this=0x7fffd8581700) at /usr/include/c++/4.9/condition_variable:144
#6  CThreadInterrupt::sleep_for (this=this@entry=0x7fffd8581700, rel_time=...) at threadinterrupt.cpp:30
#7  0x00005555559e5860 in CThreadInterrupt::sleep_for (this=this@entry=0x7fffd8581700, rel_time=...) at threadinterrupt.cpp:35
#8  0x0000555555746997 in CConnman::ThreadOpenAddedConnections (this=0x7fffd85312a0) at net.cpp:1931
#9  0x000055555571ed9d in operator() (this=0x7fffaad48e70) at /usr/include/c++/4.9/functional:2439
#10 TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555555ea9eae "addcon", func=...) at util.h:302
#11 0x000055555574f9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#12 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#13 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#14 0x00007ffff3bd5970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007ffff4cc3064 in start_thread (arg=0x7fffaad49700) at pthread_create.c:309
#16 0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 14 (Thread 0x7fffab59a700 (LWP 18035)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00005555559e57b2 in __gthread_cond_timedwait (__abs_timeout=0x7fffab599970, __mutex=0x7fffd8581730, __cond=0x7fffd8581700)
    at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7fffd8581700)
    at /usr/include/c++/4.9/condition_variable:165
#3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=<synthetic pointer>, this=0x7fffd8581700)
    at /usr/include/c++/4.9/condition_variable:105
#4  wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __atime=..., __lock=<synthetic pointer>, this=0x7fffd8581700) at /usr/include/c++/4.9/condition_variable:128
#5  wait_for<long int, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::milliseconds)::<lambda()> > (__p=..., __rtime=<synthetic pointer>, 
    __lock=<synthetic pointer>, this=0x7fffd8581700) at /usr/include/c++/4.9/condition_variable:144
#6  CThreadInterrupt::sleep_for (this=this@entry=0x7fffd8581700, rel_time=...) at threadinterrupt.cpp:30
#7  0x00005555559e5860 in CThreadInterrupt::sleep_for (this=this@entry=0x7fffd8581700, rel_time=..., rel_time@entry=...) at threadinterrupt.cpp:35
#8  0x0000555555738647 in CConnman::ThreadDNSAddressSeed (this=0x7fffd85312a0) at net.cpp:1579
#9  0x000055555571ed9d in operator() (this=0x7fffab599e70) at /usr/include/c++/4.9/functional:2439
#10 TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555555e91b33 "dnsseed", func=...) at util.h:302
#11 0x000055555574f9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#12 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#13 std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#14 0x00007ffff3bd5970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007ffff4cc3064 in start_thread (arg=0x7fffab59a700) at pthread_create.c:309
#16 0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 13 (Thread 0x7fffabdeb700 (LWP 18034)):
#0  0x00007ffff333e893 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x000055555574bdf5 in CConnman::ThreadSocketHandler (this=<optimized out>) at net.cpp:1256
#2  0x000055555571ed9d in operator() (this=0x7fffabdeae70) at /usr/include/c++/4.9/functional:2439
#3  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555555edcd31 "net", func=...) at util.h:302
#4  0x000055555574f9cb in _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#5  operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#6  std::thread::_Impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#7  0x00007ffff3bd5970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff4cc3064 in start_thread (arg=0x7fffabdeb700) at pthread_create.c:309
#9  0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 12 (Thread 0x7fffb503a700 (LWP 18033)):
#0  std::_Hashtable<COutPoint, std::pair<COutPoint const, CCoinsCacheEntry>, std::allocator<std::pair<COutPoint const, CCoinsCacheEntry> >, std::__detail::_Select1st, std::equal_to<COutPoint>, SaltedOutpointHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_find_before_node (this=this@entry=0x7fffb89cbec0, __n=332, __k=..., __code=2585514289634949751)
    at /usr/include/c++/4.9/bits/hashtable.h:1439
#1  0x00005555559a4425 in _M_find_node (__c=<optimized out>, __key=..., __bkt=<optimized out>, this=0x7fffb89cbec0) at /usr/include/c++/4.9/bits/hashtable.h:625
#2  std::_Hashtable<COutPoint, std::pair<COutPoint const, CCoinsCacheEntry>, std::allocator<std::pair<COutPoint const, CCoinsCacheEntry> >, std::__detail::_Select1st, std::equal_to<COutPoint>, SaltedOutpointHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::find (this=0x7fffb89cbec0, __k=...) at /usr/include/c++/4.9/bits/hashtable.h:1304
#3  0x00005555559a271d in find (__x=..., this=<optimized out>) at /usr/include/c++/4.9/bits/unordered_map.h:574
#4  CCoinsViewCache::HaveCoinInCache (this=<optimized out>, outpoint=...) at coins.cpp:134
#5  0x0000555555875f25 in AcceptToMemoryPoolWorker (chainparams=..., pool=..., state=..., ptx=std::shared_ptr (count 1, weak 0) 0x7fffac12dfc0, 
    fLimitFree=fLimitFree@entry=true, pfMissingInputs=pfMissingInputs@entry=0x0, nAcceptTime=1503591673, plTxnReplaced=0x0, fOverrideMempoolLimit=false, 
    nAbsurdFee=@0x7fffb5039668: 0, coins_to_uncache=std::vector of length 4, capacity 4 = {...}) at validation.cpp:538
#6  0x0000555555879d5f in AcceptToMemoryPoolWithTime (chainparams=..., pool=..., state=..., tx=std::shared_ptr (count 1, weak 0) 0x7fffac12dfc0, 
    fLimitFree=fLimitFree@entry=true, pfMissingInputs=pfMissingInputs@entry=0x0, nAcceptTime=1503591673, plTxnReplaced=0x0, fOverrideMempoolLimit=false, 
    nAbsurdFee=0) at validation.cpp:880
#7  0x0000555555881648 in LoadMempool () at validation.cpp:4302
#8  0x00005555556f9fb7 in ThreadImport (vImportFiles=std::vector of length 0, capacity 0) at init.cpp:692
#9  0x0000555555723cca in operator()<void (*)(std::vector<boost::filesystem::path>), boost::_bi::list0> (f=<optimized out>, a=<synthetic pointer>, 
    this=<optimized out>) at /usr/include/boost/bind/bind.hpp:253
#10 operator() (this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:20
#11 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >), boost::_bi::list1<boost::_bi::value<std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> > > > > >::run (this=<optimized out>)
    at /usr/include/boost/thread/detail/thread.hpp:117
#12 0x00007ffff72daaea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#13 0x00007ffff4cc3064 in start_thread (arg=0x7fffb503a700) at pthread_create.c:309
#14 0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 11 (Thread 0x7fffbce82700 (LWP 18032)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000555555a57d8b in BGThread (this=0x7fffd8402a70) at leveldb/util/env_posix.cc:641
#2  leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7fffd8402a70) at leveldb/util/env_posix.cc:556
#3  0x00007ffff4cc3064 in start_thread (arg=0x7fffbce82700) at pthread_create.c:309
#4  0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 10 (Thread 0x7fffc6f5d700 (LWP 18031)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff3bd1c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005555558d17bb in Run (this=0x7fffd8046080) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7fffd8046080) at httpserver.cpp:351
#4  0x00007ffff3bd5970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff4cc3064 in start_thread (arg=0x7fffc6f5d700) at pthread_create.c:309
#6  0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 9 (Thread 0x7fffc77ae700 (LWP 18030)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff3bd1c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005555558d17bb in Run (this=0x7fffd8046080) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7fffd8046080) at httpserver.cpp:351
#4  0x00007ffff3bd5970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff4cc3064 in start_thread (arg=0x7fffc77ae700) at pthread_create.c:309
#6  0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 8 (Thread 0x7fffc7fff700 (LWP 18029)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff3bd1c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005555558d17bb in Run (this=0x7fffd8046080) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7fffd8046080) at httpserver.cpp:351
#4  0x00007ffff3bd5970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff4cc3064 in start_thread (arg=0x7fffc7fff700) at pthread_create.c:309
#6  0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 7 (Thread 0x7fffd4e3e700 (LWP 18028)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff3bd1c7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005555558d17bb in Run (this=0x7fffd8046080) at httpserver.cpp:127
#3  HTTPWorkQueueRun (queue=0x7fffd8046080) at httpserver.cpp:351
#4  0x00007ffff3bd5970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff4cc3064 in start_thread (arg=0x7fffd4e3e700) at pthread_create.c:309
#6  0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 6 (Thread 0x7fffd568f700 (LWP 18027)):
#0  0x00007ffff3345c03 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff3e4eb08 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#2  0x00007ffff3e39d5a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#3  0x00005555558cf8ea in ThreadHTTP (base=0x7fffd80450a0, http=<optimized out>) at httpserver.cpp:303
#4  0x00005555558d765e in operator() (this=0x7fffcc0008c8) at /usr/include/c++/4.9/functional:2439
#5  operator() (this=0x7fffcc0008c0) at /usr/include/c++/4.9/future:1241
#6  std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<bool>, std::__future_base::_Result_base::_Deleter>, bool> >::_M_invoke(std::_Any_data const&) (__functor=...)
    at /usr/include/c++/4.9/functional:2025
#7  0x00005555558d7cf2 in operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:2439
#8  std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) (this=0x7fffd8046630, __f=..., __set=@0x7fffd568edcf: false) at /usr/include/c++/4.9/future:485
#9  0x00007ffff4cc8410 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:103
#10 0x00005555558d87be in __gthread_once (__func=<optimized out>, __once=0x7fffd804669c) at /usr/include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:699
#11 call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>&, bool&), std::__future_base::_State_baseV2*, std::reference_wrapper<std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> >, std::reference_wrapper<bool> > (__f=<optimized out>, __once=...) at /usr/include/c++/4.9/mutex:746
#12 _M_set_result (__ignore_failure=false, __res=..., this=0x7fffd8046630) at /usr/include/c++/4.9/future:372
#13 _M_run (__args#1=<optimized out>, __args#0=<optimized out>, this=0x7fffd8046630) at /usr/include/c++/4.9/future:1319
#14 operator() (__args#1=<optimized out>, __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/4.9/future:1453
#15 _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#16 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#17 std::thread::_Impl<std::_Bind_simple<std::packaged_task<bool (event_base*, evhttp*)> (event_base*, evhttp*)> >::_M_run() (this=<optimized out>)
    at /usr/include/c++/4.9/thread:115
#18 0x00007ffff3bd5970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x00007ffff4cc3064 in start_thread (arg=0x7fffd568f700) at pthread_create.c:309
#20 0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7fffd5ee0700 (LWP 18026)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00005555559c670f in do_wait_until (timeout=..., m=..., this=0x555556b754b8) at /usr/include/boost/thread/pthread/condition_variable.hpp:108
#2  wait_until (tp=..., lk=..., this=0x555556b754b8) at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:253
#3  wait_until<boost::chrono::duration<long, boost::ratio<1l, 1000000000l> > > (t=<synthetic pointer>, lock=..., this=0x555556b754b8)
    at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:166
#4  CScheduler::serviceQueue (this=0x555556b75488) at scheduler.cpp:67
#5  0x000055555571ed9d in operator() (this=0x7fffd5edfe40) at /usr/include/c++/4.9/functional:2439
#6  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555555fa7f25 "scheduler", func=...) at util.h:302
#7  0x000055555571d551 in operator()<void (*)(char const*, std::function<void()>), boost::_bi::list0> (f=<optimized out>, a=<synthetic pointer>, 
    this=<optimized out>) at /usr/include/boost/bind/bind.hpp:313
#8  operator() (this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:20
#9  boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > > >::run() (this=0x7fffd8044050) at /usr/include/boost/thread/detail/thread.hpp:117
#10 0x00007ffff72daaea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#11 0x00007ffff4cc3064 in start_thread (arg=0x7fffd5ee0700) at pthread_create.c:309
#12 0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7fffd6731700 (LWP 18025)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00005555555e44cb in boost::condition_variable::wait (this=0x5555562cee68 <scriptcheckqueue+40>, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
#2  0x000055555588dbb0 in CCheckQueue<CScriptCheck>::Loop (this=0x5555562cee40 <scriptcheckqueue>, fMaster=false) at checkqueue.h:102
#3  0x00007ffff72daaea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#4  0x00007ffff4cc3064 in start_thread (arg=0x7fffd6731700) at pthread_create.c:309
#5  0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7fffdd8dc700 (LWP 18024)):
#0  0x00007ffff333caed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff2481ee4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007ffff2481ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff57511cc in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#4  0x00007ffff56f6512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ffff54b6cb4 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ffff54bbd2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007ffff4cc3064 in start_thread (arg=0x7fffdd8dc700) at pthread_create.c:309
#8  0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7fffe9d61700 (LWP 18021)):
#0  0x00007ffff333caed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff0546252 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#2  0x00007ffff0547ddf in xcb_wait_for_event () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#3  0x00007fffec4b41b9 in ?? () from /usr/lib/x86_64-linux-gnu/qt5/plugins/platforms/libqxcb.so
#4  0x00007ffff54bbd2e in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ffff4cc3064 in start_thread (arg=0x7fffe9d61700) at pthread_create.c:309
#6  0x00007ffff334562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7ffff7fb4780 (LWP 18017)):
#0  0x00007ffff3292067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff3293448 in __GI_abort () at abort.c:89
#2  0x00007ffff32d01b4 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7ffff33c5210 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007ffff32d598e in malloc_printerr (action=1, str=0x7ffff33c5318 "double free or corruption (!prev)", ptr=<optimized out>) at malloc.c:4996
#4  0x00007ffff32d6696 in _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
#5  0x00007ffff56c66cd in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ffff56c6a95 in QSortFilterProxyModel::sort(int, Qt::SortOrder) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007ffff68cbff7 in QTableView::setSortingEnabled(bool) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#8  0x00005555556d4f6a in TransactionView::setModel (this=0x555556d47840, _model=_model@entry=0x555556f0dd80) at qt/transactionview.cpp:212
#9  0x0000555555675c3f in WalletView::setWalletModel (this=this@entry=0x5555563997c0, _walletModel=0x555556f0dd80) at qt/walletview.cpp:121
#10 0x00005555556625c5 in WalletFrame::addWallet (this=0x5555569ec4c0, name=..., walletModel=<optimized out>) at qt/walletframe.cpp:49
#11 0x00005555555dc8b1 in BitcoinApplication::initializeResult (this=0x7fffffffe3e0, success=<optimized out>) at qt/bitcoin.cpp:492
#12 0x00007ffff5728eda in QObject::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#13 0x00007ffff5ebe6f2 in QGuiApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5
#14 0x00007ffff665518c in QApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#15 0x00007ffff665212c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#16 0x00007ffff6657600 in QApplication::notify(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#17 0x00007ffff56f8dcb in QCoreApplication::notifyInternal(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#18 0x00007ffff56fadc3 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#19 0x00007ffff5750da3 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#20 0x00007ffff2481c5d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#21 0x00007ffff2481f48 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#22 0x00007ffff2481ffc in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#23 0x00007ffff57511b7 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#24 0x00007ffff56f6512 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#25 0x00007ffff56fe15c in QCoreApplication::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#26 0x00005555555c906e in main (argc=1, argv=<optimized out>) at qt/bitcoin.cpp:714
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe9d61700 (LWP 18041)]
[New Thread 0x7fffdd8dc700 (LWP 18042)]
[New Thread 0x7fffd6731700 (LWP 18043)]
[New Thread 0x7fffd5ee0700 (LWP 18044)]
[New Thread 0x7fffd568f700 (LWP 18045)]
[New Thread 0x7fffd4e3e700 (LWP 18046)]
[New Thread 0x7fffc7fff700 (LWP 18047)]
[New Thread 0x7fffc77ae700 (LWP 18048)]
[New Thread 0x7fffc6f5d700 (LWP 18049)]
[New Thread 0x7fffbce82700 (LWP 18050)]
[New Thread 0x7fffb5038700 (LWP 18051)]
[New Thread 0x7fffabdeb700 (LWP 18052)]
[New Thread 0x7fffab59a700 (LWP 18053)]
[New Thread 0x7fffaab35700 (LWP 18054)]
[New Thread 0x7fffaa2e4700 (LWP 18055)]
[New Thread 0x7fffa9a93700 (LWP 18056)]
[New Thread 0x7fffa8c06700 (LWP 18057)]
[New Thread 0x7fff93fff700 (LWP 18058)]
[Thread 0x7fffab59a700 (LWP 18053) exited]
@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

Another new crash message:

*** Error in '/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt': free(): corrupted unsorted chunks: 0x000055bff44a2750 ***

That's core.bak23 if anyone wants to see the stack traces.

I'm also seeing a lot of:

Segmentation fault (core dumped)

with no other error message (core.bak9, 10, 13, 16, 17, 18, 19, 20, 22, 24).

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

@TheBlueMatt Did you take a look at any of these crashes? You were helpful with the startup race condition errors in the past.

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

I kept running the bitcoin-qt over and over, while also running the 'stop' RPC every 5 seconds. I collected 52 core dumps in total, with 6 different error messages at crash time. They break down like this:

27 times it crashed saying just "Segmentation fault (core dumped)". All 27 of those crashes happened in QSortFilterProxyModel::sort(int, Qt::SortOrder), before the 'stop' RPC was called, with a stack trace that looked like this in all 27 cases:

#9  0x00007f142b329a95 in QSortFilterProxyModel::sort(int, Qt::SortOrder) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#10 0x00007f142c52eff7 in QTableView::setSortingEnabled(bool) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#11 0x0000562b39ce8f6a in TransactionView::setModel (this=0x562b3d01aa50, _model=_model@entry=0x562b3d1f2d80) at qt/transactionview.cpp:212
#12 0x0000562b39c89c3f in WalletView::setWalletModel (this=this@entry=0x562b3c6b4da0, _walletModel=0x562b3d1f2d80) at qt/walletview.cpp:121
#13 0x0000562b39c765c5 in WalletFrame::addWallet (this=0x562b3ccc84c0, name=..., walletModel=<optimized out>) at qt/walletframe.cpp:49
#14 0x0000562b39bf08b1 in BitcoinApplication::initializeResult (this=0x7ffd182aeea0, success=<optimized out>) at qt/bitcoin.cpp:492

So that's by far the most common place it crashed. Other than that I saw:

13 times: *** Error in '.../qt/bitcoin-qt': free(): invalid next size (fast): 0x00005........... ***

4 times: *** Error in '.../qt/bitcoin-qt': malloc(): memory corruption (fast): 0x00005........... ***

twice: *** Error in '.../qt/bitcoin-qt': malloc(): smallbin double linked list corrupted: 0x00005........... ***

once: *** Error in '.../qt/bitcoin-qt': free(): corrupted unsorted chunks: 0x00005........... ***

All of those were before the RPC 'stop' was called too.

Finally, 5 times the client didn't crash until after the RPC 'stop' was called, and in all 5 of those cases it said bitcoin-qt: .../pthread/recursive_mutex.hpp:101: boost::recursive_mutex::~recursive_mutex(): Assertion '!pthread_mutex_destroy(&m)' failed.

@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented Aug 25, 2017

@dooglus, if you run bitcoin with address sanitizer or valgrind, you should be able to get more useful errors. Config options to build bitcoin with address sanitizer look like:

./configure CXX=clang++ CXXFLAGS="-fsanitize=address -fno-omit-frame-pointer" LDFLAGS=-fsanitize=address
@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 25, 2017

@ryanofsky Thanks for that. I've never used clang++ before. Do you know which version I should use?

Debian stable seems to have versions 3.4, 3.5, 3.8, and defaults to 3.5:

$ apt-file search clang++ | grep '/bin/clang..$'
clang: /usr/bin/clang++
clang-3.4: /usr/lib/llvm-3.4/bin/clang++
clang-3.5: /usr/lib/llvm-3.5/bin/clang++
clang-3.8: /usr/lib/llvm-3.8/bin/clang++

$ apt-cache show clang | grep Depends
Depends: clang-3.5 (>= 3.5~)
@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented Aug 25, 2017

I'm guessing any will work, but I'm using clang-3.8.

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Aug 25, 2017

I installed clang-3.8, used the configure line you provided, rebuilt.

Now when I try to run qt/bitcoin-qt it shows the following, and then exits:

=================================================================
==353==ERROR: AddressSanitizer: global-buffer-overflow on address 0x55c46ac552cb at pc 0x55c4699b1942 bp 0x7ffc3ba08bc0 sp 0x7ffc3ba08370
READ of size 12 at 0x55c46ac552cb thread T0
    #0 0x55c4699b1941  (/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt+0x2b6941)
    #1 0x7f83289bb7d1  (/usr/lib/x86_64-linux-gnu/libQt5Core.so.5+0x3177d1)
    #2 0x7f83289bbd7c  (/usr/lib/x86_64-linux-gnu/libQt5Core.so.5+0x317d7c)
    #3 0x7f832897e120  (/usr/lib/x86_64-linux-gnu/libQt5Core.so.5+0x2da120)
    #4 0x55c469c3f724  (/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt+0x544724)
    #5 0x55c469c1ae77  (/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt+0x51fe77)
    #6 0x55c469be9f9d  (/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt+0x4eef9d)
    #7 0x55c469a89431  (/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt+0x38e431)
    #8 0x55c469a3262a  (/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt+0x33762a)
    #9 0x55c469a36185  (/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt+0x33b185)
    #10 0x7f83260f7b44  (/lib/x86_64-linux-gnu/libc.so.6+0x21b44)
    #11 0x55c4699532c7  (/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt+0x2582c7)

0x55c46ac552cb is located 53 bytes to the left of global variable '<string literal>' defined in './qt/forms/ui_debugwindow.h:913:74' (0x55c46ac55300) of size 13
  '<string literal>' is ascii string 'Debug window'
0x55c46ac552cb is located 0 bytes to the right of global variable '<string literal>' defined in './qt/forms/ui_debugwindow.h:913:60' (0x55c46ac552c0) of size 11
  '<string literal>' is ascii string 'RPCConsole'
SUMMARY: AddressSanitizer: global-buffer-overflow (/home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt+0x2b6941) 
Shadow bytes around the buggy address:
  0x0ab90d582a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ab90d582a10: 00 00 f9 f9 f9 f9 f9 f9 00 00 00 04 f9 f9 f9 f9
  0x0ab90d582a20: 06 f9 f9 f9 f9 f9 f9 f9 00 00 00 00 00 00 00 00
  0x0ab90d582a30: f9 f9 f9 f9 00 00 00 00 00 00 06 f9 f9 f9 f9 f9
  0x0ab90d582a40: 00 00 01 f9 f9 f9 f9 f9 00 04 f9 f9 f9 f9 f9 f9
=>0x0ab90d582a50: 00 00 03 f9 f9 f9 f9 f9 00[03]f9 f9 f9 f9 f9 f9
  0x0ab90d582a60: 00 05 f9 f9 f9 f9 f9 f9 00 f9 f9 f9 f9 f9 f9 f9
  0x0ab90d582a70: 00 07 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
  0x0ab90d582a80: 00 03 f9 f9 f9 f9 f9 f9 00 00 00 01 f9 f9 f9 f9
  0x0ab90d582a90: 00 f9 f9 f9 f9 f9 f9 f9 00 05 f9 f9 f9 f9 f9 f9
  0x0ab90d582aa0: 00 f9 f9 f9 f9 f9 f9 f9 05 f9 f9 f9 f9 f9 f9 f9
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==353==ABORTING

I ran it a bunch of times and the output was pretty much the same every time, other than the addresses changing.

Running the bitcoind that I just built with clang works as usual.

@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented Aug 25, 2017

That looks promising. You could also decode the stack trace with:

addr2line -f -e /home/chris/Programs/bitcoin.git/src/qt/bitcoin-q 0x2b6941 0x544724 0x51fe77 0x4eef9d 0x38e431 0x33762a 0x33b185 0x2582c7
@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Feb 18, 2018

Compiling with ./configure CFLAGS="-O0 -ggdb -g3" CXXFLAGS="-O0 -ggdb -g3" should give me a nice core dump...

@eklitzke

This comment has been minimized.

Copy link
Member

commented Mar 10, 2018

FYI you can use --enable-debug --disable-hardening to enable the full debug flags (including -O0) without having to manually set CFLAGS and CXXFLAGS.

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Mar 12, 2018

I tried compiling qt in depends with clang and asan flags, but I can't figure out what flags to pass to ./configure to properly link everything.
See 4606723#diff-5e50ddee0f3ab9f50378a3577075f10f for my WIP.

Just using the ./configure CC=clang CXX=clang++ --enable-debug --disable-asm --disable-hardening --prefix=$(pwd)/depends/x86_64-pc-linux-gnu gives a linker error:

/bitcoin/depends/x86_64-pc-linux-gnu/share/../lib/libdb_cxx-4.8.a(cxx_txn.o): In function `asan.module_ctor':
/bitcoin/depends/work/build/x86_64-pc-linux-gnu/bdb/4.8.30-3f31ffdae35/build_unix/../dist/./../cxx/cxx_txn.cpp:61: undefined reference to `__asan_init_v3'
collect2: error: ld returned 1 exit status
make[2]: *** [bitcoind] Error 1

@bitcoin bitcoin deleted a comment from Dan04ka90 Mar 17, 2018

@eklitzke

This comment has been minimized.

Copy link
Member

commented Mar 18, 2018

@MarcoFalke It looks like you're using GCC and don't have libasan installed which is kind of a strange edge case I am trying to handle in #12692.

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Dec 11, 2018

I built using clang++ and asan flags as suggested above:

./configure CXX=clang++ CXXFLAGS="-fsanitize=address -fno-omit-frame-pointer" LDFLAGS=-fsanitize=address --enable-debug --with-incompatible-bdb --with-miniupnpc=no

I built from the current master branch, commit 5f23460, Date: Mon Dec 10 14:02:15 2018 -0500, "Merge #14877: rpc: Document default values for optional arguments".

I ran bitcoin-qt in gdb in a loop until it failed:

while [[ ! -f /tmp/stop ]]; do date; gdb -ex "run -datadir=/home/me/.bitcoin-prune" -ex "quit" qt/bitcoin-qt; done

Pretty quickly it failed:

Thread 1 "bitcoin-qt" received signal SIGSEGV, Segmentation fault.
0x00007ffff6765972 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
A debugging session is active.

	Inferior 1 [process 28586] will be killed.

Quit anyway? (y or n) n
Not confirmed.
(gdb) set pagination off
(gdb) where
#0  0x00007ffff6765972 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#1  0x00007ffff676aca9 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#2  0x00007ffff7300737 in QTableView::setSortingEnabled(bool) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#3  0x0000555555c7810c in TransactionView::setModel (this=0x60f00003b230, _model=0x610000434d40) at qt/transactionview.cpp:232
#4  0x0000555555ad425e in WalletView::setWalletModel (this=0x60d0073c1910, _walletModel=0x610000434d40) at qt/walletview.cpp:128
#5  0x0000555555a9aded in WalletFrame::addWallet (this=0x60800005eb20, walletModel=0x610000434d40) at qt/walletframe.cpp:57
#6  0x00005555558dafcb in BitcoinGUI::addWallet (this=0x614000023440, walletModel=0x610000434d40) at qt/bitcoingui.cpp:531
#7  0x000055555588234a in BitcoinApplication::addWallet (this=0x7fffffffd430, walletModel=0x610000434d40) at qt/bitcoin.cpp:431
#8  0x00005555558814b5 in BitcoinApplication::initializeResult (this=0x7fffffffd430, success=true) at qt/bitcoin.cpp:485
#9  0x00005555558ae1fc in QtPrivate::FunctorCall<QtPrivate::IndexesList<0>, QtPrivate::List<bool>, void, void (BitcoinApplication::*)(bool)>::call (f=(void (BitcoinApplication::*)(BitcoinApplication * const, bool)) 0x5555558807a0 <BitcoinApplication::initializeResult(bool)>, o=0x7fffffffd430, arg=0x602000332110) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:141
#10 0x00005555558ade62 in QtPrivate::FunctionPointer<void (BitcoinApplication::*)(bool)>::call<QtPrivate::List<bool>, void> (f=(void (BitcoinApplication::*)(BitcoinApplication * const, bool)) 0x5555558807a0 <BitcoinApplication::initializeResult(bool)>, o=0x7fffffffd430, arg=0x602000332110) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:160
#11 0x00005555558ada37 in QtPrivate::QSlotObject<void (BitcoinApplication::*)(bool), QtPrivate::List<bool>, void>::impl (which=1, this_=0x6030001b2830, r=0x7fffffffd430, a=0x602000332110, ret=0x0) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobject_impl.h:120
#12 0x00007ffff67cd499 in QObject::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#13 0x00007ffff6ae4d54 in QGuiApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5
#14 0x00007ffff708992f in QApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#15 0x00007ffff7083b8c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#16 0x00007ffff708b341 in QApplication::notify(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#17 0x00007ffff67a09e0 in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#18 0x00007ffff67a316d in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#19 0x00007ffff67f4c43 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#20 0x00007ffff21db7f7 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#21 0x00007ffff21dba60 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#22 0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#23 0x00007ffff67f504f in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#24 0x00007ffff679e9ca in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#25 0x00007ffff67a713c in QCoreApplication::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#26 0x0000555555889ee6 in main (argc=2, argv=0x7fffffffe4c8) at qt/bitcoin.cpp:714
(gdb) thread apply all where

Thread 22 (Thread 0x7fff5b907700 (LWP 28633)):
#0  0x00007ffff3d9567d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00005555557c1c54 in poll () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#2  0x00007ffff21db9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007ffff67f504f in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ffff679e9ca in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ffff65cc0f3 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007ffff65d0da8 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#8  0x00007ffff5c02494 in start_thread (arg=0x7fff5b907700) at pthread_create.c:333
#9  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 21 (Thread 0x7fff5c36c700 (LWP 28632)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000555555d39dce in __gthread_cond_timedwait (__cond=0x7fffd4d1bc48, __mutex=0x7fffd4d1bc78, __abs_timeout=0x7fff5c36ade0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:871
#2  0x0000555555dab36f in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fffd4d1bc48, __lock=..., __atime=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:166
#3  0x0000555555e6a500 in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fffd4d1bc48, __lock=..., __atime=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:119
#4  0x0000555555de1d2f in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, CConnman::ThreadMessageHandler()::$_0>(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, CConnman::ThreadMessageHandler()::$_0) (this=0x7fffd4d1bc48, __lock=..., __atime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:129
#5  0x0000555555de19ec in CConnman::ThreadMessageHandler (this=0x7fffd4ccb800) at net.cpp:2077
#6  0x0000555555e70f7e in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=@0x603000e34040: (void (CConnman::*)(CConnman * const)) 0x555555de0de0 <CConnman::ThreadMessageHandler()>, __t=@0x603000e34050: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#7  0x0000555555e70dff in std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=@0x603000e34040: (void (CConnman::*)(CConnman * const)) 0x555555de0de0 <CConnman::ThreadMessageHandler()>, __args=@0x603000e34050: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#8  0x0000555555e70cf9 in std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=0x603000e34040, __args=@0x603000e34050: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#9  0x0000555555e70c17 in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x603000e34040, __args=<unknown type in /home/me/Programs/bitcoin/src/qt/bitcoin-qt, CU 0xd1122e, DIE 0xd9b04a>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#10 0x0000555555e70a3d in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=0x603000e34040) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#11 0x0000555555e70476 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#12 0x0000555555a3c0d6 in std::function<void ()>::operator()() const (this=0x7fff5c36bba0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#13 0x0000555555d423df in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555557360de0 <.str.115> "msghand", func=...) at ./util/system.h:331
#14 0x0000555555e71f7c in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=0x606003caa248) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#15 0x0000555555e71dce in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=0x606003caa248) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#16 0x0000555555e71652 in std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=0x606003caa240) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#17 0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#18 0x00007ffff5c02494 in start_thread (arg=0x7fff5c36c700) at pthread_create.c:333
#19 0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 20 (Thread 0x7fff5cbbd700 (LWP 28631)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000555555d39dce in __gthread_cond_timedwait (__cond=0x7fffd4d1bca8, __mutex=0x7fffd4d1bcd8, __abs_timeout=0x7fff5cbbb540) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:871
#2  0x0000555555dab36f in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fffd4d1bca8, __lock=..., __atime=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:166
#3  0x0000555555daadae in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fffd4d1bca8, __lock=..., __atime=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:106
#4  0x0000555556750a1f in std::condition_variable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=0x7fffd4d1bca8, __lock=..., __atime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:129
#5  0x000055555675029b in std::condition_variable::wait_for<long, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=0x7fffd4d1bca8, __lock=..., __rtime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:145
#6  0x000055555674ff86 in CThreadInterrupt::sleep_for (this=0x7fffd4d1bca8, rel_time=...) at threadinterrupt.cpp:34
#7  0x0000555555ddd703 in CConnman::ThreadOpenConnections (this=0x7fffd4ccb800, connect=std::vector of length 1, capacity 1 = {...}) at net.cpp:1766
#8  0x0000555555e74bcb in std::__invoke_impl<void, void (CConnman::* const&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (__f=@0x604005c85e10: (void (CConnman::*)(CConnman * const, std::vector<std::__cxx11::basic_string<char>, std::allocator<std::__cxx11::basic_string<char> > >)) 0x555555ddcf30 <CConnman::ThreadOpenConnections(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)>, __t=@0x604005c85e38: 0x7fffd4ccb800, __args=std::vector of length 1, capacity 1 = {...}) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#9  0x0000555555e74914 in std::__invoke<void (CConnman::* const&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (__fn=@0x604005c85e10: (void (CConnman::*)(CConnman * const, std::vector<std::__cxx11::basic_string<char>, std::allocator<std::__cxx11::basic_string<char> > >)) 0x555555ddcf30 <CConnman::ThreadOpenConnections(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)>, __args=std::vector of length 1, capacity 1 = {...}, __args=std::vector of length 1, capacity 1 = {...}) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#10 0x0000555555e747b2 in std::_Mem_fn_base<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), true>::operator()<CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (this=0x604005c85e10, __args=std::vector of length 1, capacity 1 = {...}, __args=std::vector of length 1, capacity 1 = {...}) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#11 0x0000555555e746ef in std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=0x604005c85e10, __args=<unknown type in /home/me/Programs/bitcoin/src/qt/bitcoin-qt, CU 0xd1122e, DIE 0xd9c850>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#12 0x0000555555e744dd in std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)>::operator()<, void>() (this=0x604005c85e10) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#13 0x0000555555e73ce6 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#14 0x0000555555a3c0d6 in std::function<void ()>::operator()() const (this=0x7fff5cbbcba0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#15 0x0000555555d423df in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555557360da0 <.str.114> "opencon", func=...) at ./util/system.h:331
#16 0x0000555555e71f7c in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=0x606003caa2a8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#17 0x0000555555e71dce in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=0x606003caa2a8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#18 0x0000555555e71652 in std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=0x606003caa2a0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#19 0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#20 0x00007ffff5c02494 in start_thread (arg=0x7fff5cbbd700) at pthread_create.c:333
#21 0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 19 (Thread 0x7fff5d40e700 (LWP 28630)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000555555d39dce in __gthread_cond_timedwait (__cond=0x7fffd4d1bca8, __mutex=0x7fffd4d1bcd8, __abs_timeout=0x7fff5d40cca0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:871
#2  0x0000555555dab36f in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fffd4d1bca8, __lock=..., __atime=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:166
#3  0x0000555555daadae in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fffd4d1bca8, __lock=..., __atime=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:106
#4  0x0000555556750a1f in std::condition_variable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=0x7fffd4d1bca8, __lock=..., __atime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:129
#5  0x000055555675029b in std::condition_variable::wait_for<long, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=0x7fffd4d1bca8, __lock=..., __rtime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:145
#6  0x000055555674ff86 in CThreadInterrupt::sleep_for (this=0x7fffd4d1bca8, rel_time=...) at threadinterrupt.cpp:34
#7  0x00005555567504b0 in CThreadInterrupt::sleep_for (this=0x7fffd4d1bca8, rel_time=...) at threadinterrupt.cpp:39
#8  0x0000555555de0c8c in CConnman::ThreadOpenAddedConnections (this=0x7fffd4ccb800) at net.cpp:1989
#9  0x0000555555e70f7e in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=@0x603000e340a0: (void (CConnman::*)(CConnman * const)) 0x555555de0690 <CConnman::ThreadOpenAddedConnections()>, __t=@0x603000e340b0: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#10 0x0000555555e70dff in std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=@0x603000e340a0: (void (CConnman::*)(CConnman * const)) 0x555555de0690 <CConnman::ThreadOpenAddedConnections()>, __args=@0x603000e340b0: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#11 0x0000555555e70cf9 in std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=0x603000e340a0, __args=@0x603000e340b0: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#12 0x0000555555e70c17 in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x603000e340a0, __args=<unknown type in /home/me/Programs/bitcoin/src/qt/bitcoin-qt, CU 0xd1122e, DIE 0xd9b04a>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#13 0x0000555555e70a3d in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=0x603000e340a0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#14 0x0000555555e70476 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#15 0x0000555555a3c0d6 in std::function<void ()>::operator()() const (this=0x7fff5d40dba0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#16 0x0000555555d423df in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555557360ce0 <.str.112> "addcon", func=...) at ./util/system.h:331
#17 0x0000555555e71f7c in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=0x606003caa308) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#18 0x0000555555e71dce in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=0x606003caa308) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#19 0x0000555555e71652 in std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=0x606003caa300) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#20 0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#21 0x00007ffff5c02494 in start_thread (arg=0x7fff5d40e700) at pthread_create.c:333
#22 0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 18 (Thread 0x7fff5dc5f700 (LWP 28629)):
#0  0x00007ffff3d973c3 in select () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000555555dd720e in CConnman::SocketHandler (this=0x7fffd4ccb800) at net.cpp:1324
#2  0x0000555555dd9a6d in CConnman::ThreadSocketHandler (this=0x7fffd4ccb800) at net.cpp:1466
#3  0x0000555555e70f7e in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=@0x603000e34160: (void (CConnman::*)(CConnman * const)) 0x555555dd9a10 <CConnman::ThreadSocketHandler()>, __t=@0x603000e34170: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#4  0x0000555555e70dff in std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=@0x603000e34160: (void (CConnman::*)(CConnman * const)) 0x555555dd9a10 <CConnman::ThreadSocketHandler()>, __args=@0x603000e34170: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#5  0x0000555555e70cf9 in std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=0x603000e34160, __args=@0x603000e34170: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#6  0x0000555555e70c17 in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x603000e34160, __args=<unknown type in /home/me/Programs/bitcoin/src/qt/bitcoin-qt, CU 0xd1122e, DIE 0xd9b04a>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#7  0x0000555555e70a3d in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=0x603000e34160) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#8  0x0000555555e70476 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#9  0x0000555555a3c0d6 in std::function<void ()>::operator()() const (this=0x7fff5dc5eba0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#10 0x0000555555d423df in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555557360be0 <.str.108> "net", func=...) at ./util/system.h:331
#11 0x0000555555e71f7c in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=0x606003caa368) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#12 0x0000555555e71dce in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=0x606003caa368) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#13 0x0000555555e71652 in std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=0x606003caa360) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#14 0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007ffff5c02494 in start_thread (arg=0x7fff5dc5f700) at pthread_create.c:333
#16 0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 17 (Thread 0x7fffcf555700 (LWP 28628)):
#0  COutPoint::Serialize<CHashWriter> (this=0x7fffcf54d0b0, s=...) at ./primitives/transaction.h:29
#1  0x0000555556c6098c in Serialize<CHashWriter, COutPoint> (os=..., a=...) at ./serialize.h:591
#2  0x0000555556c75512 in (anonymous namespace)::CTransactionSignatureSerializer<CTransaction>::SerializeInput<CHashWriter> (this=0x7fffcf54d070, s=..., nInput=32) at script/interpreter.cpp:1140
#3  0x0000555556c75031 in (anonymous namespace)::CTransactionSignatureSerializer<CTransaction>::Serialize<CHashWriter> (this=0x7fffcf54d070, s=...) at script/interpreter.cpp:1174
#4  0x0000555556c74e9c in Serialize<CHashWriter, (anonymous namespace)::CTransactionSignatureSerializer<CTransaction> > (os=..., a=...) at ./serialize.h:591
#5  0x0000555556c74e4d in CHashWriter::operator<< <(anonymous namespace)::CTransactionSignatureSerializer<CTransaction> > (this=0x7fffcf54d0b0, obj=...) at ./hash.h:154
#6  0x0000555556c7d759 in SignatureHash<CTransaction> (scriptCode=..., txTo=..., nIn=36, nHashType=1, amount=@0x7fffcf550318: 10614, sigversion=SigVersion::BASE, cache=0x7fffcf552770) at script/interpreter.cpp:1299
#7  0x0000555556c7c976 in GenericTransactionSignatureChecker<CTransaction>::CheckSig (this=0x7fffcf550300, vchSigIn=std::vector of length 71, capacity 71 = {...}, vchPubKey=std::vector of length 33, capacity 33 = {...}, scriptCode=..., sigversion=SigVersion::BASE) at script/interpreter.cpp:1323
#8  0x0000555556c6f78a in EvalScript (stack=std::vector of length 8, capacity 8 = {...}, script=..., flags=131039, checker=..., sigversion=SigVersion::BASE, serror=0x7fffcf55061c) at script/interpreter.cpp:1016
#9  0x0000555556c720ad in VerifyScript (scriptSig=..., scriptPubKey=..., witness=0x62300007b6c8, flags=131039, checker=..., serror=0x7fffcf55061c) at script/interpreter.cpp:1538
#10 0x000055555642bb14 in CScriptCheck::operator() (this=0x7fffcf5505e0) at validation.cpp:1334
#11 0x000055555642d0af in CheckInputs (tx=..., state=..., inputs=..., fScriptChecks=true, flags=131039, cacheSigStore=true, cacheFullScriptStore=false, txdata=..., pvChecks=0x0) at validation.cpp:1419
#12 0x0000555556470666 in AcceptToMemoryPoolWorker (chainparams=..., pool=..., state=..., ptx=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d0073ce6b0, pfMissingInputs=0x0, nAcceptTime=1544565997, plTxnReplaced=0x0, bypass_limits=false, nAbsurdFee=@0x7fffcf553ae0: 0, coins_to_uncache=std::vector of length 70, capacity 128 = {...}, test_accept=false) at validation.cpp:893
#13 0x00005555564277e6 in AcceptToMemoryPoolWithTime (chainparams=..., pool=..., state=..., tx=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d0073ce6b0, pfMissingInputs=0x0, nAcceptTime=1544565997, plTxnReplaced=0x0, bypass_limits=false, nAbsurdFee=0, test_accept=false) at validation.cpp:974
#14 0x0000555556467947 in LoadMempool () at validation.cpp:4723
#15 0x0000555555d37d55 in ThreadImport (vImportFiles=std::vector of length 0, capacity 0) at init.cpp:700
#16 0x0000555555daa2ae in boost::_bi::list1<boost::_bi::value<std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> > > >::operator()<void (*)(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >), boost::_bi::list0> (this=0x61500015b540, f=@0x61500015b538: 0x555555d37090 <ThreadImport(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >)>, a=...) at /usr/include/boost/bind/bind.hpp:259
#17 0x0000555555daa0c9 in boost::_bi::bind_t<void, void (*)(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >), boost::_bi::list1<boost::_bi::value<std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> > > > >::operator() (this=0x61500015b538) at /usr/include/boost/bind/bind.hpp:1294
#18 0x0000555555da9925 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >), boost::_bi::list1<boost::_bi::value<std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> > > > > >::run (this=0x61500015b380) at /usr/include/boost/thread/detail/thread.hpp:116
#19 0x00007ffff77a6116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
#20 0x00007ffff5c02494 in start_thread (arg=0x7fffcf555700) at pthread_create.c:333
#21 0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 16 (Thread 0x7fffcece1700 (LWP 28619)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000555556e9fa2e in leveldb::(anonymous namespace)::PosixEnv::BGThread (this=0x61300005f100) at leveldb/util/env_posix.cc:649
#2  0x0000555556e9f75e in leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x61300005f100) at leveldb/util/env_posix.cc:564
#3  0x00007ffff5c02494 in start_thread (arg=0x7fffcece1700) at pthread_create.c:333
#4  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 15 (Thread 0x7fffd660e700 (LWP 28605)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff4a3050c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005555566a8bd0 in WorkQueue<HTTPClosure>::Run (this=0x61000003fe40) at httpserver.cpp:107
#3  0x0000555556692dae in HTTPWorkQueueRun (queue=0x61000003fe40) at httpserver.cpp:342
#4  0x00005555566ae58c in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x6030001c5978) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x00005555566ae4ce in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=0x6030001c5978) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x00005555566adf92 in std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0x6030001c5970) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffd660e700) at pthread_create.c:333
#9  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 14 (Thread 0x7fffd6e82700 (LWP 28604)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff4a3050c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005555566a8bd0 in WorkQueue<HTTPClosure>::Run (this=0x61000003fe40) at httpserver.cpp:107
#3  0x0000555556692dae in HTTPWorkQueueRun (queue=0x61000003fe40) at httpserver.cpp:342
#4  0x00005555566ae58c in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x6030001c59a8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x00005555566ae4ce in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=0x6030001c59a8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x00005555566adf92 in std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0x6030001c59a0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffd6e82700) at pthread_create.c:333
#9  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 13 (Thread 0x7fffd76f6700 (LWP 28603)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff4a3050c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005555566a8bd0 in WorkQueue<HTTPClosure>::Run (this=0x61000003fe40) at httpserver.cpp:107
#3  0x0000555556692dae in HTTPWorkQueueRun (queue=0x61000003fe40) at httpserver.cpp:342
#4  0x00005555566ae58c in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x6030001c5a08) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x00005555566ae4ce in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=0x6030001c5a08) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x00005555566adf92 in std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0x6030001c5a00) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffd76f6700) at pthread_create.c:333
#9  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 12 (Thread 0x7fffd7f6a700 (LWP 28602)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff4a3050c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005555566a8bd0 in WorkQueue<HTTPClosure>::Run (this=0x61000003fe40) at httpserver.cpp:107
#3  0x0000555556692dae in HTTPWorkQueueRun (queue=0x61000003fe40) at httpserver.cpp:342
#4  0x00005555566ae58c in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x6030001c5a38) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x00005555566ae4ce in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=0x6030001c5a38) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x00005555566adf92 in std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0x6030001c5a30) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffd7f6a700) at pthread_create.c:333
#9  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 11 (Thread 0x7fffd87de700 (LWP 28601)):
#0  0x00007ffff3d9f0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007ffff53efd98 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#2  0x00007ffff53da07a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#3  0x0000555556692ca0 in ThreadHTTP (base=0x6160000ab980) at httpserver.cpp:290
#4  0x00005555566acdac in std::_Bind_simple<bool (*(event_base*))(event_base*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x6030001c5a68) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x00005555566accde in std::_Bind_simple<bool (*(event_base*))(event_base*)>::operator()() (this=0x6030001c5a68) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x00005555566ac792 in std::thread::_State_impl<std::_Bind_simple<bool (*(event_base*))(event_base*)> >::_M_run() (this=0x6030001c5a60) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffd87de700) at pthread_create.c:333
#9  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 10 (Thread 0x7fffd9052700 (LWP 28600)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ffff5c04c06 in __GI___pthread_mutex_lock (mutex=0x555557b9cae0 <cs_main>) at ../nptl/pthread_mutex_lock.c:115
#2  0x0000555555cf8290 in __gthread_mutex_lock (__mutex=0x555557b9cae0 <cs_main>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:748
#3  0x0000555555cf822e in __gthread_recursive_mutex_lock (__mutex=0x555557b9cae0 <cs_main>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:810
#4  0x0000555555d015d5 in std::recursive_mutex::lock (this=0x555557b9cae0 <cs_main>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/mutex:105
#5  0x0000555555d01534 in std::unique_lock<std::recursive_mutex>::lock (this=0x60300be2ad58) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/std_mutex.h:267
#6  0x0000555555d01006 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::Enter (this=0x60300be2ad58, pszName=0x555557441940 <.str.4> "cs_main", pszFile=0x555557441980 <.str.5> "interfaces/chain.cpp", nLine=31) at ./sync.h:128
#7  0x0000555555cfdc71 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::UniqueLock (this=0x60300be2ad58, mutexIn=..., pszName=0x555557441940 <.str.4> "cs_main", pszFile=0x555557441980 <.str.5> "interfaces/chain.cpp", nLine=31, fTry=false) at ./sync.h:149
#8  0x00005555566cf1c2 in interfaces::(anonymous namespace)::LockingStateImpl::LockingStateImpl (this=0x60300be2ad50) at interfaces/chain.cpp:23
#9  0x00005555566ced2a in MakeUnique<interfaces::(anonymous namespace)::LockingStateImpl, AnnotatedMixin<std::recursive_mutex>&, char const (&) [8], char const (&) [21], int, bool&>(AnnotatedMixin<std::recursive_mutex>&, char const (&) [8], char const (&) [21], int&&, bool&) (args=@0x7fffd904fc80: false, args=@0x7fffd904fc80: false, args=@0x7fffd904fc80: false, args=@0x7fffd904fc80: false, args=@0x7fffd904fc80: false) at ./util/memory.h:16
#10 0x00005555566ce8ba in interfaces::(anonymous namespace)::ChainImpl::lock (this=0x60200000de10, try_lock=false) at interfaces/chain.cpp:31
#11 0x000055555692c8d2 in CWallet::TransactionAddedToMempool (this=0x61a0000a6e80, ptx=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d008383060) at wallet/wallet.cpp:1147
#12 0x000055555692cb6f in non-virtual thunk to CWallet::TransactionAddedToMempool(std::shared_ptr<CTransaction const> const&) () at wallet/wallet.cpp:1146
#13 0x000055555659f6f0 in boost::_mfi::mf1<void, CValidationInterface, std::shared_ptr<CTransaction const> const&>::operator() (this=0x6060036062e0, p=0x61a0000a6fd8, a1=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d008383060) at /usr/include/boost/bind/mem_fn_template.hpp:165
#14 0x000055555659f590 in boost::_bi::list2<boost::_bi::value<CValidationInterface*>, boost::arg<1> >::operator()<boost::_mfi::mf1<void, CValidationInterface, std::shared_ptr<CTransaction const> const&>, boost::_bi::rrlist1<std::shared_ptr<CTransaction const> const&> > (this=0x6060036062f0, f=..., a=...) at /usr/include/boost/bind/bind.hpp:319
#15 0x000055555659f3c2 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, CValidationInterface, std::shared_ptr<CTransaction const> const&>, boost::_bi::list2<boost::_bi::value<CValidationInterface*>, boost::arg<1> > >::operator()<std::shared_ptr<CTransaction const> const&> (this=0x6060036062e0, a1=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d008383060) at /usr/include/boost/bind/bind.hpp:1306
#16 0x000055555659e8ae in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, CValidationInterface, std::shared_ptr<CTransaction const> const&>, boost::_bi::list2<boost::_bi::value<CValidationInterface*>, boost::arg<1> > >, void, std::shared_ptr<CTransaction const> const&>::invoke (function_obj_ptr=..., a0=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d008383060) at /usr/include/boost/function/function_template.hpp:159
#17 0x000055555660b7d0 in boost::function1<void, std::shared_ptr<CTransaction const> const&>::operator() (this=0x6060036062d8, a0=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d008383060) at /usr/include/boost/function/function_template.hpp:770
#18 0x000055555660b615 in boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::m_invoke<boost::function<void (std::shared_ptr<CTransaction const> const&)>, 0u, std::shared_ptr<CTransaction const> const&>(boost::function<void (std::shared_ptr<CTransaction const> const&)>&, boost::signals2::detail::unsigned_meta_array<0u>, std::tuple<std::shared_ptr<CTransaction const> const&> const&, boost::enable_if<boost::is_void<boost::function<void (std::shared_ptr<CTransaction const> const&)>::result_type>, void>::type*) const (this=0x7fffd9050360, func=..., args=std::tuple containing = {...}) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:105
#19 0x000055555660b5b0 in boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::operator()<boost::function<void (std::shared_ptr<CTransaction const> const&)>, std::shared_ptr<CTransaction const> const&, 1ul>(boost::function<void (std::shared_ptr<CTransaction const> const&)>&, std::tuple<std::shared_ptr<CTransaction const> const&> const&, mpl_::size_t<1ul>) const (this=0x7fffd9050360, func=..., args=std::tuple containing = {...}) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
#20 0x000055555660b514 in boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>::operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >(boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > const&) const (this=0x7fffd9050ad0, connectionBody=...) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:133
#21 0x000055555660b277 in boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> >::dereference() const (this=0x7fffd9050660) at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:110
#22 0x000055555660b03e in boost::iterators::iterator_core_access::dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > const&) (f=...) at /usr/include/boost/iterator/iterator_facade.hpp:549
#23 0x000055555660aef6 in boost::iterators::detail::iterator_facade_base<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> >, boost::signals2::detail::void_type, boost::iterators::single_pass_traversal_tag, boost::signals2::detail::void_type&, long, false, false>::operator*() const (this=0x7fffd9050660) at /usr/include/boost/iterator/iterator_facade.hpp:655
#24 0x000055555660ace2 in boost::signals2::optional_last_value<void>::operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> >) const (this=0x602000149f10, first=..., last=...) at /usr/include/boost/signals2/optional_last_value.hpp:57
#25 0x000055555660a356 in boost::signals2::detail::combiner_invoker<void>::operator()<boost::signals2::optional_last_value<void>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >(boost::signals2::optional_last_value<void>&, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> >) const (this=0x7fffd9050b70, combiner=..., first=..., last=...) at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:64
#26 0x0000555556609b6e in boost::signals2::detail::signal_impl<void (std::shared_ptr<CTransaction const> const&), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (std::shared_ptr<CTransaction const> const&)>, boost::function<void (boost::signals2::connection const&, std::shared_ptr<CTransaction const> const&)>, boost::signals2::mutex>::operator()(std::shared_ptr<CTransaction const> const&) (this=0x6040000bdc50, args=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d008383060) at /usr/include/boost/signals2/detail/signal_template.hpp:242
#27 0x00005555566093c9 in boost::signals2::signal<void (std::shared_ptr<CTransaction const> const&), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (std::shared_ptr<CTransaction const> const&)>, boost::function<void (boost::signals2::connection const&, std::shared_ptr<CTransaction const> const&)>, boost::signals2::mutex>::operator()(std::shared_ptr<CTransaction const> const&) (this=0x6120001d3ed8, args=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d008383060) at /usr/include/boost/signals2/detail/signal_template.hpp:723
#28 0x000055555652d2a5 in CMainSignals::TransactionAddedToMempool(std::shared_ptr<CTransaction const> const&)::$_3::operator()() const (this=0x603006171eb0) at validationinterface.cpp:147
#29 0x000055555652cec6 in std::_Function_handler<void (), CMainSignals::TransactionAddedToMempool(std::shared_ptr<CTransaction const> const&)::$_3>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#30 0x0000555555a3c0d6 in std::function<void ()>::operator()() const (this=0x7fffd9050f60) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#31 0x0000555556bc2a50 in SingleThreadedSchedulerClient::ProcessQueue (this=0x6120001d3f98) at scheduler.cpp:187
#32 0x0000555556bcda6e in std::__invoke_impl<void, void (SingleThreadedSchedulerClient::* const&)(), SingleThreadedSchedulerClient*&> (__f=@0x60300be2acc0: (void (SingleThreadedSchedulerClient::*)(SingleThreadedSchedulerClient * const)) 0x555556bc27a0 <SingleThreadedSchedulerClient::ProcessQueue()>, __t=@0x60300be2acd0: 0x6120001d3f98) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#33 0x0000555556bcd8ef in std::__invoke<void (SingleThreadedSchedulerClient::* const&)(), SingleThreadedSchedulerClient*&> (__fn=@0x60300be2acc0: (void (SingleThreadedSchedulerClient::*)(SingleThreadedSchedulerClient * const)) 0x555556bc27a0 <SingleThreadedSchedulerClient::ProcessQueue()>, __args=@0x60300be2acd0: 0x6120001d3f98) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#34 0x0000555556bcd7e9 in std::_Mem_fn_base<void (SingleThreadedSchedulerClient::*)(), true>::operator()<SingleThreadedSchedulerClient*&> (this=0x60300be2acc0, __args=@0x60300be2acd0: 0x6120001d3f98) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#35 0x0000555556bcd747 in std::_Bind<std::_Mem_fn<void (SingleThreadedSchedulerClient::*)()> (SingleThreadedSchedulerClient*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x60300be2acc0, __args=<unknown type in /home/me/Programs/bitcoin/src/qt/bitcoin-qt, CU 0x1ddf3c9, DIE 0x1df6c36>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#36 0x0000555556bcd56d in std::_Bind<std::_Mem_fn<void (SingleThreadedSchedulerClient::*)()> (SingleThreadedSchedulerClient*)>::operator()<, void>() (this=0x60300be2acc0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#37 0x0000555556bccf66 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (SingleThreadedSchedulerClient::*)()> (SingleThreadedSchedulerClient*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#38 0x0000555555a3c0d6 in std::function<void ()>::operator()() const (this=0x7fffd90514a0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#39 0x0000555556bc0d18 in CScheduler::serviceQueue (this=0x5555588f3a40 <scheduler>) at scheduler.cpp:83
#40 0x0000555555d9f438 in boost::_mfi::mf0<void, CScheduler>::operator() (this=0x6030001c7fb0, p=0x5555588f3a40 <scheduler>) at /usr/include/boost/bind/mem_fn_template.hpp:49
#41 0x0000555555d9f2e2 in boost::_bi::list1<boost::_bi::value<CScheduler*> >::operator()<boost::_mfi::mf0<void, CScheduler>, boost::_bi::list0> (this=0x6030001c7fc0, f=..., a=...) at /usr/include/boost/bind/bind.hpp:259
#42 0x0000555555d9f1a9 in boost::_bi::bind_t<void, boost::_mfi::mf0<void, CScheduler>, boost::_bi::list1<boost::_bi::value<CScheduler*> > >::operator() (this=0x6030001c7fb0) at /usr/include/boost/bind/bind.hpp:1294
#43 0x0000555555d9ede6 in std::_Function_handler<void (), boost::_bi::bind_t<void, boost::_mfi::mf0<void, CScheduler>, boost::_bi::list1<boost::_bi::value<CScheduler*> > > >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#44 0x0000555555a3c0d6 in std::function<void ()>::operator()() const (this=0x7fffd9051aa0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#45 0x0000555555d423df in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555557353e00 <.str> "scheduler", func=...) at ./util/system.h:331
#46 0x0000555555da0e38 in boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > >::operator()<void (*)(char const*, std::function<void ()>), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(char const*, std::function<void ()>), boost::_bi::list0&, int) (this=0x6150000edf40, f=@0x6150000edf38: 0x555555d42280 <TraceThread<std::function<void ()> >(char const*, std::function<void ()>)>, a=...) at /usr/include/boost/bind/bind.hpp:319
#47 0x0000555555da0c09 in boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > >::operator()() (this=0x6150000edf38) at /usr/include/boost/bind/bind.hpp:1294
#48 0x0000555555da0625 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > > >::run() (this=0x6150000edd80) at /usr/include/boost/thread/detail/thread.hpp:116
#49 0x00007ffff77a6116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
#50 0x00007ffff5c02494 in start_thread (arg=0x7fffd9052700) at pthread_create.c:333
#51 0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 9 (Thread 0x7fffd98d6700 (LWP 28599)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000555555d4a1cd in boost::condition_variable::wait (this=0x5555588fa468 <scriptcheckqueue+40>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
#2  0x00005555564ee2e2 in CCheckQueue<CScriptCheck>::Loop (this=0x5555588fa440 <scriptcheckqueue>, fMaster=false) at ./checkqueue.h:99
#3  0x0000555556487720 in CCheckQueue<CScriptCheck>::Thread (this=0x5555588fa440 <scriptcheckqueue>) at ./checkqueue.h:136
#4  0x000055555642eb7d in ThreadScriptCheck () at validation.cpp:1680
#5  0x0000555555d994ad in boost::detail::thread_data<void (*)()>::run (this=0x61400005fa40) at /usr/include/boost/thread/detail/thread.hpp:116
#6  0x00007ffff77a6116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
#7  0x00007ffff5c02494 in start_thread (arg=0x7fffd98d6700) at pthread_create.c:333
#8  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 8 (Thread 0x7fffda14a700 (LWP 28598)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000555555d4a1cd in boost::condition_variable::wait (this=0x5555588fa468 <scriptcheckqueue+40>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
#2  0x00005555564ee2e2 in CCheckQueue<CScriptCheck>::Loop (this=0x5555588fa440 <scriptcheckqueue>, fMaster=false) at ./checkqueue.h:99
#3  0x0000555556487720 in CCheckQueue<CScriptCheck>::Thread (this=0x5555588fa440 <scriptcheckqueue>) at ./checkqueue.h:136
#4  0x000055555642eb7d in ThreadScriptCheck () at validation.cpp:1680
#5  0x0000555555d994ad in boost::detail::thread_data<void (*)()>::run (this=0x61400005fc40) at /usr/include/boost/thread/detail/thread.hpp:116
#6  0x00007ffff77a6116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
#7  0x00007ffff5c02494 in start_thread (arg=0x7fffda14a700) at pthread_create.c:333
#8  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 7 (Thread 0x7fffdaa39700 (LWP 28597)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000555555d4a1cd in boost::condition_variable::wait (this=0x5555588fa468 <scriptcheckqueue+40>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
#2  0x00005555564ee2e2 in CCheckQueue<CScriptCheck>::Loop (this=0x5555588fa440 <scriptcheckqueue>, fMaster=false) at ./checkqueue.h:99
#3  0x0000555556487720 in CCheckQueue<CScriptCheck>::Thread (this=0x5555588fa440 <scriptcheckqueue>) at ./checkqueue.h:136
#4  0x000055555642eb7d in ThreadScriptCheck () at validation.cpp:1680
#5  0x0000555555d994ad in boost::detail::thread_data<void (*)()>::run (this=0x61400005fe40) at /usr/include/boost/thread/detail/thread.hpp:116
#6  0x00007ffff77a6116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
#7  0x00007ffff5c02494 in start_thread (arg=0x7fffdaa39700) at pthread_create.c:333
#8  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 6 (Thread 0x7fffddb11700 (LWP 28596)):
#0  0x00007ffff3d9567d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00005555557c1c54 in poll () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#2  0x00007ffff21db9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007ffff67f504f in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ffff679e9ca in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ffff65cc0f3 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007ffff65d0da8 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffddb11700) at pthread_create.c:333
#9  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 5 (Thread 0x7fffdea98700 (LWP 28595)):
#0  0x00007ffff3d9567d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00005555557c1c54 in poll () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#2  0x00007ffff21db9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007ffff67f504f in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ffff679e9ca in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ffff65cc0f3 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007ffff64ab6d5 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5DBus.so.5
#8  0x00007ffff65d0da8 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#9  0x00007ffff5c02494 in start_thread (arg=0x7fffdea98700) at pthread_create.c:333
#10 0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 4 (Thread 0x7fffe1254700 (LWP 28593)):
#0  0x00007ffff3d9567d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00005555557c1c54 in poll () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#2  0x00007ffff21db9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff21dbd82 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007fffe3991656 in ?? () from /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
#5  0x00007ffff22033d5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x00007ffff5c02494 in start_thread (arg=0x7fffe1254700) at pthread_create.c:333
#7  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 3 (Thread 0x7fffe1aa5700 (LWP 28592)):
#0  0x00007ffff3d9567d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00005555557c1c54 in poll () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#2  0x00007ffff21db9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007ffff21dbb51 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007ffff22033d5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x00007ffff5c02494 in start_thread (arg=0x7fffe1aa5700) at pthread_create.c:333
#7  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 2 (Thread 0x7fffe763c700 (LWP 28591)):
#0  0x00007ffff3d9567d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00005555557c1c54 in poll () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#2  0x00007fffefe58150 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#3  0x00007fffefe59ee9 in xcb_wait_for_event () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#4  0x00007fffe9d9db69 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5XcbQpa.so.5
#5  0x00007ffff65d0da8 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ffff5c02494 in start_thread (arg=0x7fffe763c700) at pthread_create.c:333
#7  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7ffff7e47e00 (LWP 28586)):
#0  0x00007ffff6765972 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#1  0x00007ffff676aca9 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#2  0x00007ffff7300737 in QTableView::setSortingEnabled(bool) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#3  0x0000555555c7810c in TransactionView::setModel (this=0x60f00003b230, _model=0x610000434d40) at qt/transactionview.cpp:232
#4  0x0000555555ad425e in WalletView::setWalletModel (this=0x60d0073c1910, _walletModel=0x610000434d40) at qt/walletview.cpp:128
#5  0x0000555555a9aded in WalletFrame::addWallet (this=0x60800005eb20, walletModel=0x610000434d40) at qt/walletframe.cpp:57
#6  0x00005555558dafcb in BitcoinGUI::addWallet (this=0x614000023440, walletModel=0x610000434d40) at qt/bitcoingui.cpp:531
#7  0x000055555588234a in BitcoinApplication::addWallet (this=0x7fffffffd430, walletModel=0x610000434d40) at qt/bitcoin.cpp:431
#8  0x00005555558814b5 in BitcoinApplication::initializeResult (this=0x7fffffffd430, success=true) at qt/bitcoin.cpp:485
#9  0x00005555558ae1fc in QtPrivate::FunctorCall<QtPrivate::IndexesList<0>, QtPrivate::List<bool>, void, void (BitcoinApplication::*)(bool)>::call (f=(void (BitcoinApplication::*)(BitcoinApplication * const, bool)) 0x5555558807a0 <BitcoinApplication::initializeResult(bool)>, o=0x7fffffffd430, arg=0x602000332110) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:141
#10 0x00005555558ade62 in QtPrivate::FunctionPointer<void (BitcoinApplication::*)(bool)>::call<QtPrivate::List<bool>, void> (f=(void (BitcoinApplication::*)(BitcoinApplication * const, bool)) 0x5555558807a0 <BitcoinApplication::initializeResult(bool)>, o=0x7fffffffd430, arg=0x602000332110) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:160
#11 0x00005555558ada37 in QtPrivate::QSlotObject<void (BitcoinApplication::*)(bool), QtPrivate::List<bool>, void>::impl (which=1, this_=0x6030001b2830, r=0x7fffffffd430, a=0x602000332110, ret=0x0) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobject_impl.h:120
#12 0x00007ffff67cd499 in QObject::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#13 0x00007ffff6ae4d54 in QGuiApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5
#14 0x00007ffff708992f in QApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#15 0x00007ffff7083b8c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#16 0x00007ffff708b341 in QApplication::notify(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#17 0x00007ffff67a09e0 in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#18 0x00007ffff67a316d in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#19 0x00007ffff67f4c43 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#20 0x00007ffff21db7f7 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#21 0x00007ffff21dba60 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#22 0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#23 0x00007ffff67f504f in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#24 0x00007ffff679e9ca in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#25 0x00007ffff67a713c in QCoreApplication::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#26 0x0000555555889ee6 in main (argc=2, argv=0x7fffffffe4c8) at qt/bitcoin.cpp:714
(gdb) 

The end of debug.log looks like this:

2018-12-11T22:52:11Z [wo] nFileVersion = 179900
2018-12-11T22:52:11Z [wo] Keys: 13 plaintext, 0 encrypted, 307 w/ metadata, 13 total. Unknown wallet records: 1
2018-12-11T22:52:11Z [wo] Wallet completed loading in             341ms
2018-12-11T22:52:11Z [wo] setKeyPool.size() = 10
2018-12-11T22:52:11Z [wo] mapWallet.size() = 389
2018-12-11T22:52:11Z [wo] mapAddressBook.size() = 296
2018-12-11T22:52:11Z init message: Loading wallet...
2018-12-11T22:52:11Z [lnd] nFileVersion = 179900
2018-12-11T22:52:11Z [lnd] Keys: 21 plaintext, 0 encrypted, 65 w/ metadata, 21 total. Unknown wallet records: 0
2018-12-11T22:52:11Z [lnd] Wallet completed loading in              28ms
2018-12-11T22:52:11Z [lnd] setKeyPool.size() = 20
2018-12-11T22:52:11Z [lnd] mapWallet.size() = 49
2018-12-11T22:52:11Z [lnd] mapAddressBook.size() = 44
2018-12-11T22:52:11Z Unsetting NODE_NETWORK on prune mode
2018-12-11T22:52:11Z init message: Pruning blockstore...
2018-12-11T22:52:11Z mapBlockIndex.size() = 553468
2018-12-11T22:52:11Z nBestHeight = 553452
2018-12-11T22:52:11Z init message: Loading P2P addresses...
2018-12-11T22:52:11Z Leaving InitialBlockDownload (latching to false)
2018-12-11T22:52:12Z Loaded 69696 addresses from peers.dat  1320ms
2018-12-11T22:52:12Z init message: Loading banlist...
2018-12-11T22:52:12Z init message: Starting network threads...
2018-12-11T22:52:12Z DNS seeding disabled
2018-12-11T22:52:12Z net thread start
2018-12-11T22:52:12Z addcon thread start
2018-12-11T22:52:12Z opencon thread start
2018-12-11T22:52:12Z msghand thread start
2018-12-11T22:52:12Z init message: Done loading
2018-12-11T22:52:13Z GUI: Platform customization: "other"
2018-12-11T22:52:13Z New outbound peer connected: version: 70015, blocks=553452, peer=0
2018-12-11T22:52:13Z GUI: PaymentServer::LoadRootCAs: Loaded  151  root certificates

Is there anything further I can do to help fix the problem? I've been seeing this crash on and off for a couple of years now on different computers. I'm using an SSD now instead of a spinning disk. It's quick a lot faster, but I still see the crashes pretty regularly.

@MarcoFalke MarcoFalke added the Wallet label Dec 12, 2018

@MarcoFalke MarcoFalke changed the title random abort running master/HEAD gui, wallet: random abort (segmentation fault) running master/HEAD Dec 12, 2018

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Dec 12, 2018

I think it is caused by some of the transactions in your wallet. Further debugging could inspect which transactions are responsible (I have a testnet wallet that I use to reproduce this, in case anyone is interested in debuging)

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Dec 12, 2018

Here's another, after rebuilding with a slightly different ./configure line:

./configure CXX=clang++ CXXFLAGS="-fno-omit-frame-pointer" --enable-debug --with-incompatible-bdb --with-miniupnpc=no --with-sanitizers=address

I tried the --disable-hardening flag that was suggested above, but that causes it to fail to build with an error:

crypto/sha256_sse4.cpp:44:9: error: unknown token in expression
        "shl    $0x6,%2;"

It appears that it the combination of having -fsanitize=address and not -fstack-protector-all in the build command that triggers the error:

$ clang++ -std=c++11 -I. -I../src/config                                          -c crypto/sha256_sse4.cpp
$ clang++ -std=c++11 -I. -I../src/config -fstack-protector-all -fsanitize=address -c crypto/sha256_sse4.cpp
$ clang++ -std=c++11 -I. -I../src/config                       -fsanitize=address -c crypto/sha256_sse4.cpp
crypto/sha256_sse4.cpp:44:9: error: unknown token in expression
        "shl    $0x6,%2;"
        ^

Here's the new gdb output:

Thread 1 "bitcoin-qt" received signal SIGSEGV, Segmentation fault.
0x00007ffff6765972 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
A debugging session is active.

	Inferior 1 [process 10434] will be killed.

Quit anyway? (y or n) n
Not confirmed.
(gdb) set pagination off
(gdb) where
#0  0x00007ffff6765972 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#1  0x00007ffff676aca9 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#2  0x00007ffff7300737 in QTableView::setSortingEnabled(bool) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#3  0x0000555555c75c9c in TransactionView::setModel (this=0x60f00006dd20, _model=0x610000472c40) at qt/transactionview.cpp:232
#4  0x0000555555ad1dee in WalletView::setWalletModel (this=0x60d007434ca0, _walletModel=0x610000472c40) at qt/walletview.cpp:128
#5  0x0000555555a9897d in WalletFrame::addWallet (this=0x60800005eb20, walletModel=0x610000472c40) at qt/walletframe.cpp:57
#6  0x00005555558d8b5b in BitcoinGUI::addWallet (this=0x614000023440, walletModel=0x610000472c40) at qt/bitcoingui.cpp:531
#7  0x000055555587feda in BitcoinApplication::addWallet (this=0x7fffffffd430, walletModel=0x610000472c40) at qt/bitcoin.cpp:431
#8  0x000055555587f045 in BitcoinApplication::initializeResult (this=0x7fffffffd430, success=true) at qt/bitcoin.cpp:485
#9  0x00005555558abd8c in QtPrivate::FunctorCall<QtPrivate::IndexesList<0>, QtPrivate::List<bool>, void, void (BitcoinApplication::*)(bool)>::call (f=(void (BitcoinApplication::*)(BitcoinApplication * const, bool)) 0x55555587e330 <BitcoinApplication::initializeResult(bool)>, o=0x7fffffffd430, arg=0x60200038e710) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:141
#10 0x00005555558ab9f2 in QtPrivate::FunctionPointer<void (BitcoinApplication::*)(bool)>::call<QtPrivate::List<bool>, void> (f=(void (BitcoinApplication::*)(BitcoinApplication * const, bool)) 0x55555587e330 <BitcoinApplication::initializeResult(bool)>, o=0x7fffffffd430, arg=0x60200038e710) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:160
#11 0x00005555558ab5c7 in QtPrivate::QSlotObject<void (BitcoinApplication::*)(bool), QtPrivate::List<bool>, void>::impl (which=1, this_=0x6030001b36d0, r=0x7fffffffd430, a=0x60200038e710, ret=0x0) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobject_impl.h:120
#12 0x00007ffff67cd499 in QObject::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#13 0x00007ffff6ae4d54 in QGuiApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5
#14 0x00007ffff708992f in QApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#15 0x00007ffff7083b8c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#16 0x00007ffff708b341 in QApplication::notify(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#17 0x00007ffff67a09e0 in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#18 0x00007ffff67a316d in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#19 0x00007ffff67f4c43 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#20 0x00007ffff21db7f7 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#21 0x00007ffff21dba60 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#22 0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#23 0x00007ffff67f504f in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#24 0x00007ffff679e9ca in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#25 0x00007ffff67a713c in QCoreApplication::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#26 0x0000555555887a76 in main (argc=2, argv=0x7fffffffe4c8) at qt/bitcoin.cpp:714
(gdb) thread apply all where

Thread 22 (Thread 0x7fff6009b700 (LWP 10470)):
#0  0x00007ffff3d9567d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00005555557bf7e4 in poll () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#2  0x00007ffff21db9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007ffff67f504f in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ffff679e9ca in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ffff65cc0f3 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007ffff65d0da8 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#8  0x00007ffff5c02494 in start_thread (arg=0x7fff6009b700) at pthread_create.c:333
#9  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 21 (Thread 0x7fff64b4a700 (LWP 10469)):
#0  0x0000555555863007 in __sanitizer::StackDepotPut(__sanitizer::StackTrace) () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#1  0x00005555557afa57 in __asan::asan_memalign(unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType) () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#2  0x0000555555878b29 in operator new(unsigned long) () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#3  0x0000555555928e02 in __gnu_cxx::new_allocator<unsigned char>::allocate (this=0x612000651cd0, __n=33) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/ext/new_allocator.h:104
#4  0x0000555555928d8a in std::allocator_traits<std::allocator<unsigned char> >::allocate (__a=..., __n=33) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/alloc_traits.h:436
#5  0x0000555555928d20 in std::_Vector_base<unsigned char, std::allocator<unsigned char> >::_M_allocate (this=0x612000651cd0, __n=33) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/stl_vector.h:170
#6  0x0000555555928ba1 in std::_Vector_base<unsigned char, std::allocator<unsigned char> >::_M_create_storage (this=0x612000651cd0, __n=33) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/stl_vector.h:185
#7  0x0000555555928101 in std::_Vector_base<unsigned char, std::allocator<unsigned char> >::_Vector_base (this=0x612000651cd0, __n=33, __a=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/stl_vector.h:136
#8  0x0000555555927519 in std::vector<unsigned char, std::allocator<unsigned char> >::vector (this=0x612000651cd0, __x=std::vector of length 33, capacity 33 = {...}) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/stl_vector.h:321
#9  0x0000555556c2166d in __gnu_cxx::new_allocator<std::vector<unsigned char, std::allocator<unsigned char> > >::construct<std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned char, std::allocator<unsigned char> > const&> (this=0x7fff64b3bde0, __p=0x612000651cd0, __args=std::vector of length 33, capacity 33 = {...}) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/ext/new_allocator.h:120
#10 0x0000555556c21129 in std::allocator_traits<std::allocator<std::vector<unsigned char, std::allocator<unsigned char> > > >::construct<std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned char, std::allocator<unsigned char> > const&> (__a=..., __p=0x612000651cd0, __args=std::vector of length 33, capacity 33 = {...}) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/alloc_traits.h:475
#11 0x0000555556c211e0 in std::vector<std::vector<unsigned char, std::allocator<unsigned char> >, std::allocator<std::vector<unsigned char, std::allocator<unsigned char> > > >::_M_emplace_back_aux<std::vector<unsigned char, std::allocator<unsigned char> > const&> (this=0x7fff64b3bde0, __args=std::vector of length 33, capacity 33 = {...}) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/vector.tcc:416
#12 0x0000555556c1cff0 in std::vector<std::vector<unsigned char, std::allocator<unsigned char> >, std::allocator<std::vector<unsigned char, std::allocator<unsigned char> > > >::push_back (this=0x7fff64b3bde0, __x=std::vector of length 33, capacity 33 = {...}) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/stl_vector.h:924
#13 0x0000555556c670b8 in EvalScript (stack=std::vector of length 6, capacity 6 = {...}, script=..., flags=131039, checker=..., sigversion=SigVersion::WITNESS_V0, serror=0x7fff64b3cabc) at script/interpreter.cpp:347
#14 0x0000555556c71077 in VerifyWitnessProgram (witness=..., witversion=0, program=std::vector of length 32, capacity 32 = {...}, flags=131039, checker=..., serror=0x7fff64b3cabc) at script/interpreter.cpp:1460
#15 0x0000555556c70222 in VerifyScript (scriptSig=..., scriptPubKey=..., witness=0x6190020c77c8, flags=131039, checker=..., serror=0x7fff64b3cabc) at script/interpreter.cpp:1555
#16 0x00005555564296a4 in CScriptCheck::operator() (this=0x7fff64b3ca80) at validation.cpp:1334
#17 0x000055555642ac3f in CheckInputs (tx=..., state=..., inputs=..., fScriptChecks=true, flags=131039, cacheSigStore=true, cacheFullScriptStore=false, txdata=..., pvChecks=0x0) at validation.cpp:1419
#18 0x000055555646e1f6 in AcceptToMemoryPoolWorker (chainparams=..., pool=..., state=..., ptx=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d0083172a0, pfMissingInputs=0x7fff64b42f30, nAcceptTime=1544572247, plTxnReplaced=0x7fff64b42fb0, bypass_limits=false, nAbsurdFee=@0x7fff64b3ff80: 0, coins_to_uncache=std::vector of length 10, capacity 16 = {...}, test_accept=false) at validation.cpp:893
#19 0x0000555556425376 in AcceptToMemoryPoolWithTime (chainparams=..., pool=..., state=..., tx=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d0083172a0, pfMissingInputs=0x7fff64b42f30, nAcceptTime=1544572247, plTxnReplaced=0x7fff64b42fb0, bypass_limits=false, nAbsurdFee=0, test_accept=false) at validation.cpp:974
#20 0x000055555642507b in AcceptToMemoryPool (pool=..., state=..., tx=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d0083172a0, pfMissingInputs=0x7fff64b42f30, plTxnReplaced=0x7fff64b42fb0, bypass_limits=false, nAbsurdFee=0, test_accept=false) at validation.cpp:990
Python Exception <class 'gdb.error'> There is no member named _M_dataplus.: 
#21 0x0000555555ea74b2 in ProcessMessage (pfrom=0x61c00023f880, strCommand=, vRecv=..., nTimeReceived=1544572247512912, chainparams=..., connman=0x7fffd4ccb800, interruptMsgProc=..., enable_bip61=false) at net_processing.cpp:2219
#22 0x0000555555e97df7 in PeerLogicValidation::ProcessMessages (this=0x6040000cb6d0, pfrom=0x61c00023f880, interruptMsgProc=...) at net_processing.cpp:3052
#23 0x0000555555eb1b15 in non-virtual thunk to PeerLogicValidation::ProcessMessages(CNode*, std::atomic<bool>&) () at net_processing.cpp:2977
#24 0x0000555555ddf019 in CConnman::ThreadMessageHandler (this=0x7fffd4ccb800) at net.cpp:2055
#25 0x0000555555e6eb0e in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=@0x603000e88550: (void (CConnman::*)(CConnman * const)) 0x555555dde970 <CConnman::ThreadMessageHandler()>, __t=@0x603000e88560: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#26 0x0000555555e6e98f in std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=@0x603000e88550: (void (CConnman::*)(CConnman * const)) 0x555555dde970 <CConnman::ThreadMessageHandler()>, __args=@0x603000e88560: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#27 0x0000555555e6e889 in std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=0x603000e88550, __args=@0x603000e88560: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#28 0x0000555555e6e7a7 in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x603000e88550, __args=<unknown type in /home/me/Programs/bitcoin/src/qt/bitcoin-qt, CU 0xd1122e, DIE 0xd9b04a>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#29 0x0000555555e6e5cd in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=0x603000e88550) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#30 0x0000555555e6e006 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#31 0x0000555555a39c66 in std::function<void ()>::operator()() const (this=0x7fff64b49ba0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#32 0x0000555555d3ff6f in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555557351680 <.str.115> "msghand", func=...) at ./util/system.h:331
#33 0x0000555555e6fb0c in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=0x606003c7a608) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#34 0x0000555555e6f95e in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=0x606003c7a608) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#35 0x0000555555e6f1e2 in std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=0x606003c7a600) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#36 0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#37 0x00007ffff5c02494 in start_thread (arg=0x7fff64b4a700) at pthread_create.c:333
#38 0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 20 (Thread 0x7fff6539b700 (LWP 10468)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000555555d3795e in __gthread_cond_timedwait (__cond=0x7fffd4d1bca8, __mutex=0x7fffd4d1bcd8, __abs_timeout=0x7fff65399540) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:871
#2  0x0000555555da8eff in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fffd4d1bca8, __lock=..., __atime=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:166
#3  0x0000555555da893e in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fffd4d1bca8, __lock=..., __atime=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:106
#4  0x000055555674e5af in std::condition_variable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=0x7fffd4d1bca8, __lock=..., __atime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:129
#5  0x000055555674de2b in std::condition_variable::wait_for<long, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=0x7fffd4d1bca8, __lock=..., __rtime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:145
#6  0x000055555674db16 in CThreadInterrupt::sleep_for (this=0x7fffd4d1bca8, rel_time=...) at threadinterrupt.cpp:34
#7  0x0000555555ddb3ac in CConnman::ThreadOpenConnections (this=0x7fffd4ccb800, connect=std::vector of length 1, capacity 1 = {...}) at net.cpp:1770
#8  0x0000555555e7275b in std::__invoke_impl<void, void (CConnman::* const&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (__f=@0x6040019e5950: (void (CConnman::*)(CConnman * const, std::vector<std::__cxx11::basic_string<char>, std::allocator<std::__cxx11::basic_string<char> > >)) 0x555555ddaac0 <CConnman::ThreadOpenConnections(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)>, __t=@0x6040019e5978: 0x7fffd4ccb800, __args=std::vector of length 1, capacity 1 = {...}) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#9  0x0000555555e724a4 in std::__invoke<void (CConnman::* const&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (__fn=@0x6040019e5950: (void (CConnman::*)(CConnman * const, std::vector<std::__cxx11::basic_string<char>, std::allocator<std::__cxx11::basic_string<char> > >)) 0x555555ddaac0 <CConnman::ThreadOpenConnections(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)>, __args=std::vector of length 1, capacity 1 = {...}, __args=std::vector of length 1, capacity 1 = {...}) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#10 0x0000555555e72342 in std::_Mem_fn_base<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), true>::operator()<CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (this=0x6040019e5950, __args=std::vector of length 1, capacity 1 = {...}, __args=std::vector of length 1, capacity 1 = {...}) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#11 0x0000555555e7227f in std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=0x6040019e5950, __args=<unknown type in /home/me/Programs/bitcoin/src/qt/bitcoin-qt, CU 0xd1122e, DIE 0xd9c850>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#12 0x0000555555e7206d in std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)>::operator()<, void>() (this=0x6040019e5950) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#13 0x0000555555e71876 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#14 0x0000555555a39c66 in std::function<void ()>::operator()() const (this=0x7fff6539aba0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#15 0x0000555555d3ff6f in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555557351640 <.str.114> "opencon", func=...) at ./util/system.h:331
#16 0x0000555555e6fb0c in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=0x606003c7a668) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#17 0x0000555555e6f95e in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=0x606003c7a668) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#18 0x0000555555e6f1e2 in std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=0x606003c7a660) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#19 0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#20 0x00007ffff5c02494 in start_thread (arg=0x7fff6539b700) at pthread_create.c:333
#21 0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 19 (Thread 0x7fff65bec700 (LWP 10467)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000555555d3795e in __gthread_cond_timedwait (__cond=0x7fffd4d1bca8, __mutex=0x7fffd4d1bcd8, __abs_timeout=0x7fff65beaca0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:871
#2  0x0000555555da8eff in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fffd4d1bca8, __lock=..., __atime=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:166
#3  0x0000555555da893e in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fffd4d1bca8, __lock=..., __atime=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:106
#4  0x000055555674e5af in std::condition_variable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=0x7fffd4d1bca8, __lock=..., __atime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:129
#5  0x000055555674de2b in std::condition_variable::wait_for<long, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=0x7fffd4d1bca8, __lock=..., __rtime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:145
#6  0x000055555674db16 in CThreadInterrupt::sleep_for (this=0x7fffd4d1bca8, rel_time=...) at threadinterrupt.cpp:34
#7  0x000055555674e040 in CThreadInterrupt::sleep_for (this=0x7fffd4d1bca8, rel_time=...) at threadinterrupt.cpp:39
#8  0x0000555555dde81c in CConnman::ThreadOpenAddedConnections (this=0x7fffd4ccb800) at net.cpp:1989
#9  0x0000555555e6eb0e in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=@0x603000e885b0: (void (CConnman::*)(CConnman * const)) 0x555555dde220 <CConnman::ThreadOpenAddedConnections()>, __t=@0x603000e885c0: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#10 0x0000555555e6e98f in std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=@0x603000e885b0: (void (CConnman::*)(CConnman * const)) 0x555555dde220 <CConnman::ThreadOpenAddedConnections()>, __args=@0x603000e885c0: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#11 0x0000555555e6e889 in std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=0x603000e885b0, __args=@0x603000e885c0: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#12 0x0000555555e6e7a7 in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x603000e885b0, __args=<unknown type in /home/me/Programs/bitcoin/src/qt/bitcoin-qt, CU 0xd1122e, DIE 0xd9b04a>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#13 0x0000555555e6e5cd in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=0x603000e885b0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#14 0x0000555555e6e006 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#15 0x0000555555a39c66 in std::function<void ()>::operator()() const (this=0x7fff65bebba0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#16 0x0000555555d3ff6f in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555557351580 <.str.112> "addcon", func=...) at ./util/system.h:331
#17 0x0000555555e6fb0c in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=0x606003c7a6c8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#18 0x0000555555e6f95e in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=0x606003c7a6c8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#19 0x0000555555e6f1e2 in std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=0x606003c7a6c0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#20 0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#21 0x00007ffff5c02494 in start_thread (arg=0x7fff65bec700) at pthread_create.c:333
#22 0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 18 (Thread 0x7fff6643d700 (LWP 10466)):
#0  0x00007ffff3d973c3 in select () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000555555dd4d9e in CConnman::SocketHandler (this=0x7fffd4ccb800) at net.cpp:1324
#2  0x0000555555dd75fd in CConnman::ThreadSocketHandler (this=0x7fffd4ccb800) at net.cpp:1466
#3  0x0000555555e6eb0e in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=@0x603000e88670: (void (CConnman::*)(CConnman * const)) 0x555555dd75a0 <CConnman::ThreadSocketHandler()>, __t=@0x603000e88680: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#4  0x0000555555e6e98f in std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=@0x603000e88670: (void (CConnman::*)(CConnman * const)) 0x555555dd75a0 <CConnman::ThreadSocketHandler()>, __args=@0x603000e88680: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#5  0x0000555555e6e889 in std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=0x603000e88670, __args=@0x603000e88680: 0x7fffd4ccb800) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#6  0x0000555555e6e7a7 in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x603000e88670, __args=<unknown type in /home/me/Programs/bitcoin/src/qt/bitcoin-qt, CU 0xd1122e, DIE 0xd9b04a>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#7  0x0000555555e6e5cd in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=0x603000e88670) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#8  0x0000555555e6e006 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#9  0x0000555555a39c66 in std::function<void ()>::operator()() const (this=0x7fff6643cba0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#10 0x0000555555d3ff6f in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x555557351480 <.str.108> "net", func=...) at ./util/system.h:331
#11 0x0000555555e6fb0c in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=0x606003c7a728) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#12 0x0000555555e6f95e in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=0x606003c7a728) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#13 0x0000555555e6f1e2 in std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=0x606003c7a720) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#14 0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007ffff5c02494 in start_thread (arg=0x7fff6643d700) at pthread_create.c:333
#16 0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 16 (Thread 0x7fffcf952700 (LWP 10459)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000555556e902ce in leveldb::(anonymous namespace)::PosixEnv::BGThread (this=0x61300005f100) at leveldb/util/env_posix.cc:649
#2  0x0000555556e8fffe in leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x61300005f100) at leveldb/util/env_posix.cc:564
#3  0x00007ffff5c02494 in start_thread (arg=0x7fffcf952700) at pthread_create.c:333
#4  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 15 (Thread 0x7fffd660e700 (LWP 10452)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff4a3050c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005555566a6760 in WorkQueue<HTTPClosure>::Run (this=0x61000003fe40) at httpserver.cpp:107
#3  0x000055555669093e in HTTPWorkQueueRun (queue=0x61000003fe40) at httpserver.cpp:342
#4  0x00005555566ac11c in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x6030001c5978) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x00005555566ac05e in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=0x6030001c5978) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x00005555566abb22 in std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0x6030001c5970) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffd660e700) at pthread_create.c:333
#9  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 14 (Thread 0x7fffd6e82700 (LWP 10451)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff4a3050c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005555566a6760 in WorkQueue<HTTPClosure>::Run (this=0x61000003fe40) at httpserver.cpp:107
#3  0x000055555669093e in HTTPWorkQueueRun (queue=0x61000003fe40) at httpserver.cpp:342
#4  0x00005555566ac11c in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x6030001c59a8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x00005555566ac05e in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=0x6030001c59a8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x00005555566abb22 in std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0x6030001c59a0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffd6e82700) at pthread_create.c:333
#9  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 13 (Thread 0x7fffd76f6700 (LWP 10450)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff4a3050c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005555566a6760 in WorkQueue<HTTPClosure>::Run (this=0x61000003fe40) at httpserver.cpp:107
#3  0x000055555669093e in HTTPWorkQueueRun (queue=0x61000003fe40) at httpserver.cpp:342
#4  0x00005555566ac11c in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x6030001c5a08) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x00005555566ac05e in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=0x6030001c5a08) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x00005555566abb22 in std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0x6030001c5a00) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffd76f6700) at pthread_create.c:333
#9  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 12 (Thread 0x7fffd7f6a700 (LWP 10449)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff4a3050c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005555566a6760 in WorkQueue<HTTPClosure>::Run (this=0x61000003fe40) at httpserver.cpp:107
#3  0x000055555669093e in HTTPWorkQueueRun (queue=0x61000003fe40) at httpserver.cpp:342
#4  0x00005555566ac11c in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x6030001c5a38) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x00005555566ac05e in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=0x6030001c5a38) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x00005555566abb22 in std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0x6030001c5a30) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffd7f6a700) at pthread_create.c:333
#9  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 11 (Thread 0x7fffd87de700 (LWP 10448)):
#0  0x00007ffff3d9f0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007ffff53efd98 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#2  0x00007ffff53da07a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#3  0x0000555556690830 in ThreadHTTP (base=0x6160000ab980) at httpserver.cpp:290
#4  0x00005555566aa93c in std::_Bind_simple<bool (*(event_base*))(event_base*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x6030001c5a68) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x00005555566aa86e in std::_Bind_simple<bool (*(event_base*))(event_base*)>::operator()() (this=0x6030001c5a68) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x00005555566aa322 in std::thread::_State_impl<std::_Bind_simple<bool (*(event_base*))(event_base*)> >::_M_run() (this=0x6030001c5a60) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffd87de700) at pthread_create.c:333
#9  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 10 (Thread 0x7fffd9052700 (LWP 10447)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ffff5c04c06 in __GI___pthread_mutex_lock (mutex=0x555557b8aae0 <cs_main>) at ../nptl/pthread_mutex_lock.c:115
#2  0x0000555555cf5e20 in __gthread_mutex_lock (__mutex=0x555557b8aae0 <cs_main>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:748
#3  0x0000555555cf5dbe in __gthread_recursive_mutex_lock (__mutex=0x555557b8aae0 <cs_main>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:810
#4  0x0000555555cff165 in std::recursive_mutex::lock (this=0x555557b8aae0 <cs_main>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/mutex:105
#5  0x0000555555cff0c4 in std::unique_lock<std::recursive_mutex>::lock (this=0x6030087ea918) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/std_mutex.h:267
#6  0x0000555555cfeb96 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::Enter (this=0x6030087ea918, pszName=0x5555574321e0 <.str.4> "cs_main", pszFile=0x555557432220 <.str.5> "interfaces/chain.cpp", nLine=31) at ./sync.h:128
#7  0x0000555555cfb801 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::UniqueLock (this=0x6030087ea918, mutexIn=..., pszName=0x5555574321e0 <.str.4> "cs_main", pszFile=0x555557432220 <.str.5> "interfaces/chain.cpp", nLine=31, fTry=false) at ./sync.h:149
#8  0x00005555566ccd52 in interfaces::(anonymous namespace)::LockingStateImpl::LockingStateImpl (this=0x6030087ea910) at interfaces/chain.cpp:23
#9  0x00005555566cc8ba in MakeUnique<interfaces::(anonymous namespace)::LockingStateImpl, AnnotatedMixin<std::recursive_mutex>&, char const (&) [8], char const (&) [21], int, bool&>(AnnotatedMixin<std::recursive_mutex>&, char const (&) [8], char const (&) [21], int&&, bool&) (args=@0x7fffd904fc80: false, args=@0x7fffd904fc80: false, args=@0x7fffd904fc80: false, args=@0x7fffd904fc80: false, args=@0x7fffd904fc80: false) at ./util/memory.h:16
#10 0x00005555566cc44a in interfaces::(anonymous namespace)::ChainImpl::lock (this=0x60200000de10, try_lock=false) at interfaces/chain.cpp:31
#11 0x000055555692a462 in CWallet::TransactionAddedToMempool (this=0x61a0002dc080, ptx=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d008317440) at wallet/wallet.cpp:1147
#12 0x000055555692a6ff in non-virtual thunk to CWallet::TransactionAddedToMempool(std::shared_ptr<CTransaction const> const&) () at wallet/wallet.cpp:1146
#13 0x000055555659d280 in boost::_mfi::mf1<void, CValidationInterface, std::shared_ptr<CTransaction const> const&>::operator() (this=0x6060035e7fe0, p=0x61a0002dc1d8, a1=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d008317440) at /usr/include/boost/bind/mem_fn_template.hpp:165
#14 0x000055555659d120 in boost::_bi::list2<boost::_bi::value<CValidationInterface*>, boost::arg<1> >::operator()<boost::_mfi::mf1<void, CValidationInterface, std::shared_ptr<CTransaction const> const&>, boost::_bi::rrlist1<std::shared_ptr<CTransaction const> const&> > (this=0x6060035e7ff0, f=..., a=...) at /usr/include/boost/bind/bind.hpp:319
#15 0x000055555659cf52 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, CValidationInterface, std::shared_ptr<CTransaction const> const&>, boost::_bi::list2<boost::_bi::value<CValidationInterface*>, boost::arg<1> > >::operator()<std::shared_ptr<CTransaction const> const&> (this=0x6060035e7fe0, a1=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d008317440) at /usr/include/boost/bind/bind.hpp:1306
#16 0x000055555659c43e in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, CValidationInterface, std::shared_ptr<CTransaction const> const&>, boost::_bi::list2<boost::_bi::value<CValidationInterface*>, boost::arg<1> > >, void, std::shared_ptr<CTransaction const> const&>::invoke (function_obj_ptr=..., a0=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d008317440) at /usr/include/boost/function/function_template.hpp:159
#17 0x0000555556609360 in boost::function1<void, std::shared_ptr<CTransaction const> const&>::operator() (this=0x6060035e7fd8, a0=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d008317440) at /usr/include/boost/function/function_template.hpp:770
#18 0x00005555566091a5 in boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::m_invoke<boost::function<void (std::shared_ptr<CTransaction const> const&)>, 0u, std::shared_ptr<CTransaction const> const&>(boost::function<void (std::shared_ptr<CTransaction const> const&)>&, boost::signals2::detail::unsigned_meta_array<0u>, std::tuple<std::shared_ptr<CTransaction const> const&> const&, boost::enable_if<boost::is_void<boost::function<void (std::shared_ptr<CTransaction const> const&)>::result_type>, void>::type*) const (this=0x7fffd9050360, func=..., args=std::tuple containing = {...}) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:105
#19 0x0000555556609140 in boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::operator()<boost::function<void (std::shared_ptr<CTransaction const> const&)>, std::shared_ptr<CTransaction const> const&, 1ul>(boost::function<void (std::shared_ptr<CTransaction const> const&)>&, std::tuple<std::shared_ptr<CTransaction const> const&> const&, mpl_::size_t<1ul>) const (this=0x7fffd9050360, func=..., args=std::tuple containing = {...}) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
#20 0x00005555566090a4 in boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>::operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >(boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > const&) const (this=0x7fffd9050ad0, connectionBody=...) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:133
#21 0x0000555556608e07 in boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> >::dereference() const (this=0x7fffd9050660) at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:110
#22 0x0000555556608bce in boost::iterators::iterator_core_access::dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > const&) (f=...) at /usr/include/boost/iterator/iterator_facade.hpp:549
#23 0x0000555556608a86 in boost::iterators::detail::iterator_facade_base<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> >, boost::signals2::detail::void_type, boost::iterators::single_pass_traversal_tag, boost::signals2::detail::void_type&, long, false, false>::operator*() const (this=0x7fffd9050660) at /usr/include/boost/iterator/iterator_facade.hpp:655
#24 0x0000555556608872 in boost::signals2::optional_last_value<void>::operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> >) const (this=0x602000149f10, first=..., last=...) at /usr/include/boost/signals2/optional_last_value.hpp:57
#25 0x0000555556607ee6 in boost::signals2::detail::combiner_invoker<void>::operator()<boost::signals2::optional_last_value<void>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >(boost::signals2::optional_last_value<void>&, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, std::shared_ptr<CTransaction const> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (std::shared_ptr<CTransaction const> const&), boost::function<void (std::shared_ptr<CTransaction const> const&)> >, boost::signals2::mutex> >) const (this=0x7fffd9050b70, combiner=..., first=..., last=...) at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:64
#26 0x00005555566076fe in boost::signals2::detail::signal_impl<void (std::shared_ptr<CTransaction const> const&), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (std::shared_ptr<CTransaction const> const&)>, boost::function<void (boost::signals2::connection const&, std::shared_ptr<CTransaction const> const&)>, boost::signals2::mutex>::operator()(std::shared_ptr<CTransaction const> const&) (this=0x6040000bdc50, args=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d008317440) at /usr/include/boost/signals2/detail/signal_template.hpp:242
#27 0x0000555556606f59 in boost::signals2::signal<void (std::shared_ptr<CTransaction const> const&), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (std::shared_ptr<CTransaction const> const&)>, boost::function<void (boost::signals2::connection const&, std::shared_ptr<CTransaction const> const&)>, boost::signals2::mutex>::operator()(std::shared_ptr<CTransaction const> const&) (this=0x6120001d3ed8, args=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 2, weak 0) 0x60d008317440) at /usr/include/boost/signals2/detail/signal_template.hpp:723
#28 0x000055555652ae35 in CMainSignals::TransactionAddedToMempool(std::shared_ptr<CTransaction const> const&)::$_3::operator()() const (this=0x6030087dda40) at validationinterface.cpp:147
#29 0x000055555652aa56 in std::_Function_handler<void (), CMainSignals::TransactionAddedToMempool(std::shared_ptr<CTransaction const> const&)::$_3>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#30 0x0000555555a39c66 in std::function<void ()>::operator()() const (this=0x7fffd9050f60) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#31 0x0000555556bc05e0 in SingleThreadedSchedulerClient::ProcessQueue (this=0x6120001d3f98) at scheduler.cpp:187
#32 0x0000555556bcb5fe in std::__invoke_impl<void, void (SingleThreadedSchedulerClient::* const&)(), SingleThreadedSchedulerClient*&> (__f=@0x6030087ea4c0: (void (SingleThreadedSchedulerClient::*)(SingleThreadedSchedulerClient * const)) 0x555556bc0330 <SingleThreadedSchedulerClient::ProcessQueue()>, __t=@0x6030087ea4d0: 0x6120001d3f98) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#33 0x0000555556bcb47f in std::__invoke<void (SingleThreadedSchedulerClient::* const&)(), SingleThreadedSchedulerClient*&> (__fn=@0x6030087ea4c0: (void (SingleThreadedSchedulerClient::*)(SingleThreadedSchedulerClient * const)) 0x555556bc0330 <SingleThreadedSchedulerClient::ProcessQueue()>, __args=@0x6030087ea4d0: 0x6120001d3f98) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#34 0x0000555556bcb379 in std::_Mem_fn_base<void (SingleThreadedSchedulerClient::*)(), true>::operator()<SingleThreadedSchedulerClient*&> (this=0x6030087ea4c0, __args=@0x6030087ea4d0: 0x6120001d3f98) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#35 0x0000555556bcb2d7 in std::_Bind<std::_Mem_fn<void (SingleThreadedSchedulerClient::*)()> (SingleThreadedSchedulerClient*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x6030087ea4c0, __args=<unknown type in /home/me/Programs/bitcoin/src/qt/bitcoin-qt, CU 0x1ddf3c9, DIE 0x1df6c36>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#36 0x0000555556bcb0fd in std::_Bind<std::_Mem_fn<void (SingleThreadedSchedulerClient::*)()> (SingleThreadedSchedulerClient*)>::operator()<, void>() (this=0x6030087ea4c0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#37 0x0000555556bcaaf6 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (SingleThreadedSchedulerClient::*)()> (SingleThreadedSchedulerClient*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#38 0x0000555555a39c66 in std::function<void ()>::operator()() const (this=0x7fffd90514a0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#39 0x0000555556bbe8a8 in CScheduler::serviceQueue (this=0x5555588e07c0 <scheduler>) at scheduler.cpp:83
#40 0x0000555555d9cfc8 in boost::_mfi::mf0<void, CScheduler>::operator() (this=0x6030001c7fb0, p=0x5555588e07c0 <scheduler>) at /usr/include/boost/bind/mem_fn_template.hpp:49
#41 0x0000555555d9ce72 in boost::_bi::list1<boost::_bi::value<CScheduler*> >::operator()<boost::_mfi::mf0<void, CScheduler>, boost::_bi::list0> (this=0x6030001c7fc0, f=..., a=...) at /usr/include/boost/bind/bind.hpp:259
#42 0x0000555555d9cd39 in boost::_bi::bind_t<void, boost::_mfi::mf0<void, CScheduler>, boost::_bi::list1<boost::_bi::value<CScheduler*> > >::operator() (this=0x6030001c7fb0) at /usr/include/boost/bind/bind.hpp:1294
#43 0x0000555555d9c976 in std::_Function_handler<void (), boost::_bi::bind_t<void, boost::_mfi::mf0<void, CScheduler>, boost::_bi::list1<boost::_bi::value<CScheduler*> > > >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#44 0x0000555555a39c66 in std::function<void ()>::operator()() const (this=0x7fffd9051aa0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#45 0x0000555555d3ff6f in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x5555573446a0 <.str> "scheduler", func=...) at ./util/system.h:331
#46 0x0000555555d9e9c8 in boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > >::operator()<void (*)(char const*, std::function<void ()>), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(char const*, std::function<void ()>), boost::_bi::list0&, int) (this=0x6150000edf40, f=@0x6150000edf38: 0x555555d3fe10 <TraceThread<std::function<void ()> >(char const*, std::function<void ()>)>, a=...) at /usr/include/boost/bind/bind.hpp:319
#47 0x0000555555d9e799 in boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > >::operator()() (this=0x6150000edf38) at /usr/include/boost/bind/bind.hpp:1294
#48 0x0000555555d9e1b5 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > > >::run() (this=0x6150000edd80) at /usr/include/boost/thread/detail/thread.hpp:116
#49 0x00007ffff77a6116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
#50 0x00007ffff5c02494 in start_thread (arg=0x7fffd9052700) at pthread_create.c:333
#51 0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 9 (Thread 0x7fffd98d6700 (LWP 10446)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000555555d47d5d in boost::condition_variable::wait (this=0x5555588e71e8 <scriptcheckqueue+40>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
#2  0x00005555564ebe72 in CCheckQueue<CScriptCheck>::Loop (this=0x5555588e71c0 <scriptcheckqueue>, fMaster=false) at ./checkqueue.h:99
#3  0x00005555564852b0 in CCheckQueue<CScriptCheck>::Thread (this=0x5555588e71c0 <scriptcheckqueue>) at ./checkqueue.h:136
#4  0x000055555642c70d in ThreadScriptCheck () at validation.cpp:1680
#5  0x0000555555d9703d in boost::detail::thread_data<void (*)()>::run (this=0x61400005fa40) at /usr/include/boost/thread/detail/thread.hpp:116
#6  0x00007ffff77a6116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
#7  0x00007ffff5c02494 in start_thread (arg=0x7fffd98d6700) at pthread_create.c:333
#8  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 8 (Thread 0x7fffda1c5700 (LWP 10445)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000555555d47d5d in boost::condition_variable::wait (this=0x5555588e71e8 <scriptcheckqueue+40>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
#2  0x00005555564ebe72 in CCheckQueue<CScriptCheck>::Loop (this=0x5555588e71c0 <scriptcheckqueue>, fMaster=false) at ./checkqueue.h:99
#3  0x00005555564852b0 in CCheckQueue<CScriptCheck>::Thread (this=0x5555588e71c0 <scriptcheckqueue>) at ./checkqueue.h:136
#4  0x000055555642c70d in ThreadScriptCheck () at validation.cpp:1680
#5  0x0000555555d9703d in boost::detail::thread_data<void (*)()>::run (this=0x61400005fc40) at /usr/include/boost/thread/detail/thread.hpp:116
#6  0x00007ffff77a6116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
#7  0x00007ffff5c02494 in start_thread (arg=0x7fffda1c5700) at pthread_create.c:333
#8  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 7 (Thread 0x7fffdaa39700 (LWP 10444)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000555555d47d5d in boost::condition_variable::wait (this=0x5555588e71e8 <scriptcheckqueue+40>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
#2  0x00005555564ebe72 in CCheckQueue<CScriptCheck>::Loop (this=0x5555588e71c0 <scriptcheckqueue>, fMaster=false) at ./checkqueue.h:99
#3  0x00005555564852b0 in CCheckQueue<CScriptCheck>::Thread (this=0x5555588e71c0 <scriptcheckqueue>) at ./checkqueue.h:136
#4  0x000055555642c70d in ThreadScriptCheck () at validation.cpp:1680
#5  0x0000555555d9703d in boost::detail::thread_data<void (*)()>::run (this=0x61400005fe40) at /usr/include/boost/thread/detail/thread.hpp:116
#6  0x00007ffff77a6116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
#7  0x00007ffff5c02494 in start_thread (arg=0x7fffdaa39700) at pthread_create.c:333
#8  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 6 (Thread 0x7fffddb11700 (LWP 10443)):
#0  0x00007ffff3d9567d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00005555557bf7e4 in poll () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#2  0x00007ffff21db9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007ffff67f504f in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ffff679e9ca in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ffff65cc0f3 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007ffff65d0da8 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffddb11700) at pthread_create.c:333
#9  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 5 (Thread 0x7fffdea98700 (LWP 10442)):
#0  0x00007ffff3d9567d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00005555557bf7e4 in poll () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#2  0x00007ffff21db9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007ffff67f504f in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ffff679e9ca in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ffff65cc0f3 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007ffff64ab6d5 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5DBus.so.5
#8  0x00007ffff65d0da8 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#9  0x00007ffff5c02494 in start_thread (arg=0x7fffdea98700) at pthread_create.c:333
#10 0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 4 (Thread 0x7fffe1254700 (LWP 10440)):
#0  0x00007ffff3d9567d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00005555557bf7e4 in poll () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#2  0x00007ffff21db9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff21dbd82 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007fffe3991656 in ?? () from /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
#5  0x00007ffff22033d5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x00007ffff5c02494 in start_thread (arg=0x7fffe1254700) at pthread_create.c:333
#7  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 3 (Thread 0x7fffe1aa5700 (LWP 10439)):
#0  0x00007ffff3d9567d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00005555557bf7e4 in poll () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#2  0x00007ffff21db9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007ffff21dbb51 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007ffff22033d5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x00007ffff5c02494 in start_thread (arg=0x7fffe1aa5700) at pthread_create.c:333
#7  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 2 (Thread 0x7fffe763c700 (LWP 10438)):
#0  0x00007ffff3d9567d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00005555557bf7e4 in poll () at /usr/include/boost/exception/detail/exception_ptr.hpp:144
#2  0x00007fffefe58150 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#3  0x00007fffefe59ee9 in xcb_wait_for_event () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#4  0x00007fffe9d9db69 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5XcbQpa.so.5
#5  0x00007ffff65d0da8 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ffff5c02494 in start_thread (arg=0x7fffe763c700) at pthread_create.c:333
#7  0x00007ffff3d9eacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7ffff7e47e00 (LWP 10434)):
#0  0x00007ffff6765972 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#1  0x00007ffff676aca9 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#2  0x00007ffff7300737 in QTableView::setSortingEnabled(bool) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#3  0x0000555555c75c9c in TransactionView::setModel (this=0x60f00006dd20, _model=0x610000472c40) at qt/transactionview.cpp:232
#4  0x0000555555ad1dee in WalletView::setWalletModel (this=0x60d007434ca0, _walletModel=0x610000472c40) at qt/walletview.cpp:128
#5  0x0000555555a9897d in WalletFrame::addWallet (this=0x60800005eb20, walletModel=0x610000472c40) at qt/walletframe.cpp:57
#6  0x00005555558d8b5b in BitcoinGUI::addWallet (this=0x614000023440, walletModel=0x610000472c40) at qt/bitcoingui.cpp:531
#7  0x000055555587feda in BitcoinApplication::addWallet (this=0x7fffffffd430, walletModel=0x610000472c40) at qt/bitcoin.cpp:431
#8  0x000055555587f045 in BitcoinApplication::initializeResult (this=0x7fffffffd430, success=true) at qt/bitcoin.cpp:485
#9  0x00005555558abd8c in QtPrivate::FunctorCall<QtPrivate::IndexesList<0>, QtPrivate::List<bool>, void, void (BitcoinApplication::*)(bool)>::call (f=(void (BitcoinApplication::*)(BitcoinApplication * const, bool)) 0x55555587e330 <BitcoinApplication::initializeResult(bool)>, o=0x7fffffffd430, arg=0x60200038e710) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:141
#10 0x00005555558ab9f2 in QtPrivate::FunctionPointer<void (BitcoinApplication::*)(bool)>::call<QtPrivate::List<bool>, void> (f=(void (BitcoinApplication::*)(BitcoinApplication * const, bool)) 0x55555587e330 <BitcoinApplication::initializeResult(bool)>, o=0x7fffffffd430, arg=0x60200038e710) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:160
#11 0x00005555558ab5c7 in QtPrivate::QSlotObject<void (BitcoinApplication::*)(bool), QtPrivate::List<bool>, void>::impl (which=1, this_=0x6030001b36d0, r=0x7fffffffd430, a=0x60200038e710, ret=0x0) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobject_impl.h:120
#12 0x00007ffff67cd499 in QObject::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#13 0x00007ffff6ae4d54 in QGuiApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5
#14 0x00007ffff708992f in QApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#15 0x00007ffff7083b8c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#16 0x00007ffff708b341 in QApplication::notify(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#17 0x00007ffff67a09e0 in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#18 0x00007ffff67a316d in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#19 0x00007ffff67f4c43 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#20 0x00007ffff21db7f7 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#21 0x00007ffff21dba60 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#22 0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#23 0x00007ffff67f504f in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#24 0x00007ffff679e9ca in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#25 0x00007ffff67a713c in QCoreApplication::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#26 0x0000555555887a76 in main (argc=2, argv=0x7fffffffe4c8) at qt/bitcoin.cpp:714
(gdb) 
@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Dec 12, 2018

@MarcoFalke I pretty much always run with the same 5 wallets loaded. One of them is almost 8 years old so it could be related to that I guess.

I'm willing to spend some time attempting to debug this but have no idea where to start. What makes you think it's caused by particular transactions?

Do you have a testnet wallet that reliably crashes on startup, or is it intermittent?

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Dec 12, 2018

It is intermittent and I can not reproduce with a fresh wallet at all.

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Dec 12, 2018

I tried using the "thread sanitizer":

$ ./configure CXX=clang++-6.0 CXXFLAGS="-fno-omit-frame-pointer" --enable-debug --with-incompatible-bdb --with-miniupnpc=no --with-sanitizers=thread
$ make
$ cd src
$ TSAN_OPTIONS="second_deadlock_stack=1 suppressions=/path/to/suppress.txt" gdb -ex run qt/bitcoin-qt

It produced enormous amounts of output, mostly about potential deadlocks. One of the things it found mentioned addWallet() and appears to show thread 20 free()ing something that the main thread wrote to without acquiring a semaphore first, which I guess could cause a crash if those things happened to happen in the opposite order:

WARNING: ThreadSanitizer: data race (pid=9412)
  Write of size 8 at 0x7b0400000040 by thread T20:
    #0 free ??:? (bitcoin-qt+0xcac87)
    #1 tzset_internal /build/glibc-yWQXbR/glibc-2.24/time/tzset.c:437 (libc.so.6+0xa9831)
    #2 void boost::_bi::list6<boost::_bi::value<ClientModel*>, boost::arg<1>, boost::arg<2>, boost::arg<3>, boost::arg<4>, boost::_bi::value<bool> >::operator()<void (*)(ClientModel*, bool, int, long, double, bool), boost::_bi::rrlist4<bool, int, long, double> >(boost::_bi::type<void>, void (*&)(ClientModel*, bool, int, long, double, bool), boost::_bi::rrlist4<bool, int, long, double>&, int) /usr/include/boost/bind/bind.hpp:602 (bitcoin-qt+0x194e35)
    #3 void boost::_bi::bind_t<void, void (*)(ClientModel*, bool, int, long, double, bool), boost::_bi::list6<boost::_bi::value<ClientModel*>, boost::arg<1>, boost::arg<2>, boost::arg<3>, boost::arg<4>, boost::_bi::value<bool> > >::operator()<bool, int, long, double>(bool&&, int&&, long&&, double&&) /usr/include/boost/bind/bind.hpp:1342 (bitcoin-qt+0x194b5c)
    #4 std::_Function_handler<void (bool, int, long, double), boost::_bi::bind_t<void, void (*)(ClientModel*, bool, int, long, double, bool), boost::_bi::list6<boost::_bi::value<ClientModel*>, boost::arg<1>, boost::arg<2>, boost::arg<3>, boost::arg<4>, boost::_bi::value<bool> > > >::_M_invoke(std::_Any_data const&, bool&&, int&&, long&&, double&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731 (bitcoin-qt+0x19489f)
    #5 std::function<void (bool, int, long, double)>::operator()(bool, int, long, double) const /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127 (bitcoin-qt+0x2ff079)
    #6 operator() /home/me/Programs/bitcoin/src/interfaces/node.cpp:292 (bitcoin-qt+0x2f645e)
    #7 std::_Function_handler<void (bool, CBlockIndex const*), interfaces::(anonymous namespace)::NodeImpl::handleNotifyHeaderTip(std::function<void (bool, int, long, double)>)::{lambda(bool, CBlockIndex const*)#1}>::_M_invoke(std::_Any_data const&, bool&&, CBlockIndex const*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731 (bitcoin-qt+0x2f6130)
    #8 std::function<void (bool, CBlockIndex const*)>::operator()(bool, CBlockIndex const*) const /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127 (bitcoin-qt+0x6134df)
    #9 boost::detail::function::void_function_obj_invoker2<std::function<void (bool, CBlockIndex const*)>, void, bool, CBlockIndex const*>::invoke(boost::detail::function::function_buffer&, bool, CBlockIndex const*) /usr/include/boost/function/function_template.hpp:159 (bitcoin-qt+0x6130e0)
    #10 boost::function2<void, bool, CBlockIndex const*>::operator()(bool, CBlockIndex const*) const /usr/include/boost/function/function_template.hpp:770 (bitcoin-qt+0x622d87)
    #11 boost::signals2::detail::void_type boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::m_invoke<boost::function<void (bool, CBlockIndex const*)>, 0u, 1u, bool&, CBlockIndex const*&>(boost::function<void (bool, CBlockIndex const*)>&, boost::signals2::detail::unsigned_meta_array<0u, 1u>, std::tuple<bool&, CBlockIndex const*&> const&, boost::enable_if<boost::is_void<boost::function<void (bool, CBlockIndex const*)>::result_type>, void>::type*) const /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:105 (bitcoin-qt+0x622ca2)
    #12 boost::signals2::detail::void_type boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::operator()<boost::function<void (bool, CBlockIndex const*)>, bool&, CBlockIndex const*&, 2ul>(boost::function<void (bool, CBlockIndex const*)>&, std::tuple<bool&, CBlockIndex const*&> const&, mpl_::size_t<2ul>) const /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:90 (bitcoin-qt+0x622c08)
    #13 boost::signals2::detail::void_type boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, bool, CBlockIndex const*>::operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >(boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > const&) const /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:133 (bitcoin-qt+0x622b9d)
    #14 boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, bool, CBlockIndex const*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> >::dereference() const /usr/include/boost/signals2/detail/slot_call_iterator.hpp:110 (bitcoin-qt+0x622ab6)
    #15 boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, bool, CBlockIndex const*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> >::reference boost::iterators::iterator_core_access::dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, bool, CBlockIndex const*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, bool, CBlockIndex const*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > const&) /usr/include/boost/iterator/iterator_facade.hpp:549 (bitcoin-qt+0x6229c6)
    #16 boost::iterators::detail::iterator_facade_base<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, bool, CBlockIndex const*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> >, boost::signals2::detail::void_type, boost::iterators::single_pass_traversal_tag, boost::signals2::detail::void_type&, long, false, false>::operator*() const /usr/include/boost/iterator/iterator_facade.hpp:655 (bitcoin-qt+0x62282e)
    #17 void boost::signals2::optional_last_value<void>::operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, bool, CBlockIndex const*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, bool, CBlockIndex const*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, bool, CBlockIndex const*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> >) const /usr/include/boost/signals2/optional_last_value.hpp:57 (bitcoin-qt+0x62271b)
    #18 void boost::signals2::detail::combiner_invoker<void>::operator()<boost::signals2::optional_last_value<void>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, bool, CBlockIndex const*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >(boost::signals2::optional_last_value<void>&, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, bool, CBlockIndex const*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, bool, CBlockIndex const*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (bool, CBlockIndex const*), boost::function<void (bool, CBlockIndex const*)> >, boost::signals2::mutex> >) const /usr/include/boost/signals2/detail/result_type_wrapper.hpp:64 (bitcoin-qt+0x62224c)
    #19 boost::signals2::detail::signal_impl<void (bool, CBlockIndex const*), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (bool, CBlockIndex const*)>, boost::function<void (boost::signals2::connection const&, bool, CBlockIndex const*)>, boost::signals2::mutex>::operator()(bool, CBlockIndex const*) /usr/include/boost/signals2/detail/signal_template.hpp:242 (bitcoin-qt+0x621ee5)
    #20 boost::signals2::signal<void (bool, CBlockIndex const*), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (bool, CBlockIndex const*)>, boost::function<void (boost::signals2::connection const&, bool, CBlockIndex const*)>, boost::signals2::mutex>::operator()(bool, CBlockIndex const*) /usr/include/boost/signals2/detail/signal_template.hpp:723 (bitcoin-qt+0x5b7b22)
    #21 CClientUIInterface::NotifyHeaderTip(bool, CBlockIndex const*) /home/me/Programs/bitcoin/src/ui_interface.cpp:58 (bitcoin-qt+0x5b5f06)
    #22 NotifyHeaderTip() /home/me/Programs/bitcoin/src/validation.cpp:2643 (bitcoin-qt+0x632336)
    #23 ProcessNewBlockHeaders(std::vector<CBlockHeader, std::allocator<CBlockHeader> > const&, CValidationState&, CChainParams const&, CBlockIndex const**, CBlockHeader*) /home/me/Programs/bitcoin/src/validation.cpp:3422 (bitcoin-qt+0x632215)
    #24 ProcessHeadersMessage(CNode*, CConnman*, std::vector<CBlockHeader, std::allocator<CBlockHeader> > const&, CChainParams const&, bool) /home/me/Programs/bitcoin/src/net_processing.cpp:1412 (bitcoin-qt+0x3c4bbb)
    #25 ProcessMessage(CNode*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, CDataStream&, long, CChainParams const&, CConnman*, std::atomic<bool> const&, bool) /home/me/Programs/bitcoin/src/net_processing.cpp:2712 (bitcoin-qt+0x3b39b1)
    #26 PeerLogicValidation::ProcessMessages(CNode*, std::atomic<bool>&) /home/me/Programs/bitcoin/src/net_processing.cpp:3052 (bitcoin-qt+0x3aca7a)
    #27 non-virtual thunk to PeerLogicValidation::ProcessMessages(CNode*, std::atomic<bool>&) ??:? (bitcoin-qt+0x3b61fa)
    #28 CConnman::ThreadMessageHandler() /home/me/Programs/bitcoin/src/net.cpp:2055 (bitcoin-qt+0x35b672)
    #29 void std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&>(std::__invoke_memfun_deref, void (CConnman::* const&)(), CConnman*&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227 (bitcoin-qt+0x39a2da)
    #30 std::result_of<void (CConnman::* const&(CConnman*&))()>::type std::__invoke<void (CConnman::* const&)(), CConnman*&>(void (CConnman::* const&)(), CConnman*&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250 (bitcoin-qt+0x39a1d4)
    #31 _ZNKSt12_Mem_fn_baseIM8CConnmanFvvELb1EEclIJRPS0_EEEDTclsr3stdE8__invokedtdefpT6_M_pmfspclsr3stdE7forwardIT_Efp_EEEDpOS7_ /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604 (bitcoin-qt+0x39a0b9)
    #32 void std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933 (bitcoin-qt+0x39a019)
    #33 void std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991 (bitcoin-qt+0x399f42)
    #34 std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731 (bitcoin-qt+0x399abe)
    #35 std::function<void ()>::operator()() const /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127 (bitcoin-qt+0x1f3a59)
    #36 void TraceThread<std::function<void ()> >(char const*, std::function<void ()>) /home/me/Programs/bitcoin/src/./util/system.h:331 (bitcoin-qt+0x314e74)
    #37 void std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390 (bitcoin-qt+0x39b189)
    #38 std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380 (bitcoin-qt+0x39b0c6)
    #39 std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197 (bitcoin-qt+0x39a9ba)
    #40 std::error_code::default_error_condition() const ??:? (libstdc++.so.6+0xb9e6e)

  Previous write of size 8 at 0x7b0400000040 by main thread:
    #0 malloc ??:? (bitcoin-qt+0xf0488)
    #1 __GI___strdup /build/glibc-yWQXbR/glibc-2.24/string/strdup.c:42 (libc.so.6+0x803b9)
    #2 QTime::QTime(int, int, int, int) ??:? (libQt5Core.so.5+0xf09b8)
    #3 OverviewPage /home/me/Programs/bitcoin/src/qt/overviewpage.cpp:120 (bitcoin-qt+0x274079)
    #4 WalletView /home/me/Programs/bitcoin/src/qt/walletview.cpp:40 (bitcoin-qt+0x235d12)
    #5 WalletFrame::addWallet(WalletModel*) /home/me/Programs/bitcoin/src/qt/walletframe.cpp:54 (bitcoin-qt+0x21aa3a)
    #6 BitcoinGUI::addWallet(WalletModel*) /home/me/Programs/bitcoin/src/qt/bitcoingui.cpp:531 (bitcoin-qt+0x17ebef)
    #7 BitcoinApplication::addWallet(WalletModel*) /home/me/Programs/bitcoin/src/qt/bitcoin.cpp:431 (bitcoin-qt+0x15e60f)
    #8 BitcoinApplication::initializeResult(bool) /home/me/Programs/bitcoin/src/qt/bitcoin.cpp:485 (bitcoin-qt+0x15e181)
    #9 QtPrivate::FunctorCall<QtPrivate::IndexesList<0>, QtPrivate::List<bool>, void, void (BitcoinApplication::*)(bool)>::call(void (BitcoinApplication::*)(bool), BitcoinApplication*, void**) /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:141 (bitcoin-qt+0x171697)
    #10 void QtPrivate::FunctionPointer<void (BitcoinApplication::*)(bool)>::call<QtPrivate::List<bool>, void>(void (BitcoinApplication::*)(bool), BitcoinApplication*, void**) /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:160 (bitcoin-qt+0x1715e1)
    #11 QtPrivate::QSlotObject<void (BitcoinApplication::*)(bool), QtPrivate::List<bool>, void>::impl(int, QtPrivate::QSlotObjectBase*, QObject*, void**, bool*) /usr/include/x86_64-linux-gnu/qt5/QtCore/qobject_impl.h:120 (bitcoin-qt+0x171516)
    #12 QObject::event(QEvent*) ??:? (libQt5Core.so.5+0x2ab498)
    #13 __libc_start_main /build/glibc-yWQXbR/glibc-2.24/csu/../csu/libc-start.c:291 (libc.so.6+0x202e0)

  Thread T20 'bitcoin-msghand' (tid=9474, running) created by thread T5 at:
    #0 pthread_create ??:? (bitcoin-qt+0xcbcd6)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ??:? (libstdc++.so.6+0xba164)
    #2 CConnman::Start(CScheduler&, CConnman::Options const&) /home/me/Programs/bitcoin/src/net.cpp:2385 (bitcoin-qt+0x35d5c5)
    #3 AppInitMain(InitInterfaces&) /home/me/Programs/bitcoin/src/init.cpp:1742 (bitcoin-qt+0x30e1be)
    #4 interfaces::(anonymous namespace)::NodeImpl::appInitMain() /home/me/Programs/bitcoin/src/interfaces/node.cpp:73 (bitcoin-qt+0x2f2e8a)
    #5 BitcoinCore::initialize() /home/me/Programs/bitcoin/src/qt/bitcoin.cpp:254 (bitcoin-qt+0x15d5f8)
    #6 QtPrivate::FunctorCall<QtPrivate::IndexesList<>, QtPrivate::List<>, void, void (BitcoinCore::*)()>::call(void (BitcoinCore::*)(), BitcoinCore*, void**) /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:141 (bitcoin-qt+0x171f2c)
    #7 void QtPrivate::FunctionPointer<void (BitcoinCore::*)()>::call<QtPrivate::List<>, void>(void (BitcoinCore::*)(), BitcoinCore*, void**) /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:160 (bitcoin-qt+0x171e91)
    #8 QtPrivate::QSlotObject<void (BitcoinCore::*)(), QtPrivate::List<>, void>::impl(int, QtPrivate::QSlotObjectBase*, QObject*, void**, bool*) /usr/include/x86_64-linux-gnu/qt5/QtCore/qobject_impl.h:120 (bitcoin-qt+0x171dc6)
    #9 QObject::event(QEvent*) ??:? (libQt5Core.so.5+0x2ab498)

SUMMARY: ThreadSanitizer: data race ??:? in free

I don't know if that's relevant or not, but perhaps this 'thread sanitizer' thing is useful here.

@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented Dec 17, 2018

Looking at stack trace from #9683 (comment), nothing obvious stood out.

Active threads are:

  • Thread 1 which is crashing calling setSortingEnabled
  • Thread 10 which is a scheduler thread sending TransactionAddedToMempool to the wallet and is blocked in CWallet::TransactionAddedToMempool waiting for cs_main
  • Thread 21 which is a net processing thread calling vector::push_back

The three threads seems unrelated, so if this is a race, it doesn't look like any of these are interfering with the other.

Since crash is happening in sort code, I also took a look there for anything that might be uninitialized. The only things I noticed was that we were not calling setSortRole or choosing a default sort column. Patch below does these things and might be worth trying:

--- a/src/qt/transactionview.cpp
+++ b/src/qt/transactionview.cpp
@@ -218,6 +218,7 @@ void TransactionView::setModel(WalletModel *_model)
     {
         transactionProxyModel = new TransactionFilterProxy(this);
         transactionProxyModel->setSourceModel(_model->getTransactionTableModel());
+        transactionProxyModel->setSortRole(Qt::EditRole);
         transactionProxyModel->setDynamicSortFilter(true);
         transactionProxyModel->setSortCaseSensitivity(Qt::CaseInsensitive);
         transactionProxyModel->setFilterCaseSensitivity(Qt::CaseInsensitive);
@@ -229,8 +230,8 @@ void TransactionView::setModel(WalletModel *_model)
         transactionView->setAlternatingRowColors(true);
         transactionView->setSelectionBehavior(QAbstractItemView::SelectRows);
         transactionView->setSelectionMode(QAbstractItemView::ExtendedSelection);
+        transactionView->horizontalHeader()->setSortIndicator(TransactionTableModel::Date, Qt::DescendingOrder);
         transactionView->setSortingEnabled(true);
-        transactionView->sortByColumn(TransactionTableModel::Date, Qt::DescendingOrder);
         transactionView->verticalHeader()->hide();
 
         transactionView->setColumnWidth(TransactionTableModel::Status, STATUS_COLUMN_WIDTH);
@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Dec 19, 2018

Thanks @ryanofsky. I'll try that patch and report back.

I'm having a hard time reproducing the crash at the moment. It just took 30 attempts at starting and stopping bitcoin-qt to get a single crash. Last time I looked at it it was crashing maybe 1 in 3 times. I hadn't rebuilt the bitcoin-qt binary since so I don't know what changed. I guess it's related to what else is going on on the computer at the same time.

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Dec 19, 2018

After applying @ryanofsky's patch the first crash I got said this in stdout:

Assertion failed: detected inconsistent lock order at sync.cpp:104, details in debug log.

This in the log file:

2018-12-19T15:38:35Z New outbound peer connected: version: 70015, blocks=554504, peer=0
2018-12-19T15:38:35Z POTENTIAL DEADLOCK DETECTED
2018-12-19T15:38:35Z Previous lock order was:
2018-12-19T15:38:35Z  pnode->cs_sendProcessing  net.cpp:2061
2018-12-19T15:38:35Z  (1) ppmutexOpenSSL[i]  util/system.cpp:94
2018-12-19T15:38:35Z  (2) cs_main  net_processing.cpp:3291 (TRY)
2018-12-19T15:38:35Z Current lock order is:
2018-12-19T15:38:35Z  (2) cs_main  init.cpp:1429
2018-12-19T15:38:35Z  (2) cs_main  validation.cpp:4080
2018-12-19T15:38:35Z  (1) ppmutexOpenSSL[i]  util/system.cpp:94

and here's a stack trace:

(gdb) thread apply all where

Thread 21 (Thread 0x7fff6960c700 (LWP 12435)):
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff40f642a in __GI_abort () at abort.c:89
#2  0x0000555555aa26a0 in potential_deadlock_detected (mismatch=..., s1=std::vector of length 3, capacity 3 = {...}, s2=...) at sync.cpp:105
#3  0x0000555555aa1d24 in push_lock (c=<optimized out>, locklocation=...) at sync.cpp:128
#4  0x0000555555aa1b22 in EnterCritical (pszName=<optimized out>, pszFile=<optimized out>, nLine=<optimized out>, cs=0x2, fTry=<optimized out>) at sync.cpp:139
#5  0x0000555555725be4 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::TryEnter (this=0x7fff6960b6c0, pszName=0x2 <error: Cannot access memory at address 0x2>, pszFile=0x7fff6960b150 "", nLine=0) at ./sync.h:136
#6  0x0000555555724504 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::UniqueLock (this=0x7fff6960b6c0, mutexIn=..., pszName=0x5555560a4119 "cs_main", pszFile=0x5555560abbb4 "net_processing.cpp", nLine=3291, fTry=true) at ./sync.h:147
#7  0x00005555557b7d37 in PeerLogicValidation::SendMessages (this=<optimized out>, pto=<optimized out>) at net_processing.cpp:3291
#8  0x00005555557bbdee in non-virtual thunk to PeerLogicValidation::SendMessages(CNode*) () at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/move.h:77
#9  0x0000555555771b9b in CConnman::ThreadMessageHandler (this=0x7fffc93256e0) at net.cpp:2062
#10 0x00005555557a0c66 in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=@0x7fffcae86150: (void (CConnman::*)(CConnman * const)) 0x555555771990 <CConnman::ThreadMessageHandler()>, __t=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#11 0x00005555557a0bc6 in std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=<optimized out>, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#12 0x00005555557a0ae9 in std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=0x2, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#13 0x00005555557a0a71 in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x7fffcae86150, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#14 0x00005555557a09ca in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=0x2) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#15 0x00005555557a0682 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#16 0x0000555555660775 in std::function<void ()>::operator()() const (this=0x7fff6960bcb0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#17 0x000055555573cbfe in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x5555560ab34f "msghand", func=...) at ./util/system.h:331
#18 0x00005555557a1748 in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#19 0x00005555557a16ba in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=0x2) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#20 0x00005555557a119e in std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#21 0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#22 0x00007ffff5c02494 in start_thread (arg=0x7fff6960c700) at pthread_create.c:333
#23 0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 20 (Thread 0x7fff69e5d700 (LWP 12434)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x000055555573a0ba in __gthread_cond_timedwait (__cond=<optimized out>, __mutex=<optimized out>, __abs_timeout=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:871
#2  0x0000555555761dd6 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fffc9375b8c, __lock=..., __atime=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:166
#3  0x0000555555761cba in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fffc9375b8c, __lock=..., __atime=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:106
#4  0x0000555555aa9f2e in std::condition_variable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=0x7fffc9375b88, __lock=..., __atime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:129
#5  0x0000555555aa9d1b in std::condition_variable::wait_for<long, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=0x7fffc9375b8c, __lock=..., __rtime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:145
#6  0x0000555555aa9c78 in CThreadInterrupt::sleep_for (this=0x7fffc9375b88, rel_time=...) at threadinterrupt.cpp:34
#7  0x0000555555770e7c in CConnman::ThreadOpenConnections (this=0x7fffc93256e0, connect=...) at net.cpp:1770
#8  0x00005555557a2de2 in std::__invoke_impl<void, void (CConnman::* const&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (__f=<optimized out>, __t=<optimized out>, __args=warning: Type size unknown, assuming 1. Try casting to a known type, or void *.
warning: Type size unknown, assuming 1. Try casting to a known type, or void *.
Python Exception <class 'gdb.error'> Cannot perform pointer math on incomplete type "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >", try casting to a known type, or void *.: 
std::vector of length 32, capacity 32) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#9  0x00005555557a2d16 in std::__invoke<void (CConnman::* const&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (__fn=<optimized out>, __args=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#10 0x00005555557a2bee in std::_Mem_fn_base<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), true>::operator()<CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (this=0x7fffc9375b8c, __args=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#11 0x00005555557a2b85 in std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=0x7fff9aaac700, __args=<unknown type in /home/me/Programs/bitcoin/src/qt/bitcoin-qt, CU 0xd32f11, DIE 0xdbee56>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#12 0x00005555557a2aba in std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)>::operator()<, void>() (this=0x7fffc9375b8c) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#13 0x00005555557a25a2 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#14 0x0000555555660775 in std::function<void ()>::operator()() const (this=0x7fff69e5ccb0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#15 0x000055555573cbfe in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x5555560ab347 "opencon", func=...) at ./util/system.h:331
#16 0x00005555557a1748 in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#17 0x00005555557a16ba in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=0x7fffc9375b8c) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#18 0x00005555557a119e in std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#19 0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#20 0x00007ffff5c02494 in start_thread (arg=0x7fff69e5d700) at pthread_create.c:333
#21 0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 19 (Thread 0x7fff6a6ae700 (LWP 12433)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x000055555573a0ba in __gthread_cond_timedwait (__cond=<optimized out>, __mutex=<optimized out>, __abs_timeout=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:871
#2  0x0000555555761dd6 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fffc9375b8c, __lock=..., __atime=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:166
#3  0x0000555555761cba in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fffc9375b8c, __lock=..., __atime=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:106
#4  0x0000555555aa9f2e in std::condition_variable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=0x7fffc9375b88, __lock=..., __atime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:129
#5  0x0000555555aa9d1b in std::condition_variable::wait_for<long, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=0x7fffc9375b8c, __lock=..., __rtime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:145
#6  0x0000555555aa9c78 in CThreadInterrupt::sleep_for (this=0x7fffc9375b88, rel_time=...) at threadinterrupt.cpp:34
#7  0x0000555555aa9d71 in CThreadInterrupt::sleep_for (this=0x7fffc9375b8c, rel_time=...) at threadinterrupt.cpp:39
#8  0x0000555555771917 in CConnman::ThreadOpenAddedConnections (this=0x7fffc93256e0) at net.cpp:1989
#9  0x00005555557a0c66 in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=@0x7fff9e3983c0: (void (CConnman::*)(CConnman * const)) 0x555555771730 <CConnman::ThreadOpenAddedConnections()>, __t=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#10 0x00005555557a0bc6 in std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=<optimized out>, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#11 0x00005555557a0ae9 in std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=0x7fffc9375b8c, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#12 0x00005555557a0a71 in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x7fff9e3983c0, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#13 0x00005555557a09ca in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=0x7fffc9375b8c) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#14 0x00005555557a0682 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#15 0x0000555555660775 in std::function<void ()>::operator()() const (this=0x7fff6a6adcb0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#16 0x000055555573cbfe in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x5555560ab2e4 "addcon", func=...) at ./util/system.h:331
#17 0x00005555557a1748 in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#18 0x00005555557a16ba in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=0x7fffc9375b8c) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#19 0x00005555557a119e in std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#20 0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#21 0x00007ffff5c02494 in start_thread (arg=0x7fff6a6ae700) at pthread_create.c:333
#22 0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 18 (Thread 0x7fff6aeff700 (LWP 12432)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ffff5c04b95 in __GI___pthread_mutex_lock (mutex=0x5555566bf7d8 <lockdata+104>) at ../nptl/pthread_mutex_lock.c:80
#2  0x0000555555739cf4 in __gthread_mutex_lock (__mutex=0x5555566bf7d8 <lockdata+104>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:748
#3  0x0000555555753fca in std::mutex::lock (this=0x5555566bf7d8 <lockdata+104>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/std_mutex.h:103
#4  0x000055555577a710 in std::lock_guard<std::mutex>::lock_guard (this=<optimized out>, __m=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/std_mutex.h:162
#5  0x0000555555aa1b9e in push_lock (c=<optimized out>, locklocation=...) at sync.cpp:112
#6  0x0000555555aa1b22 in EnterCritical (pszName=<optimized out>, pszFile=<optimized out>, nLine=<optimized out>, cs=0x5555566bf7d8 <lockdata+104>, fTry=<optimized out>) at sync.cpp:139
#7  0x0000555555725c71 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::Enter (this=0x7fff6aeee960, pszName=0x5555566bf7d8 <lockdata+104> "\002", pszFile=0x80 <error: Cannot access memory at address 0x80>, nLine=0) at ./sync.h:123
#8  0x0000555555724517 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::UniqueLock (this=0x7fff6aeee960, mutexIn=..., pszName=0x5555560aa7e5 "cs_vNodes", pszFile=0x5555560bc58b "net.cpp", nLine=1360, fTry=false) at ./sync.h:149
#9  0x000055555576ec74 in CConnman::SocketHandler (this=0x7fffc93256e0) at net.cpp:1360
#10 0x000055555576f598 in CConnman::ThreadSocketHandler (this=0x7fffc93256e0) at net.cpp:1466
#11 0x00005555557a0c66 in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=@0x7fffc9c623b0: (void (CConnman::*)(CConnman * const)) 0x55555576f550 <CConnman::ThreadSocketHandler()>, __t=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#12 0x00005555557a0bc6 in std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=<optimized out>, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#13 0x00005555557a0ae9 in std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=0x5555566bf7d8 <lockdata+104>, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#14 0x00005555557a0a71 in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x7fffc9c623b0, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#15 0x00005555557a09ca in std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=0x5555566bf7d8 <lockdata+104>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#16 0x00005555557a0682 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#17 0x0000555555660775 in std::function<void ()>::operator()() const (this=0x7fff6aefecb0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#18 0x000055555573cbfe in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x5555560bc87c "net", func=...) at ./util/system.h:331
#19 0x00005555557a1748 in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#20 0x00005555557a16ba in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=0x5555566bf7d8 <lockdata+104>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#21 0x00005555557a119e in std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#22 0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#23 0x00007ffff5c02494 in start_thread (arg=0x7fff6aeff700) at pthread_create.c:333
#24 0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 17 (Thread 0x7fff9759b700 (LWP 12431)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ffff5c04b95 in __GI___pthread_mutex_lock (mutex=0x5555566bf7d8 <lockdata+104>) at ../nptl/pthread_mutex_lock.c:80
#2  0x0000555555739cf4 in __gthread_mutex_lock (__mutex=0x5555566bf7d8 <lockdata+104>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:748
#3  0x0000555555753fca in std::mutex::lock (this=0x5555566bf7d8 <lockdata+104>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/std_mutex.h:103
#4  0x000055555577a710 in std::lock_guard<std::mutex>::lock_guard (this=<optimized out>, __m=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/std_mutex.h:162
#5  0x0000555555aa1b9e in push_lock (c=<optimized out>, locklocation=...) at sync.cpp:112
#6  0x0000555555aa1b22 in EnterCritical (pszName=<optimized out>, pszFile=<optimized out>, nLine=<optimized out>, cs=0x5555566bf7d8 <lockdata+104>, fTry=<optimized out>) at sync.cpp:139
#7  0x0000555555725c71 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::Enter (this=0x7fff9759a598, pszName=0x5555566bf7d8 <lockdata+104> "\002", pszFile=0x80 <error: Cannot access memory at address 0x80>, nLine=0) at ./sync.h:123
#8  0x0000555555724517 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::UniqueLock (this=0x7fff9759a598, mutexIn=..., pszName=0x555556127e5c "pool.cs", pszFile=0x5555560ec903 "validation.cpp", nLine=567, fTry=false) at ./sync.h:149
#9  0x00005555559ae95d in AcceptToMemoryPoolWorker (chainparams=..., pool=..., state=..., ptx=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 1, weak 0) 0x7fff8813c430, pfMissingInputs=0x0, nAcceptTime=1545233607, plTxnReplaced=0x0, bypass_limits=248, nAbsurdFee=@0x7fff9759aa00: 0, coins_to_uncache=std::vector of length 0, capacity 0, test_accept=false) at validation.cpp:567
#10 0x000055555599a6ee in AcceptToMemoryPoolWithTime (chainparams=..., pool=..., state=..., tx=<error reading variable: Cannot access memory at address 0x841f0fc35a4162>, pfMissingInputs=0x5555566bf7d8 <lockdata+104>, nAcceptTime=0, plTxnReplaced=0x0, bypass_limits=<optimized out>, nAbsurdFee=0, test_accept=<optimized out>) at validation.cpp:974
#11 0x00005555559ad513 in LoadMempool () at validation.cpp:4723
#12 0x00005555557397cb in ThreadImport (vImportFiles=std::vector of length 2932031665529, capacity 0 = {...}) at init.cpp:700
#13 0x00005555557617e0 in boost::_bi::list1<boost::_bi::value<std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> > > >::operator()<void (*)(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >), boost::_bi::list0> (this=<optimized out>, f=<optimized out>, a=...) at /usr/include/boost/bind/bind.hpp:259
#14 0x0000555555761768 in boost::_bi::bind_t<void, void (*)(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >), boost::_bi::list1<boost::_bi::value<std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> > > > >::operator() (this=0x7fff9aaabe78) at /usr/include/boost/bind/bind.hpp:1294
#15 0x00005555557613a1 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >), boost::_bi::list1<boost::_bi::value<std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> > > > > >::run (this=<optimized out>) at /usr/include/boost/thread/detail/thread.hpp:116
#16 0x00007ffff77a6116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
#17 0x00007ffff5c02494 in start_thread (arg=0x7fff9759b700) at pthread_create.c:333
#18 0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 16 (Thread 0x7fff97fff700 (LWP 12430)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000555555cc866b in leveldb::(anonymous namespace)::PosixEnv::BGThread (this=<optimized out>) at leveldb/util/env_posix.cc:649
#2  0x0000555555cc84e9 in leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7fffc83dd6c4) at leveldb/util/env_posix.cc:564
#3  0x00007ffff5c02494 in start_thread (arg=0x7fff97fff700) at pthread_create.c:333
#4  0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 15 (Thread 0x7fffb966a700 (LWP 12429)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff4a3050c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x0000555555a71feb in WorkQueue<HTTPClosure>::Run (this=0x7fffc8048700) at httpserver.cpp:107
#3  0x0000555555a6926a in HTTPWorkQueueRun (queue=0x7fffc804872c) at httpserver.cpp:342
#4  0x0000555555a74e21 in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fffc8049278) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x0000555555a74dba in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=0x7fffc804872c) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x0000555555a749ee in std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffb966a700) at pthread_create.c:333
#9  0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 14 (Thread 0x7fffb9ebb700 (LWP 12428)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff4a3050c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x0000555555a71feb in WorkQueue<HTTPClosure>::Run (this=0x7fffc8048700) at httpserver.cpp:107
#3  0x0000555555a6926a in HTTPWorkQueueRun (queue=0x7fffc804872c) at httpserver.cpp:342
#4  0x0000555555a74e21 in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fffc8048e18) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x0000555555a74dba in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=0x7fffc804872c) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x0000555555a749ee in std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffb9ebb700) at pthread_create.c:333
#9  0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 13 (Thread 0x7fffba70c700 (LWP 12427)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff4a3050c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x0000555555a71feb in WorkQueue<HTTPClosure>::Run (this=0x7fffc8048700) at httpserver.cpp:107
#3  0x0000555555a6926a in HTTPWorkQueueRun (queue=0x7fffc804872c) at httpserver.cpp:342
#4  0x0000555555a74e21 in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fffc8049298) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x0000555555a74dba in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=0x7fffc804872c) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x0000555555a749ee in std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffba70c700) at pthread_create.c:333
#9  0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 12 (Thread 0x7fffbaf5d700 (LWP 12426)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff4a3050c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x0000555555a71feb in WorkQueue<HTTPClosure>::Run (this=0x7fffc8048700) at httpserver.cpp:107
#3  0x0000555555a6926a in HTTPWorkQueueRun (queue=0x7fffc804872c) at httpserver.cpp:342
#4  0x0000555555a74e21 in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fffc8048e38) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x0000555555a74dba in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=0x7fffc804872c) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x0000555555a749ee in std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffbaf5d700) at pthread_create.c:333
#9  0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 11 (Thread 0x7fffbb7ae700 (LWP 12425)):
#0  0x00007ffff41ab0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007ffff53efd98 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#2  0x00007ffff53da07a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#3  0x0000555555a69204 in ThreadHTTP (base=0x7fffc8047e40) at httpserver.cpp:290
#4  0x0000555555a74121 in std::_Bind_simple<bool (*(event_base*))(event_base*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fffc8048b18) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
#5  0x0000555555a740ba in std::_Bind_simple<bool (*(event_base*))(event_base*)>::operator()() (this=0x16) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
#6  0x0000555555a73cee in std::thread::_State_impl<std::_Bind_simple<bool (*(event_base*))(event_base*)> >::_M_run() (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
#7  0x00007ffff4a35e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffbb7ae700) at pthread_create.c:333
#9  0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 10 (Thread 0x7fffbbfff700 (LWP 12424)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ffff5c04b95 in __GI___pthread_mutex_lock (mutex=0x5555566bf7d8 <lockdata+104>) at ../nptl/pthread_mutex_lock.c:80
#2  0x0000555555739cf4 in __gthread_mutex_lock (__mutex=0x5555566bf7d8 <lockdata+104>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:748
#3  0x0000555555753fca in std::mutex::lock (this=0x5555566bf7d8 <lockdata+104>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/std_mutex.h:103
#4  0x000055555577a710 in std::lock_guard<std::mutex>::lock_guard (this=<optimized out>, __m=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/std_mutex.h:162
#5  0x0000555555aa1b9e in push_lock (c=<optimized out>, locklocation=...) at sync.cpp:112
#6  0x0000555555aa1b22 in EnterCritical (pszName=<optimized out>, pszFile=<optimized out>, nLine=<optimized out>, cs=0x5555566bf7d8 <lockdata+104>, fTry=<optimized out>) at sync.cpp:139
#7  0x0000555555725c71 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::Enter (this=0x7fffbbffe978, pszName=0x5555566bf7d8 <lockdata+104> "\002", pszFile=0x80 <error: Cannot access memory at address 0x80>, nLine=0) at ./sync.h:123
#8  0x0000555555724517 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::UniqueLock (this=0x7fffbbffe978, mutexIn=..., pszName=0x5555561322a8 "m_cs_callbacks_pending", pszFile=0x5555561321c0 "scheduler.cpp", nLine=164, fTry=false) at ./sync.h:149
#9  0x0000555555c25668 in SingleThreadedSchedulerClient::ProcessQueue (this=0x7fffc8043898) at scheduler.cpp:164
#10 0x0000555555c29586 in std::__invoke_impl<void, void (SingleThreadedSchedulerClient::* const&)(), SingleThreadedSchedulerClient*&> (__f=@0x7fffb4000b10: (void (SingleThreadedSchedulerClient::*)(SingleThreadedSchedulerClient * const)) 0x555555c25620 <SingleThreadedSchedulerClient::ProcessQueue()>, __t=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
#11 0x0000555555c294e6 in std::__invoke<void (SingleThreadedSchedulerClient::* const&)(), SingleThreadedSchedulerClient*&> (__fn=<optimized out>, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
#12 0x0000555555c29409 in std::_Mem_fn_base<void (SingleThreadedSchedulerClient::*)(), true>::operator()<SingleThreadedSchedulerClient*&> (this=0x5555566bf7d8 <lockdata+104>, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
#13 0x0000555555c293c1 in std::_Bind<std::_Mem_fn<void (SingleThreadedSchedulerClient::*)()> (SingleThreadedSchedulerClient*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x7fffb4000b10, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
#14 0x0000555555c2931a in std::_Bind<std::_Mem_fn<void (SingleThreadedSchedulerClient::*)()> (SingleThreadedSchedulerClient*)>::operator()<, void>() (this=0x5555566bf7d8 <lockdata+104>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
#15 0x0000555555c28f92 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (SingleThreadedSchedulerClient::*)()> (SingleThreadedSchedulerClient*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#16 0x0000555555660775 in std::function<void ()>::operator()() const (this=0x7fffbbffeaf0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#17 0x0000555555c24fb6 in CScheduler::serviceQueue (this=0x5555566bb2e0 <scheduler>) at scheduler.cpp:83
#18 0x000055555575d00d in boost::_mfi::mf0<void, CScheduler>::operator() (this=<optimized out>, p=<optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:49
#19 0x000055555575cf73 in boost::_bi::list1<boost::_bi::value<CScheduler*> >::operator()<boost::_mfi::mf0<void, CScheduler>, boost::_bi::list0> (this=<optimized out>, f=..., a=...) at /usr/include/boost/bind/bind.hpp:259
#20 0x000055555575cec8 in boost::_bi::bind_t<void, boost::_mfi::mf0<void, CScheduler>, boost::_bi::list1<boost::_bi::value<CScheduler*> > >::operator() (this=0x7fffb40008c0) at /usr/include/boost/bind/bind.hpp:1294
#21 0x000055555575ccf2 in std::_Function_handler<void (), boost::_bi::bind_t<void, boost::_mfi::mf0<void, CScheduler>, boost::_bi::list1<boost::_bi::value<CScheduler*> > > >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
#22 0x0000555555660775 in std::function<void ()>::operator()() const (this=0x7fffbbffec78) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#23 0x000055555573cbfe in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=0x5555561321eb "scheduler", func=...) at ./util/system.h:331
#24 0x000055555575db44 in boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > >::operator()<void (*)(char const*, std::function<void ()>), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(char const*, std::function<void ()>), boost::_bi::list0&, int) (this=<optimized out>, f=<optimized out>, a=...) at /usr/include/boost/bind/bind.hpp:319
#25 0x000055555575dab8 in boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > >::operator()() (this=0x7fffc80435a8) at /usr/include/boost/bind/bind.hpp:1294
#26 0x000055555575d831 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > > >::run() (this=<optimized out>) at /usr/include/boost/thread/detail/thread.hpp:116
#27 0x00007ffff77a6116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
#28 0x00007ffff5c02494 in start_thread (arg=0x7fffbbfff700) at pthread_create.c:333
#29 0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 9 (Thread 0x7fffccf5b700 (LWP 12423)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055555573ffdb in boost::condition_variable::wait (this=0x5555566be630 <scriptcheckqueue+40>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
#2  0x00005555559dcbe7 in CCheckQueue<CScriptCheck>::Loop (this=0x5555566be608 <scriptcheckqueue>, fMaster=false) at ./checkqueue.h:99
#3  0x00005555559b7d2c in CCheckQueue<CScriptCheck>::Thread (this=0x5555566be65c <scriptcheckqueue+84>) at ./checkqueue.h:136
#4  0x000055555599ca1d in ThreadScriptCheck () at validation.cpp:1680
#5  0x000055555575a99b in boost::detail::thread_data<void (*)()>::run (this=0x5555566be65c <scriptcheckqueue+84>) at /usr/include/boost/thread/detail/thread.hpp:116
#6  0x00007ffff77a6116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
#7  0x00007ffff5c02494 in start_thread (arg=0x7fffccf5b700) at pthread_create.c:333
#8  0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 8 (Thread 0x7fffcd7ac700 (LWP 12422)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055555573ffdb in boost::condition_variable::wait (this=0x5555566be630 <scriptcheckqueue+40>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
#2  0x00005555559dcbe7 in CCheckQueue<CScriptCheck>::Loop (this=0x5555566be608 <scriptcheckqueue>, fMaster=false) at ./checkqueue.h:99
#3  0x00005555559b7d2c in CCheckQueue<CScriptCheck>::Thread (this=0x5555566be65c <scriptcheckqueue+84>) at ./checkqueue.h:136
#4  0x000055555599ca1d in ThreadScriptCheck () at validation.cpp:1680
#5  0x000055555575a99b in boost::detail::thread_data<void (*)()>::run (this=0x5555566be65c <scriptcheckqueue+84>) at /usr/include/boost/thread/detail/thread.hpp:116
#6  0x00007ffff77a6116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
#7  0x00007ffff5c02494 in start_thread (arg=0x7fffcd7ac700) at pthread_create.c:333
#8  0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 7 (Thread 0x7fffcdffd700 (LWP 12421)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055555573ffdb in boost::condition_variable::wait (this=0x5555566be630 <scriptcheckqueue+40>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
#2  0x00005555559dcbe7 in CCheckQueue<CScriptCheck>::Loop (this=0x5555566be608 <scriptcheckqueue>, fMaster=false) at ./checkqueue.h:99
#3  0x00005555559b7d2c in CCheckQueue<CScriptCheck>::Thread (this=0x5555566be65c <scriptcheckqueue+84>) at ./checkqueue.h:136
#4  0x000055555599ca1d in ThreadScriptCheck () at validation.cpp:1680
#5  0x000055555575a99b in boost::detail::thread_data<void (*)()>::run (this=0x5555566be65c <scriptcheckqueue+84>) at /usr/include/boost/thread/detail/thread.hpp:116
#6  0x00007ffff77a6116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
#7  0x00007ffff5c02494 in start_thread (arg=0x7fffcdffd700) at pthread_create.c:333
#8  0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 6 (Thread 0x7fffdcdc7700 (LWP 12420)):
#0  0x00007ffff41a167d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007ffff21db9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff67f504f in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#4  0x00007ffff679e9ca in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ffff65cc0f3 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ffff65d0da8 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7  0x00007ffff5c02494 in start_thread (arg=0x7fffdcdc7700) at pthread_create.c:333
#8  0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 5 (Thread 0x7fffdd66c700 (LWP 12419)):
#0  0x00007ffff41a167d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007ffff21db9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff67f504f in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#4  0x00007ffff679e9ca in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ffff65cc0f3 in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6  0x00007ffff64ab6d5 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5DBus.so.5
#7  0x00007ffff65d0da8 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#8  0x00007ffff5c02494 in start_thread (arg=0x7fffdd66c700) at pthread_create.c:333
#9  0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 4 (Thread 0x7fffdf8b8700 (LWP 12418)):
#0  0x00007ffff41a167d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007ffff21db9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007ffff21dbd82 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007fffe2017656 in ?? () from /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
#4  0x00007ffff22033d5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007ffff5c02494 in start_thread (arg=0x7fffdf8b8700) at pthread_create.c:333
#6  0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 3 (Thread 0x7fffe0109700 (LWP 12417)):
#0  0x00007ffff41a167d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007ffff21db9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff21dbb51 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007ffff22033d5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007ffff5c02494 in start_thread (arg=0x7fffe0109700) at pthread_create.c:333
#6  0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 2 (Thread 0x7fffe9d34700 (LWP 12416)):
#0  0x00007ffff41a167d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007fffefe58150 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#2  0x00007fffefe59ee9 in xcb_wait_for_event () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#3  0x00007fffec495b69 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5XcbQpa.so.5
#4  0x00007ffff65d0da8 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#5  0x00007ffff5c02494 in start_thread (arg=0x7fffe9d34700) at pthread_create.c:333
#6  0x00007ffff41aaacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7ffff7e47dc0 (LWP 12412)):
#0  __GI___libc_malloc (bytes=40) at malloc.c:2928
#1  0x00007ffff4d75c88 in CRYPTO_malloc () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#2  0x00007ffff4e4597a in ASN1_OBJECT_new () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#3  0x00007ffff4e45b00 in c2i_ASN1_OBJECT () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#4  0x00007ffff4e53820 in asn1_ex_c2i () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#5  0x00007ffff4e53d33 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#6  0x00007ffff4e54c27 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#7  0x00007ffff4e54f5e in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#8  0x00007ffff4e551cd in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#9  0x00007ffff4e5497e in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#10 0x00007ffff4e54e20 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#11 0x00007ffff4e551cd in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#12 0x00007ffff4e543bf in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#13 0x00007ffff4e54e20 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#14 0x00007ffff4e551cd in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#15 0x00007ffff4e543bf in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#16 0x00007ffff4e55355 in ASN1_item_ex_d2i () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#17 0x00007ffff4e4d7fc in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#18 0x00007ffff4e542e0 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#19 0x00007ffff4e54f5e in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#20 0x00007ffff4e551cd in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#21 0x00007ffff4e5497e in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#22 0x00007ffff4e54f5e in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#23 0x00007ffff4e551cd in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#24 0x00007ffff4e5497e in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#25 0x00007ffff4e55355 in ASN1_item_ex_d2i () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#26 0x00007ffff4e553ab in ASN1_item_d2i () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#27 0x00007ffff4e48667 in ASN1_item_dup () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
#28 0x00007ffff7f61a0f in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Network.so.5
#29 0x00007ffff7f63202 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Network.so.5
#30 0x00007ffff7f4b035 in QSslCertificate::fromData(QByteArray const&, QSsl::EncodingFormat) () from /usr/lib/x86_64-linux-gnu/libQt5Network.so.5
#31 0x00007ffff7f4cd64 in QSslCertificate::fromPath(QString const&, QSsl::EncodingFormat, QRegExp::PatternSyntax) () from /usr/lib/x86_64-linux-gnu/libQt5Network.so.5
#32 0x00007ffff7f720c3 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Network.so.5
#33 0x00007ffff7f5788d in QSslSocket::systemCaCertificates() () from /usr/lib/x86_64-linux-gnu/libQt5Network.so.5
#34 0x000055555567659a in PaymentServer::LoadRootCAs (_store=<optimized out>) at qt/paymentserver.cpp:445
#35 0x00005555555ee58c in BitcoinApplication::initializeResult (this=0x7fffffffe268, success=<optimized out>) at qt/bitcoin.cpp:468
#36 0x00005555555fcec9 in QtPrivate::FunctorCall<QtPrivate::IndexesList<0>, QtPrivate::List<bool>, void, void (BitcoinApplication::*)(bool)>::call (f=<optimized out>, o=0x15e, arg=0x7fff9e398ca0) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:141
#37 0x00005555555fce6a in QtPrivate::FunctionPointer<void (BitcoinApplication::*)(bool)>::call<QtPrivate::List<bool>, void> (f=(void (BitcoinApplication::*)(BitcoinApplication * const, bool)) 0x28, this adjustment 1, o=0x15e, arg=0x0) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:160
#38 0x00005555555fce08 in QtPrivate::QSlotObject<void (BitcoinApplication::*)(bool), QtPrivate::List<bool>, void>::impl (which=<optimized out>, this_=0x555556913dc0, r=0x15e, a=0x0, ret=0x7fffffffc99a) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobject_impl.h:120
#39 0x00007ffff67cd499 in QObject::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#40 0x00007ffff6ae4d54 in QGuiApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5
#41 0x00007ffff708992f in QApplication::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#42 0x00007ffff7083b8c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#43 0x00007ffff708b341 in QApplication::notify(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5
#44 0x00007ffff67a09e0 in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#45 0x00007ffff67a316d in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#46 0x00007ffff67f4c43 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#47 0x00007ffff21db7f7 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#48 0x00007ffff21dba60 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#49 0x00007ffff21dbb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#50 0x00007ffff67f504f in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#51 0x00007ffff679e9ca in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#52 0x00007ffff67a713c in QCoreApplication::exec() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#53 0x00005555555f085c in main (argc=<optimized out>, argv=0x7fffffffe518) at qt/bitcoin.cpp:714
(gdb) 
@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented Dec 19, 2018

I think the "deadlock detected" error reporting is just broken in this case and the error is spurious. The deadlock detection is something we should fix, but for now you could work around it by building without DEBUG_LOCKORDER.

What happened is that one ThreadMessageHandler thread locked ppmutexOpenSSL and then did a TRY_LOCK on cs_main. The next ThreadMessageHandler locked cs_main and then ppmutexOpenSSL. I don't know the background behind ThreadMessageHandler and ppmutexOpenSSL, but the deadlock detection seems broken because it is ignoring the fact that TRY_LOCK is different than LOCK and would avoid a deadlock.

This is a strange bug because deadlock detection code actually seems to know this is a try lock (it prints net_processing.cpp:3291 (TRY)) but it aborts and complains anyway.

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Dec 19, 2018

@ryanofsky I wasn't explicitly building with DEBUG_LOCKORDER. I used:

./configure CXX=clang++-6.0 CXXFLAGS="-fno-omit-frame-pointer" --enable-debug --with-incompatible-bdb --with-miniupnpc=no

It looks like --enable_debug is turning on the lockorder debugging. I'll comment out the line that mentions DEBUG_LOCKORDER in configure.ac and rebuild.

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Dec 21, 2018

@ryanofsky the patch here is working well for me. Today I ran bitcoin-qt without the patch 3 times, and all 3 times it crashed in the same place. I also ran bitcoin-qt around 100 times with the patch and it didn't crash at all.

I think it's pretty safe to say you've found the solution.

I don't know anything about Qt but I guess the bug was that you're asking for the table to be sorted before you've defined which column to sort by. I don't see how your patch changes that. The old code defines which column to sort by just one line too late:

    transactionView->setSortingEnabled(true);
    transactionView->sortByColumn(TransactionTableModel::Date, Qt::DescendingOrder);

I would have thought the fix would be to switch the order of those two lines, but your patch actually deletes the sortByColumn() call yet still seems to work.

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Jan 14, 2019

@dooglus or @ryanofsky Mind to submit this patch as a pull request?

dooglus added a commit to dooglus/bitcoin that referenced this issue Jan 18, 2019

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Jan 18, 2019

dooglus added a commit to dooglus/bitcoin that referenced this issue Feb 5, 2019

Fix issue bitcoin#9683 "gui, wallet: random abort (segmentation fault…
…) running master/HEAD".

Patch taken from @ryanofsky's comment
  bitcoin#9683 (comment)
and refined according to
  bitcoin#15203 (comment)

MarcoFalke added a commit to MarcoFalke/bitcoin that referenced this issue Feb 5, 2019

Merge bitcoin#15203: Fix issue bitcoin#9683 "gui, wallet: random abor…
…t (segmentation fault)

364cff1 Fix issue bitcoin#9683 "gui, wallet: random abort (segmentation fault) running master/HEAD". (Chris Moore)

Pull request description:

  Patch taken from @ryanofsky's comment bitcoin#9683 (comment).

  [MarcoFalke wrote](bitcoin#9683 (comment)):
  > Mind to submit this patch as a pull request?

  So that's what I'm doing.

  I was regularly seeing crashes on startup before applying this patch and haven't seen a single crash on startup since applying it almost a month ago.

Tree-SHA512: 3bbb2291cdf03ab7e7b5b796df68d76272491e35d473a89f4550065554c092f867659a7b8d7a1a91461ae4dc9a3b13b72541eafdbd732536463e9f3cf82300c8

@MarcoFalke MarcoFalke closed this Feb 5, 2019

HashUnlimited pushed a commit to HashUnlimited/chaincoin that referenced this issue Feb 6, 2019

Fix issue bitcoin#9683 "gui, wallet: random abort (segmentation fault…
…) running master/HEAD".

Patch taken from @ryanofsky's comment
  bitcoin#9683 (comment)
and refined according to
  bitcoin#15203 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.