Segfault on startup at a04440

[hkjn]

With a binary built at master @ a04440, bitcoind crashes with segfault on startup on one of my nodes.

Expected: No segfault.

Actual: Segfault, with no more info in debug.log, even with -debug set (see below).

Crash seems reproducible, but only on this node. I'm recompiling with --enable-debug to see if that gives more info.

Self-compiled at a04440. Switching to the version built at 6324c6 that the node was running earlier starts up fine.

Linux, armv7l, SSD disks.

Debug logs:

$ bitcoind -debug
2018-04-11T11:57:14Z 


















2018-04-11T11:57:14Z Bitcoin Core version v0.16.99.0-a04440feb (release build)
2018-04-11T11:57:14Z InitParameterInteraction: parameter interaction: -blocksonly=1 -> setting -whitelistrelay=0                                                                        
2018-04-11T11:57:14Z Assuming ancestors of block 0000000000000000005214481d2d96f898e3d5416e43359c145944a909d242e0 have valid signatures.                                                
2018-04-11T11:57:14Z Setting nMinimumChainWork=000000000000000000000000000000000000000000f91c579d57cad4bc5278cc                                                                         
2018-04-11T11:57:14Z ParameterInteraction: parameter interaction: -blocksonly=1 -> setting -walletbroadcast=0                                                                           
2018-04-11T11:57:14Z Using the 'standard' SHA256 implementation
2018-04-11T11:57:14Z Default data directory /home/bitcoin/.bitcoin                          
2018-04-11T11:57:14Z Using data directory /home/bitcoin/.bitcoin                            
2018-04-11T11:57:14Z Using config file /home/bitcoin/.bitcoin/bitcoin.conf                  
2018-04-11T11:57:14Z Using at most 16 automatic connections (1048576 file descriptors available)                                                                                        
2018-04-11T11:57:14Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements                                                                              
2018-04-11T11:57:14Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements                                                                       
2018-04-11T11:57:14Z Using 4 threads for script verification                                
2018-04-11T11:57:14Z scheduler thread start
2018-04-11T11:57:14Z Allowing HTTP connections from: 127.0.0.0/8 ::1/128 163.172.184.153/32 10.4.2.0/24                                                                                 
2018-04-11T11:57:14Z Binding RPC on address :: port 8332
2018-04-11T11:57:14Z Binding RPC on address 0.0.0.0 port 8332                               
2018-04-11T11:57:14Z Binding RPC on address 0.0.0.0 port 8332 failed.                       
2018-04-11T11:57:14Z Initialized HTTP server
2018-04-11T11:57:14Z HTTP: creating work queue of depth 16
2018-04-11T11:57:14Z Starting RPC
2018-04-11T11:57:14Z Starting HTTP RPC server
2018-04-11T11:57:14Z No rpcpassword set - using random cookie authentication
2018-04-11T11:57:14Z Generated RPC authentication cookie /home/bitcoin/.bitcoin/.cookie     
2018-04-11T11:57:14Z Registering HTTP handler for / (exactmatch 1)
2018-04-11T11:57:14Z Registering HTTP handler for /wallet/ (exactmatch 0)                   
2018-04-11T11:57:14Z Starting HTTP server
2018-04-11T11:57:14Z HTTP: starting 4 worker threads                                        
2018-04-11T11:57:14Z Using wallet directory /home/bitcoin/.bitcoin                          
2018-04-11T11:57:14Z init message: Verifying wallet(s)...
2018-04-11T11:57:14Z Entering http event loop
2018-04-11T11:57:14Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)          
2018-04-11T11:57:14Z Using wallet wallet.dat
2018-04-11T11:57:14Z CDBEnv::Open: LogDir=/home/bitcoin/.bitcoin/database ErrorFile=/home/bitcoin/.bitcoin/db.log                                                                       
2018-04-11T11:57:14Z net: setting try another outbound peer=false
2018-04-11T11:57:14Z Cache configuration:
2018-04-11T11:57:14Z * Using 12.5MiB for block index database
2018-04-11T11:57:14Z * Using 8.0MiB for chain state database
2018-04-11T11:57:14Z * Using 79.5MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space)
2018-04-11T11:57:14Z init message: Loading block index...
2018-04-11T11:57:14Z LevelDB using max_open_files=64 (default=1000)
2018-04-11T11:57:14Z Opening LevelDB in /home/bitcoin/.bitcoin/blocks/index
2018-04-11T11:57:14Z leveldb: Recovering log #109195
2018-04-11T11:57:14Z leveldb: Level-0 table #109210: started
2018-04-11T11:57:14Z leveldb: Level-0 table #109210: 179 bytes OK
2018-04-11T11:57:14Z leveldb: Delete type=0 #109195
2018-04-11T11:57:14Z leveldb: Delete type=3 #109193
2018-04-11T11:57:14Z Opened LevelDB successfully
2018-04-11T11:57:14Z Using obfuscation key for /home/bitcoin/.bitcoin/blocks/index: 0000000000000000
2018-04-11T11:57:37Z LoadBlockIndexDB: last block file = 1230
2018-04-11T11:57:37Z leveldb: Compacting 1@3 + 13@4 files
2018-04-11T11:57:37Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=196, size=119299357, heights=517496...517691, time=2018-04-10...2018-04-11)
2018-04-11T11:57:37Z Checking all blk files are present...
2018-04-11T11:57:37Z leveldb: Generated table #109212@3: 14790 keys, 2141156 bytes
2018-04-11T11:57:37Z leveldb: Generated table #109213@3: 494 keys, 71674 bytes
2018-04-11T11:57:38Z leveldb: Generated table #109214@3: 14790 keys, 2141044 bytes
2018-04-11T11:57:38Z LoadBlockIndexDB: transaction index enabled
2018-04-11T11:57:38Z LevelDB using max_open_files=64 (default=1000)
2018-04-11T11:57:38Z Opening LevelDB in /home/bitcoin/.bitcoin/chainstate
2018-04-11T11:57:38Z leveldb: Generated table #109215@3: 313 keys, 45397 bytes
2018-04-11T11:57:38Z leveldb: Recovering log #118330
2018-04-11T11:57:38Z leveldb: Level-0 table #118367: started
2018-04-11T11:57:38Z leveldb: Level-0 table #118367: 308 bytes OK
2018-04-11T11:57:38Z leveldb: Delete type=3 #118328
2018-04-11T11:57:38Z leveldb: Delete type=2 #118175
2018-04-11T11:57:38Z leveldb: Delete type=2 #118069
2018-04-11T11:57:38Z leveldb: Delete type=2 #118062
2018-04-11T11:57:38Z leveldb: Delete type=2 #118068
2018-04-11T11:57:38Z leveldb: Delete type=2 #118061
2018-04-11T11:57:38Z leveldb: Delete type=2 #118066
2018-04-11T11:57:38Z leveldb: Delete type=2 #118064
2018-04-11T11:57:38Z leveldb: Delete type=0 #118330
2018-04-11T11:57:38Z leveldb: Delete type=2 #118032
2018-04-11T11:57:38Z leveldb: Delete type=2 #118063
2018-04-11T11:57:38Z leveldb: Delete type=2 #118060
2018-04-11T11:57:38Z leveldb: Delete type=2 #118065
2018-04-11T11:57:38Z leveldb: Delete type=2 #118368
2018-04-11T11:57:38Z leveldb: Delete type=2 #118366
2018-04-11T11:57:38Z leveldb: Delete type=2 #118067
2018-04-11T11:57:38Z Opened LevelDB successfully
2018-04-11T11:57:38Z Using obfuscation key for /home/bitcoin/.bitcoin/chainstate: 72e7c68d386988b3
2018-04-11T11:57:38Z leveldb: Generated table #109216@3: 42165 keys, 2173766 bytes
2018-04-11T11:57:38Z Loaded best chain: hashBestChain=0000000000000000003f6dfa302cd20c2721555a8004ef8c84864f64e68df67c height=517691 date=2018-04-11T11:45:00Z progress=0.999991
2018-04-11T11:57:38Z init message: Rewinding blocks...
2018-04-11T11:57:38Z leveldb: Generated table #109217@3: 42164 keys, 2173669 bytes
2018-04-11T11:57:39Z leveldb: Generated table #109218@3: 15988 keys, 823981 bytes
2018-04-11T11:57:39Z leveldb: Generated table #109219@3: 42164 keys, 2173673 bytes
2018-04-11T11:57:39Z leveldb: Generated table #109220@3: 42164 keys, 2173039 bytes
2018-04-11T11:57:40Z leveldb: Generated table #109221@3: 42164 keys, 2173419 bytes
2018-04-11T11:57:40Z leveldb: Generated table #109222@3: 26480 keys, 1364917 bytes
2018-04-11T11:57:40Z leveldb: Generated table #109223@3: 42164 keys, 2173787 bytes
2018-04-11T11:57:41Z leveldb: Generated table #109224@3: 37474 keys, 1931396 bytes
2018-04-11T11:57:41Z leveldb: Compacted 1@3 + 13@4 files => 21560918 bytes
2018-04-11T11:57:41Z leveldb: compacted to: files[ 3 7 62 725 5024 2064 0 ]
2018-04-11T11:57:41Z leveldb: Delete type=2 #109137
2018-04-11T11:57:41Z leveldb: Delete type=2 #109145
2018-04-11T11:57:41Z leveldb: Delete type=2 #109144
2018-04-11T11:57:41Z leveldb: Delete type=2 #109139
2018-04-11T11:57:41Z leveldb: Delete type=2 #109142
2018-04-11T11:57:41Z leveldb: Delete type=2 #109143
2018-04-11T11:57:41Z leveldb: Delete type=2 #109134
2018-04-11T11:57:41Z leveldb: Delete type=2 #109146
2018-04-11T11:57:41Z leveldb: Delete type=2 #109136
2018-04-11T11:57:41Z leveldb: Delete type=2 #109141
2018-04-11T11:57:41Z leveldb: Delete type=2 #109135
2018-04-11T11:57:41Z leveldb: Delete type=2 #109140
2018-04-11T11:57:41Z leveldb: Delete type=2 #108990
2018-04-11T11:57:41Z leveldb: Delete type=2 #109196
2018-04-11T11:57:45Z WriteBatch memory usage: db=index, before=0.1MiB, after=0.1MiB
2018-04-11T11:57:45Z Writing final batch of 0.00 MiB
2018-04-11T11:57:45Z WriteBatch memory usage: db=chainstate, before=0.0MiB, after=0.0MiB
2018-04-11T11:57:45Z Committed 0 changed transaction outputs (out of 0) to coin database...
2018-04-11T11:57:45Z init message: Verifying blocks...
2018-04-11T11:57:45Z Verifying last 6 blocks at level 3
2018-04-11T11:57:45Z [0%]...leveldb: Moved #118367 to level-1 308 bytes OK: files[ 0 1 24 362 1179 0 0 ]
2018-04-11T11:57:45Z leveldb: Compacting 1@1 + 24@2 files
2018-04-11T11:57:45Z leveldb: Generated table #118369@1: 1 keys, 207 bytes
2018-04-11T11:57:45Z leveldb: Generated table #118370@1: 4217 keys, 210255 bytes
2018-04-11T11:57:45Z leveldb: Generated table #118371@1: 4078 keys, 200014 bytes
2018-04-11T11:57:45Z leveldb: Generated table #118372@1: 4087 keys, 203922 bytes
2018-04-11T11:57:45Z leveldb: Generated table #118373@1: 4531 keys, 216005 bytes
2018-04-11T11:57:45Z [16%]...leveldb: Generated table #118374@1: 4119 keys, 199759 bytes
2018-04-11T11:57:45Z leveldb: Generated table #118375@1: 4318 keys, 216798 bytes
2018-04-11T11:57:45Z leveldb: Generated table #118376@1: 98 keys, 4814 bytes
2018-04-11T11:57:45Z leveldb: Generated table #118377@1: 4387 keys, 209206 bytes
2018-04-11T11:57:45Z leveldb: Generated table #118378@1: 4345 keys, 211747 bytes
2018-04-11T11:57:45Z leveldb: Generated table #118379@1: 4036 keys, 200497 bytes
2018-04-11T11:57:45Z leveldb: Generated table #118380@1: 4146 keys, 202248 bytes
2018-04-11T11:57:45Z leveldb: Generated table #118381@1: 4196 keys, 198305 bytes
2018-04-11T11:57:45Z leveldb: Generated table #118382@1: 3832 keys, 190471 bytes
2018-04-11T11:57:46Z leveldb: Generated table #118383@1: 4051 keys, 201361 bytes
2018-04-11T11:57:46Z leveldb: Generated table #118384@1: 4464 keys, 221466 bytes
2018-04-11T11:57:46Z leveldb: Generated table #118385@1: 4096 keys, 198885 bytes
2018-04-11T11:57:46Z leveldb: Generated table #118386@1: 4382 keys, 217587 bytes
2018-04-11T11:57:46Z leveldb: Generated table #118387@1: 4917 keys, 229917 bytes
2018-04-11T11:57:46Z leveldb: Generated table #118388@1: 4300 keys, 211286 bytes
2018-04-11T11:57:46Z leveldb: Generated table #118389@1: 4003 keys, 196340 bytes
2018-04-11T11:57:46Z leveldb: Generated table #118390@1: 4157 keys, 206832 bytes
2018-04-11T11:57:46Z leveldb: Generated table #118391@1: 4596 keys, 215723 bytes
2018-04-11T11:57:46Z leveldb: Generated table #118392@1: 1466 keys, 71785 bytes
2018-04-11T11:57:46Z leveldb: Compacted 1@1 + 24@2 files => 4435430 bytes
2018-04-11T11:57:46Z leveldb: compacted to: files[ 0 0 24 362 1179 0 0 ]
2018-04-11T11:57:46Z [33%]...[50%]...[66%]...leveldb: Compacting 1@3 + 10@4 files
2018-04-11T11:57:50Z [83%]...leveldb: Generated table #118393@3: 38486 keys, 2174479 bytes
2018-04-11T11:57:51Z leveldb: Generated table #118394@3: 37469 keys, 2173916 bytes
2018-04-11T11:57:51Z leveldb: Generated table #118395@3: 38878 keys, 2175990 bytes
2018-04-11T11:57:51Z leveldb: Generated table #118396@3: 38694 keys, 2174535 bytes
2018-04-11T11:57:52Z leveldb: Generated table #118397@3: 37903 keys, 2174138 bytes
2018-04-11T11:57:52Z leveldb: Generated table #118398@3: 38108 keys, 2174691 bytes
2018-04-11T11:57:52Z leveldb: Generated table #118399@3: 38036 keys, 2173630 bytes
2018-04-11T11:57:53Z [99%]...leveldb: Generated table #118400@3: 38108 keys, 2173306 bytes
2018-04-11T11:57:53Z [DONE].
2018-04-11T11:57:53Z No coin database inconsistencies in last 7 blocks (7607 transactions)
2018-04-11T11:57:53Z  block index           38962ms
2018-04-11T11:57:53Z Reading estimates: 190 buckets counting confirms up to 48 blocks
2018-04-11T11:57:53Z Reading estimates: 190 buckets counting confirms up to 12 blocks
2018-04-11T11:57:53Z Reading estimates: 190 buckets counting confirms up to 1008 blocks
2018-04-11T11:57:53Z init message: Loading wallet...
2018-04-11T11:57:53Z leveldb: Generated table #118401@3: 38342 keys, 2173156 bytes
2018-04-11T11:57:54Z nFileVersion = 169900
2018-04-11T11:57:54Z Keys: 2001 plaintext, 0 encrypted, 2001 w/ metadata, 2001 total
2018-04-11T11:57:54Z leveldb: Generated table #118402@3: 38613 keys, 2173466 bytes
2018-04-11T11:57:54Z  wallet                  374ms
2018-04-11T11:57:54Z setKeyPool.size() = 2000
2018-04-11T11:57:54Z mapWallet.size() = 0
2018-04-11T11:57:54Z mapAddressBook.size() = 0
Segmentation fault (core dumped)

Backtrace with gdb:

2018-04-11T12:21:29Z Loaded best chain: hashBestChain=0000000000000000003f6dfa302cd20c2721555a8004ef8c84864f64e68df67c height=517691 date=2018-04-11T11:45:00Z progress=0.999975        
2018-04-11T12:21:29Z init message: Rewinding blocks...                                                                                                                                  
2018-04-11T12:21:35Z init message: Verifying blocks...                                                                                                                                  
2018-04-11T12:21:35Z Verifying last 6 blocks at level 3                                     
2018-04-11T12:21:35Z [0%]...[New Thread 0xaf12d450 (LWP 57)]                                
[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].                                     
2018-04-11T12:21:46Z No coin database inconsistencies in last 7 blocks (7607 transactions)  
2018-04-11T12:21:46Z  block index           40569ms                                                                                                                                     
2018-04-11T12:21:46Z init message: Loading wallet...                                                                                                                                    
2018-04-11T12:21:46Z nFileVersion = 169900                                                                                                                                              
2018-04-11T12:21:46Z Keys: 2001 plaintext, 0 encrypted, 2001 w/ metadata, 2001 total                                                                                                    
2018-04-11T12:21:46Z  wallet                  351ms                                                                                                                                     
2018-04-11T12:21:46Z setKeyPool.size() = 2000                                                                                                                                           
2018-04-11T12:21:46Z mapWallet.size() = 0                                                                                                                                               
2018-04-11T12:21:46Z mapAddressBook.size() = 0                                              
[New Thread 0xae92c450 (LWP 58)]                                                            
2018-04-11T12:21:46Z mapBlockIndex.size() = 517694                                          
2018-04-11T12:21:46Z nBestHeight = 517691                                                                                                 
[New Thread 0xae12b450 (LWP 59)]                                                                                                                                                        
                                             
Thread 12 "bitcoin-loadblk" received signal SIGSEGV, Segmentation fault.    
[Switching to Thread 0xae92c450 (LWP 58)]                                              
0xb6f81868 in pthread_setschedparam () from /usr/lib/libpthread.so.0
(gdb) backtrace                 
#0  0xb6f81868 in pthread_setschedparam () from /usr/lib/libpthread.so.0
#1  0x0075b29c in ScheduleBatchPriority() ()
#2  0x0047509c in ThreadImport(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >) ()
#3  0x00494f40 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >), boost::_bi::list1<boost:
:_bi::value<std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> > > > > >::run() ()
#4  0x0082aed4 in thread_proxy ()                        
#5  0xb6f7fee4 in start_thread () from /usr/lib/libpthread.so.0                   
#6  0xb6d04328 in ?? () from /usr/lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)                                       
(gdb)

[maflcko]

Please try master as of a7cbe38 or later

[hkjn]

Alright @MarcoFalke, I started a build at latest HEAD (f15b72), will update here once it finishes.

[fanquake]

@hkjn Did compiling a more recent master solve your problem? If so can you please close this issue.

[hkjn]

Sorry for the delay (the armv7 build takes many hours, and the build machine ran out of disk the first time), but I can confirm that the crash on startup no longer happens at f15b72.

Closing.