"Activating best chain" can take very long

[laanwj]

This startup phase can take extremely long, especially for nodes catching up.

Scenario:

a) User starts client after say, a week, client downloads the rest of the chain in the background
b) User shuts down client before validation completes
c) User restarts client

Now, the user will have to wait for all blocks that were previously downloaded to be validated.
This can take a long time, and meanwhile the user cannot do anything. Also the application cannot be interrupted (with e.g. signals) in this start-up phase.

Would be preferable to move this to after AppInit2, in the background, if possible.

(see also #7001, #6459)

[MrChrisJ]

I am running Raspberry Pi 2 Model B running Bitcoin Core 0.12 with Raspbian Jessie Lite. I am getting the "Activating Best Chain" error after each time I restart Bitcoind the same way you describe above. It is only an issue if blockchain is not synced to the latest block height before closing down. It can take several hours and up to a day sometimes for Bitcoin to come online on the Arm 7.

At the moment the bitcoin data is stored on a USB Stick but I can also try testing it on an SSD to see if the wait times are due to the drive being used.

My motivation for raising the issue is that as these smaller lower cost devices become more popular as full nodes used by hobbyists it might be good if I could dedicate some time to maximising Bitcoin's performance specifically for these types of devices.

[laanwj]

Performance isn't the issue I'm raising here - the blocks have to be checked, for better of worse. This has already become lots faster in 0.12 by using the secp256k1 library for validation. I've also spent some time optimizing some of the operations in assembly for ARM a while ago, but this isn't merged yet:

• ARM assembly implementation of field_10x26 inner bitcoin-core/secp256k1#173
• ARM assembly implementation of field_10x26 inner (rebase of #173) bitcoin-core/secp256k1#366

But what I mean is that the order of operations at startup could be better. I don't think there is a strong reason it has to finish checking all those blocks before letting the user do anything else. Normally this is done in the background.

Offtopic: Storing bitcoin data, especially the databases on an USB stick is not a very good idea. Not only is it slow, the access pattern tends to kill them off quickly.

[dthorpe]

Your block verfication/ catch up rate is almost certainly being throttled by low I/O to the USB stick. When I moved my bitcoin data from HD to SSD, block verification / catch up rate went from 30-90 seconds per block to ~5 seconds per block. All other aspects of the system were identical - only the location of the data changed.

Prior to the update to SSD, I noticed that the bitcoin core disk I/O is "clumpy" during block verification. A burst of disk I/O, followed by a burst of CPU load, followed by the verified block count incrementing in the UI. I would imagine the disk I/O burst is to load the UTXOs referenced/spent by the block being checked, and the CPU load was to actually verify the signatures and the block hash.

Bitcoin Core doesn't require high computing power to keep up with the Bitcoin network, It does require high-speed storage I/O.

[MrChrisJ]

Thanks very much. Yes I understood your intention with the issue (I am kind of new to this so please forgive me) I just raised my report in case you had planned to fix it in 0.12 and hadn't realised. I don't notice the bug at all on my more powerful machines.

Briefly on the off topic: I find the SD Cards are a good tradeoff if the user is just looking for low cost device. The MTBF of Sandisk Cruzers is very good and have been running one for a year now. SDD is much faster but still find I have to overclock the processor to make it more usable for development.

Thanks for the fast response.

[pstratem]

@laanwj a number of people have commenting on this to me in person.

the initialization stuff definitely should not be calling activate best chain

[tommcintyre]

@dthorpe @MrChrisJ

Offtopic: Storing bitcoin data, especially the databases on an USB stick is not a very good idea. Not only is it slow, the access pattern tends to kill them off quickly.

(Untested yet, but just an educated hypothesis - VFAT [1] is not a great choice for flash memory (USB sticks, SSD); might a log-structured FS like UBIFS (or possibly YAFFS2) be a better choice?)

[1] As proposed by each of the Fullnode setup guides linked here.

[laanwj]

After #7821 it's possible to shut down the client during 'Activating best chain', this improves the situation somewhat.

[sipa]

This should also be improved by #7917, which moves the "Activating best chain" step to ThreadImport.

[maflcko]

Activating Best Chain is no longer done during the splash screen. Anything left to do here?