blockchain: Convert to full block index in mem.

[davecgh]

This reworks the block index code such that it loads all of the headers in the main chain at startup and constructs the full block index accordingly.

Since the full index from the current best tip all the way back to the genesis block is now guaranteed to be in memory, this also removes all code related to dynamically loading the nodes and updates some of the logic to take advantage of the fact traversing the block index can no longer potentially fail. There are also many more optimizations and simplifications that can be made in the future as a result of this.

Due to removing all of the extra overhead of tracking the dynamic state, and ensuring the block node structs are aligned to eliminate extra padding, the end result of a fully populated block index now takes quite a bit less memory than the previous dynamically loaded version.

It also speeds up the initial startup process by roughly 2x since it is faster to bulk load the nodes in order as opposed to dynamically loading only the nodes near the tip in backwards order.

For example, here is some startup timing information before and after this commit on a node that contains roughly 238,000 blocks:

7200 RPM HDD:

Startup time before this commit: ~7.71s
Startup time after this commit: ~3.47s

SSD:

Startup time before this commit: ~6.34s
Startup time after this commit: ~3.51s

Some additional benefits are:

• Since every block node is in memory, the code which reconstructs headers from block nodes means that all headers can always be served from memory which will be important since the network will be moving to header-based semantics
• Several of the error paths can be removed since they are no longer necessary
• It is no longer expensive to calculate CSV sequence locks or median times of blocks way in the past
• It is much less expensive to calculate the initial states for the various intervals such as the stake and voter version
• It will be possible to create much more efficient iteration and simplified views of the overall index

An overview of the logic changes are as follows:

• Move AncestorNode from blockIndex to blockNode and greatly simplify since it no longer has to deal with the possibility of dynamically loading nodes and related failures
• Replace nodeAtHeightFromTopNode from BlockChain with RelativeAncestor on blockNode and define it in terms of AncestorNode
• Move CalcPastMedianTime from blockIndex to blockNode and remove no longer necessary test for nil
• Remove findNode and replace all of its uses with direct queries of the block index
• Remove blockExists and replace all of its uses with direct queries of the block index
• Remove all functions and fields related to dynamically loading nodes
  • children and parentHash fields from blockNode
  • depNodes from blockIndex
  • loadBlockNode from blockIndex
  • PrevNodeFromBlock from blockIndex
  • {p,P}revNodeFromNode from blockIndex
  • RemoveNode
• Replace all instances of iterating backwards through nodes to directly access the parent now that nodes don't potentially need to be dynamically loaded
• Introduce a lookupNode function on blockIndex which allows the initialization code to locklessly query the index
• No longer take the chain lock when only access to the block index, which has its own lock, is needed
• Remove the error paths from several functions that can no longer fail
  • getReorganizeNodes
  • findPrevTestNetDifficulty
  • sumPurchasedTickets
  • findStakeVersionPriorNode
• Remove all error paths related to node iteration that can no longer fail
• Modify FetchUtxoView to return an empty view for the genesis block

This is work towards #1145.

[davecgh]

Rebased for latest changes to ProcessBlock.

[dnldd]

Tested this PR yesterday on testnet. Re-synced to the current block as expected, didn't encounter any errors. I also verified the expected tip via https://testnet.dcrdata.org/

[AlanBarber]

Question: how does this affect memory usage? The descriptions mentions several times about having data always in memory... Faster load times are nice but I hope it's not at the expense of too much memory usage as I'd question how salable that becomes as the blockchain grows over time.

[davecgh]

From the PR description:

Due to removing all of the extra overhead of tracking the dynamic state, and ensuring the block node structs are aligned to eliminate extra padding, the end result of a fully populated block index now takes quite a bit less memory than the previous dynamically loaded version.

In terms of concrete usage, I've been running a node with it since slightly before I created the PR and it is using roughly 330MB.

[jrick]

After running this PR for a couple days:

Mainnet:

   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
  6195 decred    387M  371M sleep   59    0   1:10:30 0.6% dcrd/25

Testnet2:

   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
  6210 decred    405M  392M sleep    1    0   0:48:01 0.1% dcrd/25

[davecgh]

As a point of comparison, a couple of long running nodes on mainnet with the current latest release (v1.2.0) show:

v1.2.0 mainnet
--------------
node a: 376.3M (812.6M virtual) 
node b: 366.5M (766.3M virtual)

Running with this PR, for obviously a much shorter period (~3 days), on a couple of other nodes (no incoming connections on these nodes):

This PR mainnet
---------------
node c: 300.9M (329.7M virtual)
node d: 292.4M (305.7M virtual)

[matheusd]

Both on testnet, running the compiled binary, both after a few hours running (ps aux):

Before PR:
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT   TIME COMMAND
user     11305  2.8  7.7 1152920 392816 pts/0  Sl+    2:35 dcrd 

After PR:
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT  TIME COMMAND
user     26318  1.9  6.7 1133820 357500 pts/2  Sl+   5:13 dcrd 

[dajohi]

t'ok testnet2 miner

[jrick]

great work!