Introduce auxiliary block requests

[jonasschnelli]

This PR will allow auxiliary block requests. This will be required to run SPV wallets as well as this solves some other interesting use cases.

CAuxiliaryBlockRequest

• A new "lock-free" class that handles the auxiliary blocks-download/processing
• Contains a single global CAuxiliaryBlockRequest pointer (switching to a queue would be trivial).

SyncTransaction signal

• A new parameter will be added that defines if the transactions are coming from a validated or non-validated block

New RPC call "requestblocks"

Allows to request blocks, if available on disk, they will be optionally passed tough the SyncTransaction signal (Wallet/ZMQ). If some or all of the blocks are not available, they will be downloaded (prioritized over "normal" IBD downloads).
The blocks will be downloaded in parallel using the current block download mechanisms.
CAuxiliaryBlockRequest will ensure that the blocks get processes in the correct order (as requested).

If someone has a good testplan for this, please stand up.

[jonasschnelli]

Renamed "Out-of-band Block Requests" to "Auxiliary Block Request" to avoid confusion.

[jonasschnelli]

Did the non-trivial post main.cpp split rebase.
Travis issue is related to #9364.

Short explanation how this works.

• This PR allows someone to request blocks during IBD (or catch-up). These requested blocks can be newer the the current chaintip (validated blocks tip).

  • Example:
    requestblocks start '["0000000000000000000a394efd567b68581485fa91bbec42b24a4268fda73a42"]'

• If one requests blocks with a auxiliary block request (RPC requestblocks), bitcoin-core checks if the blocks are already available on disk and eventually download the missing ones from other peers.

• Downloads of auxiliary block requests are prioritized over "normal IBD block downloads".

• Optionally, requestblocks will stuff the requested blocks through the SyncTransaction signal (example. ZMQ).

• CAuxiliaryBlockRequest ensures that the blocks get passed in the right sequence through the signal (download can be asynchronous, but signal passing is in order [example: download can be D-F-E-G-B-C-A, but signal passing is always A-B-C-D-E-F-G])

• You can check the status of the auxiliary request (if all blocks has been downloaded) by requestblock status
  ** During download, it can look like:

{
  "request_present": 1,
  "created": 1481888029,
  "is_cancled": false,
  "requested_blocks": 80,
  "loaded_blocks": 65,
  "processed_blocks": 11
}

• Once the download/processing has been completed, you can get details from the block with things like getblock 0000000000000000000a394efd567b68581485fa91bbec42b24a4268fda73a4.

Use cases

• This could serve as the core infrastructure for the fullblock hybrid SPV mode.
• One could process (only header/PoW checked) blocks with this PR (attach an SPV wallet over ZMQ as example)

[jonasschnelli]

I forgot to mention that this could – in general – be useful for pruned nodes.

[jonasschnelli]

Added a hidden RPC command (setautorequestblocks) as well as a debug startup configuration (-autorequestblocks) to stop requesting blocks (IBD / catch-up).

This allows regression-tests of the auxiliary block requests.

Added a simple RPC test.

[rebroad]

I'm not really understanding why this PR is needed or what new functionality (that was needed) it does... sorry.

[jonasschnelli]

Force push fixed a rebase error.

[jtimon]

Concept ACK

[jtimon]

Needs rebase

[ryanofsky]

Maybe this PR should be closed? #9483 is newer and includes these changes. Also I posted a big batch of review comments there in February.

[jonasschnelli]

Closing in favour of #10794