Add ability to generate dumps of node, slpdb and fulcrum

[sajal]

Currently users of bch-toolkit need to wait long times (often several days) to fully sync node+slpdb to mainnet. We should provide dumps so that the bootstrap process can be made faster.

[sajal]

Dump URL/data format (WIP)

Backup file : <prefix>/[bchn|bu|fulcrum|slpdb]/<blockheight>/<fname>
Manifest : <prefix>/[bchn|bu|fulcrum|slpdb]/<blockheight>/manifest.json
Signature : <prefix>/[bchn|bu|fulcrum|slpdb]/<blockheight>/sig.json

Example:

Lets say our storage root (prefix) is https://example.com/devkit-dumps/ , and we have a dump of bitcoin unlimited at block 1,500,000

• This dump would be available at https://example.com/devkit-dumps/bchu/1500000/bu.tar.gz
• Manifest file at https://example.com/devkit-dumps/bchu/1500000/manifest.json
• Signature file at https://example.com/devkit-dumps/bchu/1500000/sig.json

Dump format : Depends on the application, the devkit needs to know how to deal with it

manifest.json proposed format

{
    "height": 1500000, //Last known block included in the dump
    "filename": "bu.tar.gz", //Filename of the actual dump
    "sha256sum" : "c98c24b677eff44860afea6f493bbaec5bb1c4cbb209c6fc2bbb47f66ff2ad31", //sha256sum of bu.tar.gz
    "version": "1.9.2", //Version used to make this dump. The users version must be same or newer 
    "devkit_version": "1.2.3", //The version of devkit this dump supports. Restore script would take this into account to determine how to process this dump
    "time" : "2021-12-20T11:38:39.730026" //Timestamp when this dump was created
}

sig.json proposed format

{
    "signer": "qp3wjpa3tjlj042z2wv7hahsldgwhwy0rq9sywjpyy", //The key used to sign the manifest, the user needs to trust this signer.
    "signature": "IBOvIfsAs/da1e36W8kw1cQOPqPVXCW5zJgNQ5kI8m57FycZXdeFmeyoIqJSREzE4W7vfDmdmPk0HokuJPvgPPE=" //Entire manifest.json signed by signer
}

User flow when restoring (lets say they want to do bitcoin unlimited)

1. Do directory listing of https://example.com/devkit-dumps/bchu/
2. Find directory name with highest block number
3. Fetch manifest.json and sig.json
4. Validate that signer is someone user trusts, and check if manifest.json is signed correctly.
5. Check if version is compatible. If not find an alternate block number and goto step 3.
6. Download the file referred in filename.
7. Validate sha256sum of downloaded file is same as mentioned in the manifest
8. Restore Bitcoin Unlimited node, wait for it to startup and stabilize (initialblockdownload turns to false)

Note: We either need directory listing enabled, or some application server that processes some logic to point the user to the correct directory.

[sajal]

Updated work-in-progress example for a smartbch restore

0. Ensure BCH node is synced up
1. Fetch index https://devkit-dumps.actorforth.org/mainnet/smartbch/
2. Observe newest directory is 2808567 . Directory names are approximate last-known block number at the time of making the dump.
3. Fetch manifest from https://devkit-dumps.actorforth.org/mainnet/smartbch/2808567/manifest.json
4. check if constraints are satisfied, i.e. smartbch >= 0.3.5 and dump made for devsuite 1.1.0 is compatible with currently running devsuite or not. If not then retry from step 2 using an older dump.
   5. fetch signature file https://devkit-dumps.actorforth.org/mainnet/smartbch/2808567/sig.json (currently its a placeholder)
   6. Compare if signature matches(check previous comment) and the signer is trusted.
5. Check if /tmp/smartbch.tar.gz already exists and sha256sum matches manifest. if matches GOTO 10
6. ^ If not then download https://devkit-dumps.actorforth.org/mainnet/smartbch/2808567/smartbch.tar.gz . Its better to use a download accelerator (like axel) instead of wget, axel allows for multi-connection downloads, and allows download from several mirrors at once.
7. check sha256sum matches, if not its an error
8. proceed to untaring it as mentioned in Dump and restore smartbch #50

[sajal]

Restore step for slpdb is similar.

Find the file https://devkit-dumps.actorforth.org/mainnet/slpdb/710040/slpdb.mongo.gz

then run

mongorestore  --drop  --authenticationDatabase admin --archive=/tmp/slpdb.mongo.gz --gzip   mongodb://mongouser:mongopass@mongodb-slpdb:27017

This needs to run somewhere (ideally in a container) that has access to the dump file and can connect to mongodb