How to produce snapshots

[abailly-iohk]

As the actual value of Mithril lies in the capability it gives to cardano-node users to boot their node faster, in minutes instead of hours, we want to understand how to produce reliable and reproducible snapshots from a node's DB.

Tasks to do:

• Determine information needed to create a valid snapshot

• the whole immutable folder of the database (required)
• the protocolMagicId file (required)
• the latest ledger state snapshot file in the ledger folder (optional)

• Verify that the snapshotted files are platform independent

Tests successfully ran on the first 20 chunks of the immutable folders (macOS, Ubuntu, Ubuntu on Docker, Windows on 3 separate computers) 🟢

• Determine best option to create a snapshot (difficulty to produce vs size vs time to restore)

Best option is Immutable + Ledger State, but this implies to modify the Cardano Node

Mainnet

Data	Node	Full	Archive	Snapshot	Upload	Download	Restore	Startup
Immutable Only	standard	43GB	24GB	~28m	~45m	~25m	~12m	~420m
With Ledger State	modified	45GB	25GB	~28m	~45m	~25m	~12m	~65m

Testnet

Data	Node	Full	Archive	Snapshot	Upload	Download	Restore	Startup
Immutable Only	standard	9.5 GB	3.5 GB	~7m	~5m	~3m	~2m	~130m
With Ledger State	modified	10 GB	3.5 GB	~7m	~5m	~3m	~2m	~6m

Host: x86 / +2 cores / +8GB RAM / +100GB HDD
Network: Download 150Mbps / Upload 75Mbps
Compression: gzip
Cardano Node: not running during snapshotting

• Modify the Cardano Node such that it can produce deterministic snapshots

Commit: abailly-iohk/ouroboros-network@ae552cc
This task will be finalized in a separate issue: #100

• Create a light CLI/shell script that can produce a snapshot and a deterministic digest from it

Question: Do we need to stop the node when the snapshot is done?
Answer: We can keep it running
Commit: 8fcbce9