Ledger storage

Ledger storage

Upcoming. This page describes the ledger-node branch. It is not part of a release, and details may change.

Part of Ledger implementation. The ledger keeps two files: the block log, which holds every block, and the sign state, which holds what a validator has signed for the height it works on. The state itself is not stored.

The files of a node

The daemon keeps both files in the folder ledger of the data directory, see Running Hedgehog, in a folder of its own for each chain, named by the first 16 characters of the chain id:

File	Holds	Who writes it
ledger/<chain>/blocks.log	Every block, in order	Every node with --ledger
ledger/<chain>/sign.state	What this validator signed for the height it works on	A validator only
ledger/<chain>/sign.state.lock	The lock that keeps a second node off the sign state	A validator only
ledger/<chain>/sign.state.tmp	A save in progress; a stale one is ignored	A validator only

A sign state says what a validator signed on one chain, and a block log holds that chain, so the files of another chain must never meet them. A node that is given another genesis has another chain id, opens a folder of its own, and leaves the folders of the chains it had as they were.

The block log

A record is length (4), CRC32C of the length (4), CRC32C of the block (4) and the block itself. There is no header for the file as a whole. append writes the record, forces the channel with force(false) and only then keeps the block in memory. If a write fails, the file is cut back to where the record began, so the next block does not land behind half a record; if even that fails, the next open repairs it. range copies a window of the log for a peer.

open creates the file and its directory if they are missing, reads the file from the start and decides what each record is:

What it finds	What it does
A header or data cut short	Torn last write: drops the rest, with a warning
A length that fails its checksum, and only zeros to the end of the file	Torn last write: drops the rest, with a warning
A length that fails its checksum, and anything else after it	Refuses with CorruptLogException
A good length checksum, but a length below 0 or above the largest block, 355,322 bytes	Refuses
A block checksum that fails on the very last record	Torn last write: drops it
A block checksum that fails with more records behind it	Refuses
A record whose checksums pass but which does not decode	Refuses

A refusal leaves the file untouched, since dropping one record would drop every block behind it. The checksum of the length is what tells a torn end from damage in the middle.

The log has no lock of its own: only the sign state is locked, so a driver opens one node per data directory and closes it before it opens another over the same files.

Replay

The state is rebuilt from the blocks every time a Ledger is made: new Ledger(state, log) replays every block of the log into the state with all checks, so a log that was tampered with makes the constructor throw InvalidBlockException instead of coming up as a state. A replay is vouched, so it ignores what the authority says now, and it goes through the same snapshot that the genesis commits to, which is why the daemon waits for the snapshot before it builds a ledger.

A validator replays the log twice at each start: once for the observer it starts as, and again for the signing node that replaces it, see Ledger networking.

The sign state file

Part	Size
Magic UGSS	4
Version 2	1
Size of the validator set	4
Length of the payload	4
Payload
SHA-512 of everything before	64

A save writes <name>.tmp, forces it, moves it over the real file in one atomic step and forces the directory, where the file system supports that, so a crash leaves the old state or the new one. A stale .tmp is ignored.

flowchart LR
    Write[write sign.state.tmp] --> Force[force it to disk] --> Move[move it over sign.state in one step] --> Dir[force the directory]
    Write -. crash .-> Old[the old state is on disk]
    Force -. crash .-> Old
    Move -. crash .-> Either[the old or the new state]
    Dir --> New[the new state is on disk, messages may leave]

Loading

The first load or save takes an exclusive lock on <name>.lock and holds it until the file is closed, so a second node over the same path gets FileLockedException. The set size is stored because the set can change, and the largest accepted file is computed from it, for up to 4096 views.

A damaged file throws CorruptSignStateException: too short, wrong magic or version, a set size outside 1 to 1000, a length or checksum that does not match, or a payload that does not decode. What the node does with the state it loads is on Ledger consensus engine.

The protection depends on the file surviving: a sign state that is deleted or replaced by an older copy is not noticed, and the directory is not flushed on Windows or an in-memory file system.

In the source

• ledger/src/main/java/org/unigrid/hedgehog/ledger/BlockLog.java: the file format and its recovery
• ledger/src/main/java/org/unigrid/hedgehog/ledger/SignStateFile.java: the file of what was signed
• ledger/src/main/java/org/unigrid/hedgehog/ledger/Ledger.java: the replay in its constructor
• application/src/main/java/org/unigrid/hedgehog/service/ledger/LedgerNodes.java: where the daemon puts the files