Ledger networking

Ledger networking

Upcoming. This page describes the ledger-node branch. It is not part of a release, and details may change.

Part of Ledger implementation. This page covers LedgerDriver, which runs a node against a network, and the classes of the application module that carry the ledger over the node's QUIC connections. The ledger module has no socket code: the driver sees the network only through LedgerNetwork and LedgerPeer.

The driver

LedgerDriver is the only caller of a node. It is given a NodeFactory, a Serial executor, a Scheduler, a LedgerNetwork, a PeerLimiter, a clock, DriverOptions and a seed, or none for an observer. All its logic runs one task at a time on the Serial. Network threads only enqueue.

The NodeFactory builds the nodes: its create(signing) must close the block log between two builds, open the sign state only for a signing node and return a node whose signs() agrees. The daemon's factory is described on Ledger in the daemon.

Intake

The network calls onTransaction, onHeartbeat, onMessage, onStatus and onBlockRequest, each with the size of the frame, and onPeerConnected when a stream opens:

1. The Door sorts the item on the calling thread, from a snapshot the driver publishes after each run. A penalised peer is cut. A transaction or request is plausible; so is a heartbeat from a current validator, a status from a connected peer, a ballot or proposal for the next two heights from a validator, and a decision for the next height. The rest is junk.
2. QueueBudget bounds the queue in items and bytes, in total, per peer, and for junk, which gets a tenth. An item that does not fit is dropped and counted.
3. InboundLanes runs the control lane, which is never dropped, then at most 64 inbound items per run.
4. PeerLimiter takes a token per item. Block requests are served while the ledger can be read, the rest only while a node is ready.
5. Gossip and the node take the item.

flowchart LR
    Net[network thread] --> Door{Door}
    Door -->|penalised peer| Cut[dropped]
    Door -->|plausible| Budget[QueueBudget]
    Door -->|junk, a tenth of the room| Budget
    Budget -->|no room| Counted[dropped and counted]
    Budget --> Lanes[InboundLanes: control first, then 64 a run]
    Lanes --> Limiter{PeerLimiter token}
    Limiter -->|bucket empty| Refused[refused, maybe a point]
    Limiter --> Gossip[Gossip]
    Gossip --> Node[the node]
    Gossip -->|new| Relay[up to 8 other peers]

Loading

Everything left of Lanes runs on the calling thread and only decides whether an item is queued at all; the rest runs one task at a time on the Serial.

The driver also takes submit(transaction) from its own node's caller, which answers with the ledger's reason for a refusal, or empty when the transaction went in.

Limits

The PeerLimiter has a token bucket per peer and per kind, and a shared one per kind, given as capacity and refill per second. These are the defaults of Policy.DEFAULT:

Kind	Per peer	Shared
Transaction	200 / 50	2000 / 500
Heartbeat	20 / 2	500 / 20
Consensus	2000 / 500	20,000 / 5000
Status	10 / 1	200 / 20
Block request	40 / 10	200 / 100

Junk draws from one bucket of a twentieth of the shared consensus bucket, 1000 / 250 by default.

Sized for the network. The daemon uses Policy.forNetwork(validators, 50) instead, which grows with the size of the genesis validator set and keeps the defaults as a floor. An honest neighbour relays each new item once, so each rate is twice what the network produces, with a burst of four seconds:

• a heartbeat: one per validator every ten seconds, and a per-peer burst of twice the number of validators, the one limit with no floor;
• a transaction: 50 a second;
• a consensus message: a proposal, a prevote and a precommit of every validator and a decision, at no more than five heights a second;
• the shared limits are nine times the rates of the items, since a node relays to at most eight others and so hears an item in nine copies at most. A status is heard from each of the other validators.

Scores. A peer's score decays by 10 points a minute and a peer that reaches 100 is penalised for 60 seconds: dropped at the door, and its connections closed once. A refusal for the peer's own empty bucket costs 1 point under the default policy and a quarter point under the policy of the network, and one for the shared bucket costs nothing. The limiter never counts a clock set back. It protects the node and authenticates no one: a forwarder on the loopback address that hands every connection to the node as one from the same host gives each its own id, buckets and score.

Gossip

A node sends what it makes itself to every peer. What it takes in as new it passes on to at most relayFanout (8) other peers, ranked by a hash of the item and the peer id, never back to the sender. The sender is charged only for what is provably forged: 1 point for a transaction with a bad signature, 2 for a ballot from a non-validator or with a bad signature, a proposal from a non-proposer or a decision whose certificate fails. Honest nodes differ in policy and state, so nothing else costs anything.

Status and sync

A status goes to all peers every statusMillis, after a decision (at most once a second), to a peer when it connects, and after a sync that moved the ledger. SyncState keeps at most 256 claims of the form "peer X is at height H" and lets one lapse after twice the status interval. A peer that stays silent is doubted for the status interval, doubling with each failure in a row up to ten minutes; a peer that served a wrong block is doubted for ten.

sequenceDiagram
    participant A as This node, at height 40
    participant B as Peer B
    participant C as Peer C
    B->>A: status: height 90
    A->>B: BlockRequest(41, 16)
    B-->>A: BlockBatch: blocks 41 to 56
    A->>A: each block checked in full, as a decision
    A->>B: BlockRequest(57, 16), 250 ms later
    Note over A,B: no answer within 3 seconds
    A->>A: doubt B for a while
    A->>C: BlockRequest(57, 16)
    C-->>A: BlockBatch: blocks 57 to 72

Loading

Syncer runs one request at a time, to a peer chosen by: nodes this one dialled first, then the one that last delivered, then a salted hash. It starts on a tick when a claim is one block above the ledger, and on a status two or more above. A request is BlockRequest(height + 1, 16) with a timeout of 3 seconds. Each block of a batch goes through the node as a decision, so it is checked in full, one block per task. A wrong first block costs the peer 50 points and ten minutes of doubt, and an empty batch costs 5. Batches are paced 250 ms apart.

Promotion

A node with a seed starts as an observer. Promotion lets it vote when the grace period has passed and either no peer claims a height above its ledger or its ledger has reached the highest claim, or no request is outstanding. A status with another chain id costs its sender 10 points and is not counted. On promotion the observer is closed and a signing node is built and started; if that fails, the observer stays and the driver tries again with a delay that doubles to 60 seconds. A voting node is never demoted.

The waiting transactions are handed over. The transactions that wait for a block are in the ledger and not on disk, and every node the supervisor builds has a ledger of its own. A new node therefore takes over the transactions that waited in the one it replaces, before it starts and asks for work (Ledger.takeWaitingFrom), whether the replacement is a promotion or follows a failure. What the new state no longer allows is left out. Without it a transaction that nodes had taken in would be lost when its node was promoted, and would wait until its sender sent it again. A restart of the process still loses them.

Supervision and halting

NodeSupervisor holds the single node and never reuses one that threw. A failure of the node itself restarts it at once; a second within restartWindowMillis halts the driver. A peer's item that makes the node throw bans that peer and replaces the node: at once the first time, and after a delay that doubles to 60 seconds for later ones within the window.

A halt keeps the first reason, closes the node, which releases the sign-state lock, stops the timers and keeps the ledger so blocks can still be served. The reasons are a block that differs from the trusted checkpoint, two certified blocks at one height, a chain that diverged, a node that could not be built or failed twice, a factory that built a node with the wrong signing flag, an Error thrown in the node, and a failure of the driver's own machinery. ChainGuard checks certificates against the current validator set, so it does not see a fork among an earlier set.

Heartbeats. A voting node signs one every heartbeatMillis, submits it to its own ledger and sends it to all peers.

Options

DriverOptions.DEFAULT; every value must be above 0. The daemon uses these, with its --ledger-trusted checkpoint:

Option	Default
tickMillis	1000
heartbeatMillis	10,000
statusMillis	10,000
graceMillis	30,000
restartWindowMillis	10,000
requestTimeoutMillis	3000
maxQueuedInbound	10,000
maxQueuedPerPeer	16
maxQueuedBytes	64 MiB
maxQueuedBytesPerPeer	8 MiB
relayFanout	8
trusted	none

The driver starts the observer, checks the blocks already held against the checkpoint, and ticks once a second: it re-sends held messages, looks for a sync, checks promotion and, if a node is ready, signs a heartbeat and reports a status. close() ends the node, cancels the timers and completes closed(); a caller waits for it before reopening the same files.

In the application

The application module carries the ledger's traffic over the existing P2P stack. The ledger rides on the same UDP port and the same QUIC connections as the other protocols, 52883 by default.

• The protocol. With --ledger, Network.getProtocols() lists hedgehog-ledger/0.0.1, hedgehog/0.0.4 and gridspork/0.0.4, in that order, and the server and the client add the ledger pipeline. Without it the list has only the last two and the node behaves as one of the earlier release: it offers no ledger and installs no ledger pipeline. A stream carries the ledger only if its connection settled on the first: LedgerProtocol reads the answer once per connection and does not cache a connection that has not finished its handshake, and LedgerGateway.speaks is the gate. GET /version lists the protocols.
• The packets. Packet.java has LEDGER_TRANSACTION 4000, LEDGER_CONSENSUS 4010, LEDGER_HEARTBEAT 4020, LEDGER_STATUS 4030, LEDGER_BLOCK_REQUEST 4040 and LEDGER_BLOCK_BATCH 4050. A frame is the magic 0xBABE (2 bytes), the type (2) and the size (4), then the body. The decoder bounds the size by type before it buffers anything, so a heartbeat or status has an exact size. A variable body has a 4-byte length, and a request and its batch carry an 8-byte request id that pairs them. A malformed frame closes only its stream, and so does a ledger packet on a stream that did not settle on the protocol.
• The gateway. LedgerGateway is an application-scoped bean that hands what the handlers receive to the attached driver, and drops it when none is attached. A request for blocks is answered at once with an empty batch, so a peer that syncs does not wait for its timeout on a node that has no ledger running, but asks another; the node says so in its log at most once a minute. attach(driver) and detach() are called by LedgerService. It also answers speaks(stream) and announces a stream to the driver on its first word.
• The peers. NettyLedgerNetwork implements LedgerNetwork: the open streams of the topology's nodes, plus the newest inbound streams that have spoken the ledger, at most 64 and two per id, each filtered by the protocol gate. One id is one peer, and the stream this node opened wins. NettyLedgerPeer is one stream: it sends a transaction, message, heartbeat or status as its packet, queues what the stream cannot take yet (see The send queue), and pairs a block request with its batch. A peer's id is the IP address, the first 64 bits of an IPv6 address with /64, or ip:port on the loopback address, where the port is the socket's until the peer's Hello gives the port it listens on.

The send queue

A QUIC stream takes about as much as its window allows at a time, and at the start of a connection that is less than one block. A proposal or a decision with many transactions fills it, and the votes that the others wait for would be stuck behind it. Each NettyLedgerPeer therefore keeps an OutboundQueue of two lanes, and writes whenever the stream says it is writable again:

Lane	Holds	Limit	When it is full
Votes	Ballots, heartbeats and statuses	512 messages of up to 256 bytes	Drops its oldest, since a newer vote stands for an older one
Bulk	Proposals, decisions and transactions	Two of the largest messages a decoder takes	Refuses the new message and keeps what it has, since a block waits for the ones before it

The votes are always written before the bulk. Both lanes are bounded in bytes, so a peer that never reads cannot make this node queue without end, and at most 512 messages wait to be put in a queue from another thread. A message that is lost to a full lane or to that bound is counted, by NettyLedgerNetwork.droppedOutbound, and the protocol copes with a lost message as it did before: votes are sent again and a node that misses a block syncs it. What waits for a stream is dropped when the stream closes. A block request does not use the queue.

In the source

• ledger/src/main/java/org/unigrid/hedgehog/ledger/LedgerDriver.java: a node run against a network, with DriverOptions, NodeFactory and Checkpoint inside
• ledger/src/main/java/org/unigrid/hedgehog/ledger/NodeSupervisor.java: replacing a node that failed, with Promotion.java and Recency.java
• ledger/src/main/java/org/unigrid/hedgehog/ledger/Syncer.java: fetching blocks from a peer that is ahead, with SyncState.java and PeerView.java
• ledger/src/main/java/org/unigrid/hedgehog/ledger/Gossip.java: passing on what is new
• ledger/src/main/java/org/unigrid/hedgehog/ledger/PeerLimiter.java: the rate limits and penalties
• ledger/src/main/java/org/unigrid/hedgehog/ledger/Door.java: sorting what arrives, with DoorKeeper.java, InboundLanes.java and QueueBudget.java
• ledger/src/main/java/org/unigrid/hedgehog/ledger/ChainGuard.java: the checkpoint and fork checks
• application/src/main/java/org/unigrid/hedgehog/service/ledger/LedgerGateway.java: where the pipeline hands packets to the driver
• application/src/main/java/org/unigrid/hedgehog/service/ledger/NettyLedgerNetwork.java: the peers, with NettyLedgerPeer.java for one stream
• application/src/main/java/org/unigrid/hedgehog/service/ledger/OutboundQueue.java: what waits to be written to one stream, votes first
• application/src/main/java/org/unigrid/hedgehog/model/Network.java: the protocols a node offers