Skip to content

Commit

Permalink
Modifications to support "merkle tree" block proofs
Browse files Browse the repository at this point in the history 
 * Updated block header and items with review changes
 * Updated state changes to support add and remove for named states
 * Updated block service with review changes
 * Major rebuild of block state proof to work with both
   the "block merkle tree" concept and a  single TSS-BLS
   signature instead of 30+ individual RSA signatures.

Signed-off-by: Joseph Sinclair <joseph.sinclair@swirldslabs.com>
  • Loading branch information
@jsync-swirlds
jsync-swirlds committed Jun 18, 2024
1 parent 55e7b70 commit efdafef
Show file tree
Hide file tree
Showing 8 changed files with 280 additions and 414 deletions.
2 changes: 1 addition & 1 deletion block/block_service.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ message PublishStreamRequest {
/**
* A single item written to the block stream.
*/
com.hedera.hapi.block.stream.BlockItem block_item = 2;
com.hedera.hapi.block.stream.BlockItem block_item = 1;
}

/**
Expand Down
24 changes: 17 additions & 7 deletions block/stream/block_header.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -70,16 +70,16 @@ message BlockHeader {
uint64 number = 2;

/**
* The running hash of the previous block.
* The block proof hash for the previous block.
* <p>
* This value MUST match the block hash of the previous block in the
* block stream.<br/>
* This value MUST match the block merkle tree root hash of the previous
* block in the block stream.<br/>
* This value SHALL be empty for the genesis block, and SHALL NOT be empty
* for any other block.<br/>
* Client systems SHOULD optimistically reject any block with a
* `start_running_hash` that does not match the block hash of the previous
* block and MAY assume the block stream has encountered data loss,
* data corruption, or unauthorized modification.
* `previous_block_proof_hash` that does not match the block hash of the
* previous block and MAY assume the block stream has encountered data
* loss, data corruption, or unauthorized modification.
* <p>
* The process for computing a block hash is somewhat complex, and involves
* creating a "virtual" merkle tree to obtain the root merkle hash of
Expand Down Expand Up @@ -127,12 +127,22 @@ message BlockHeader {
* of this merkle tree.</li>
* </ul>
*/
bytes start_running_hash = 3;
bytes previous_block_proof_hash = 3;

/**
* The hash algorithm used in this block.
*/
BlockHashAlgorithm hash_algorithm = 4;

/**
* A version for the network address book.<br/>
* The address book version is needed to determine the correct public
* key(s) to use to validate block signatures and state proofs.
* <p>
* This MUST be the version of the address book that signed this
* block.
*/
proto.SemanticVersion address_book_version = 5;
}

/**
Expand Down
4 changes: 0 additions & 4 deletions block/stream/block_item.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -175,10 +175,6 @@ message BlockItem {
*/
FilteredBlockItem filtered_item = 9;
}

// REVIEWER NOTE: Do we need a "End Stream" item? How else does a block
// node know when the source *intends* to end the stream (which should
// happen regularly, perhaps every hour or every fifteen minutes)?
}

/**
Expand Down
301 changes: 89 additions & 212 deletions block/stream/block_state_proof.proto
View file
Original file line number Diff line number Diff line change
@@ -1,17 +1,6 @@
/**
* # Block State Proof
* A state proof for the block information in state at the end of an
* Hedera block.
*
* The basic process for a state proof (which provides cryptographic assurance
* that a particular value is present in the network state) is as follows.<br/>
* Compute the SHA384 hash of the value to be proven, then take that and
* the sibling hashes and compute a SHA384 hash of those values to produce
* the hash at the next level up the tree. Repeat this until you have the root
* hash of the merkle tree. The signatures on that root hash are proof that
* the hash (and hence the contents of the merkle path to the leaf value to
* be proven) is attested by sufficient consensus nodes to be considered the
* actual state of the network.
* A state proof for the block streamed from a consensus node.
*
* ### Keywords
* The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
Expand Down Expand Up @@ -40,220 +29,108 @@ package com.hedera.hapi.block.stream;
* limitations under the License.
*/

import "state/blockrecords/block_info.proto";

option java_package = "com.hedera.hapi.block.stream";
// <<<pbj.java_package = "com.hedera.hapi.block.stream">>> This comment is special code for setting PBJ Compiler java package
option java_multiple_files = true;

/**
* A state proof for the `BlockInfo` singleton in network state.
* A state proof for the "Block Merkle Tree".
*
* > Important Note
* >> This design is very much IN FLUX and should not be relied upon.
* >> Significant changes are known to be needed to support efficient
* >> computation of an effective state proof while also supporting
* >> reasonably efficient validation of that state proof in an
* >> EVM-compatible smart contract execution environment.
* >
* >> The major items of concern include inefficient computation of 512-bit
* >> values required for SHA2-384 hashes (and all SHA3 hashes), signature
* >> verification of BLS signatures using the BLS12 family of curves, and
* >> possible needs for zero-knowledge proofs, including SNARKS based on
* >> the KZG commitments already present for Ethereum and future STARKS
* >> based on quantum-resistant hash functions.
* This message SHALL offer a state proof for the "Block Merkle Tree".
* The information in the "Block Merkle Tree" SHALL be used to validate the
* full content of the most recent block, and, with chained validation,
* all prior blocks.
*
* This message SHALL offer a state proof for the `BlockInfo` singleton at
* the end of a block. The information in the `BlockInfo` singleton SHALL be
* used to validate the full content of the most recent block, and, with
* chained validation, all prior blocks.
*/
message BlockStateProof {
/**
* The `BlockInfo` from state.<br/>
* This state proof SHALL be proof that this `BlockInfo` value is in
* state at the end of the "proven" block.
*/
proto.BlockInfo state_block_info = 1;

/**
* A list of sibling hashes.<br/>
* This SHALL contain sibling hashes at each level of tree from the
* `BlockInfo` leaf up to the root of the merkle tree.<br/>
* These values MUST be in the same "merkle path" order as present in state.
*/
repeated SiblingHashList sibling_hashes = 2;

/**
* A list of consensus node signatures over the state merkle tree root.<br/>
* This MUST contain signatures from strictly greater than two-thirds
* of all consensus nodes active in the network address book present in
* network state at the beginning of the current block.<br/>
* This MAY be a different valid subset of all active consensus nodes for
* each such node that produces a block stream.<br/>
* All signatures MUST be RSA signatures over the SHA2-384 hash at the
* root of the merkle tree.<br/>
* These signatures SHOULD be verified before accepting this state proof.
* <p>
* <blockquote>Note<blockquote>
* We intend to replace this field with a different structure; most likely
* a TSS-BLS threshold signature.
* </blockquote></blockquote>
*/
repeated NodeSignature node_signatures = 3;
}

/**
* A state signature.
* ### Block Merkle Tree
* The Block Hash of any block is a merkle root hash comprised of a 4 leaf
* binary merkle tree. The 4 leaves represent
* 1. Previous block proof hash
* 1. Merkle root of transaction inputs tree
* 1. Merkle root of transaction outputs tree
* 1. Merkle rook of state tree
*
* This value is a cryptographic signature (TSS_BLS, RSA_DSS, edDSA, or ECDSA)
* over the hash at the root of the network state merkle tree.
* Each such signature SHALL serve as assertion by a particular node that the
* given hash is the actual root hash of the network merkle state as observed
* by that node at the time the signature is generated.
* #### Computing the hash
* The process for computing a block hash is somewhat complex, and involves
* creating a "virtual" merkle tree to obtain the root merkle hash of
* that virtual tree.<br/>
* The merkle tree SHALL have a 4 part structure with 2 internal nodes,
* structured in a strictly binary tree.
* - The merkle tree root SHALL be the parent of both
* internal nodes.
* 1. The first "internal" node SHALL be the parent of the
* two "left-most" nodes.
* 1. The first leaf MUST be the previous block hash, and is a
* single 48-byte value.
* 1. The second leaf MUST be the root of a, strictly binary, merkle tree
* composed of all "input" block items in the block.<br/>
* Input items SHALL be transactions, system transactions,
* and events.<br/>
* Leaf nodes in this subtree SHALL be ordered in the same order
* that the block items are encountered in the stream.
* 1. The second "internal" node SHALL be the parent of the two
* "right-most" nodes.
* 1. The third leaf MUST be the root of a, strictly binary, merkle tree
* composed of all "output" block items in the block.<br/>
* Output items SHALL be transaction result, transaction
* output, and state changes.<br/>
* Leaf nodes in this subtree SHALL be ordered in the same order that
* the block items are encountered in the stream.
* 1. The fourth leaf MUST be the merkle tree root hash for network state
* at the end of the block, and is a single 48-byte value.
* - The block hash SHALL be the SHA-384 hash calculated for the root
* of this merkle tree.
*
* > REVIEW QUESTION
* >> Do we need time (e.g. consensus time or round)? The address book varies
* >> over time, and we require, here, the address book _as of_ the moment
* >> when the signature was generated, not necessarily the address book
* >> at some later instant. TSS_BLS may resolve this issue...
* The "inputs" and "outputs" subtrees SHALL be "complete" binary merkle trees,
* with nodes that would otherwise be missing replaced by a "null" hash
* leaf.
*/
message NodeSignature {
/**
* A cryptographic signature.
* <p>
* This value MUST be generated using the private key of the node
* recorded in `node_id`.<br/>
* This value MUST be generated by signing the SHA2-384 hash at the root of
* the merkle tree network state.<br/>
* This value MUST be generated using the current accepted signature
* algorithm. The current signature algorithm is RSA_DSS (TSS_BLS is a
* high probability near-future replacement).
*/
bytes signature = 1;

/**
* The node that generated this signature.
* <p>
* This SHALL provide the reference required to look up that node's
* public key in the address book at the time of signature. The correct
* node public key is REQUIRED in order to verify this signature.
*/
uint64 node_id = 2;
}

/**
* Sibling hashes at one level in the merkle tree.
*
* The set of hash values "adjacent" to a particular tree node are REQUIRED
* to generate the hash value of the "parent" tree node.<br/>
* This message SHALL contain the list of all such "sibling" hash values for
* a given node (not referenced here).<br/>
* This message MUST record the sibling hashes in the same "merkle tree order"
* as those hash values are stored in the network state.
*
* > REVIEW NOTE
* >> We need to identify where, in a list of sibling hashes, the current
* >> "target" hash belongs. For a purely binary tree, that would be are we
* >> the "left" or the "right" node. For an n-ary tree that's 'which
* >> siblings are to our "left", which are to our "right"?'.<br/>
* >> Options:
* >> <ol>
* >> <li>Include the "target" hash in the list at the correct location.</li>
* >> <li>For each sibling hash (which becomes a message), have a boolean for
* >> left/right.</li>
* >> <li>For each sibling hash (which becomes a message), have a oneof with
* >> left_hash and right_hash.</li>
* >> </ol>
* >> Option (1) is simple, works for n-ary trees, and makes parent hash
* >> calculation very simple. It does require more data (one added hash per
* >> level, which is expensive for a pure binary tree structure).<br/>
* >> Option (2) is slightly more complex, works well for a pure binary tree
* >> structure, but requires an added boolean field.<br/>
* >> Option (3) is slightly more complex, works well for a pure binary tree
* >> structure, and is the most data-efficient option. This also works for
* >> n-ary trees, with slightly more complex validation logic and a more
* >> complex specification of requirements.
*/
message SiblingHashList {
/**
* A list of hash values.
* <p>
* Each hash value in this list MUST be generated using the
* SHA384 algorithm. This list determines right-left ordering by
* including the target hash in the list in the correct
* merkle-tree-ordered location.
*/
repeated bytes sibling_hash = 1;

/**
* A list of hash values.
* <p>
* Each hash value in this list MUST be generated using the
* SHA384 algorithm. This list determines right-left by setting
* a boolean value.
*/
repeated BinarySiblingHash sibling_hashes_bool = 2;

/**
* A list of hash values.
* <p>
* Each hash value in this list MUST be generated using the
* SHA384 algorithm. This list determines right-left by using
* different fields in a `oneof`.
*/
repeated SiblingHash sibling_hashes_oneof = 3;
}

/**
* A single "sibling" hash.
*
* Sibling hashes have an order from "left" to "right". This message
* SHALL enable the specification of order as a matter of message schema.<br/>
* A system sending sibling hash data MUST structure any `repeated` field
* to contain all sibling hashes "left" of a target hash first, in tree
* order, and add all sibling hashes "right" of the target hash second,
* also in tree order.
*/
message SiblingHash {
oneof sibling_hash {
/**
* A "sibling" hash that is "left" of the target item hash in
* merkle-tree order.
*/
bytes left_sibling_hash = 1;
message BlockStateProof {

/**
* A "sibling" hash that is "right" of the target item hash in
* merkle-tree order.
*/
bytes right_sibling_hash = 2;
}
/**
* The round this proof secures.<br/>
* We provide this because a proof for a future round can be used to prove
* the state of the ledger at that round and the rounds before it.<br/>
* <p>
* This value SHOULD match the round of the current block,
* under normal operation.
*/
uint64 round = 1;

/**
* A merkle root hash.
* <p>
* This MUST contain a SHA-384 hash of the "block" merkle tree root.
*/
bytes block_root_hash = 2;

/**
* A TSS-BLS network signature.
* <p>
* This signature SHALL use a TSS-BLS threshold signature to provide
* a single signature that represents the consensus signature of at least
* the current threshold (i.e. 2/3 + 1) of consensus nodes. The exact subset
* of nodes that signed SHALL neither be known nor tracked, but it SHALL be
* cryptographically verifiable that the threshold was met if the signature
* itself can be validated with the network public key.
*/
BlockSignature block_signature = 3;
}

/**
* A single "sibling" hash.
* A TSS-BLS signature for one block.
* This is a single signature representing the collection of partial
* signatures from nodes holding strictly greater than 2/3 of the current
* network "weight" in aggregate. The signature is produced by cryptographic
* "aggregation" of the partial signatures to produce a single signature that
* can be verified with the network public key, but could not be produced by
* fewer nodes than required to meet the threshold for network stake "weight".
*
* Sibling hashes have an order from "left" to "right". This message SHALL
* enable the specification of order as an explicit left/right boolean.<br/>
* A system sending sibling hash data MUST structure any `repeated` field
* to contain all sibling hashes "left" of a target hash first, in tree
* order, and add all sibling hashes "right" of the target hash second,
* also in tree order.<br/>
* Any sibling hash "left" of the target hash MUST set `is_left_hash` to
* `true`.<br/>
* Any sibling hash "right" of the target hash MUST set `is_left_hash` to
* `false`.
* This message MUST make use of a threshold signature scheme like `BLS` which
* provides the necessary cryptographic guarantees.
*/
message BinarySiblingHash {
/**
* A "sibling" hash of the target item hash.
*/
bytes sibling_hash = 1;

/**
* A flag indicating that this "sibling" hash that is "left" of the target
* item hash in merkle-tree order.
*/
bool is_left_hash = 2;
message BlockSignature {
/**
* A single TSS-BLS signature for the block merkle tree.
*/
bytes signature = 1;
}

0 comments on commit efdafef

Please sign in to comment.