Skip to content

Commit

Permalink
Adjust structure and specification for new hashing approach.
Browse files Browse the repository at this point in the history 
* Moved "input" item to "input" folder and package.
* Moved "output" items to "output" folder and package.
* Specified the "merkle" approach to generating the block
  hash in the block header.
   * Added Jasper's diagram for this process in documents.

Signed-off-by: Joseph Sinclair <joseph.sinclair@swirldslabs.com>
  • Loading branch information
@jsync-swirlds
jsync-swirlds committed Jun 12, 2024
1 parent 62bdbe8 commit 55e7b70
Show file tree
Hide file tree
Showing 33 changed files with 413 additions and 272 deletions.
55 changes: 44 additions & 11 deletions block/block_service.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,39 @@
* # Block Service
* The Service API exposed by the Block Nodes.
*
* ## Workarounds
* > There are incorrect elements in this file to work around bugs in the
* > PBJ Compiler.
* >> Issues 262, 263, 240, 218, 217, and 216 are related.
* >
* > Issue 263
* >> A number of fields reference child messages, these _should_ specify
* >> the parent message (i.e. `Parent.child field = #;`) but do not do
* >> so due to issue 263.
* >
* > Issue 262
* >> Some fields reference messages defined in other packages that share
* >> a common prefix (e.g. `com.hedera.hapi.block.stream`). These fields
* >> specify the entire package instead of the shorter and clearer suffix
* >> due to issue 262
* >
* > Issue 240
* >> These files currently cause PBJ integration tests to fail if included
* >> due to issue 240.
* >
* > Issue 218
* >> These files have the same value for package and java_package. Ideally
* >> we would not specify `java_package` or the pbj comment in that situation,
* >> but Issue 218 prevents eliding the unnecessary directives.
* >
* > Issue 217
* >> These files may cause PBJ to fail compilation due to comments preceeding
* >> the `syntax` keyword.
* >
* > Issue 216
* >> These files would do well with validation support, but cannot make
* >> use of validation, even as an advisory element, due to Issue 216.
*
* ### Keywords
* The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
* "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
Expand Down Expand Up @@ -55,7 +88,7 @@ message PublishStreamRequest {
/**
* A single item written to the block stream.
*/
stream.BlockItem block_item = 2;
com.hedera.hapi.block.stream.BlockItem block_item = 2;
}

/**
Expand All @@ -79,12 +112,12 @@ message PublishStreamResponse {
/**
* A response sent for each item and for each block.
*/
PublishStreamResponse.Acknowledgement acknowledgement = 1;
Acknowledgement acknowledgement = 1;

/**
* A response sent when a stream ends.
*/
PublishStreamResponse.EndOfStream status = 2;
EndOfStream status = 2;
}

/**
Expand All @@ -96,12 +129,12 @@ message PublishStreamResponse {
/**
* A response type to acknowledge a full and complete block.
*/
PublishStreamResponse.BlockAcknowledgement block_ack = 1;
BlockAcknowledgement block_ack = 1;

/**
* A response type to acknowledge a single `BlockItem`.
*/
PublishStreamResponse.ItemAcknowledgement item_ack = 2;
ItemAcknowledgement item_ack = 2;
}
}

Expand Down Expand Up @@ -173,7 +206,7 @@ message PublishStreamResponse {
* This code indicates the reason the stream ended.<br/>
* This value MUST be set to a non-default value.
*/
PublishStreamResponse.ResponseCode status = 1;
ResponseCode status = 1;

/**
* The number of the last completed and _verified_ block.
Expand Down Expand Up @@ -283,7 +316,7 @@ message SingleBlockResponse {
* The reported status SHALL reflect the success of the request, or
* a detailed reason the request failed.
*/
SingleBlockResponse.ResponseCode status = 1;
ResponseCode status = 1;

/**
* The requested block.
Expand All @@ -295,7 +328,7 @@ message SingleBlockResponse {
* The items in this message SHALL begin with a `BlockHeader` and end with
* a `BlockStateProof` applicable to this block.
*/
stream.Block block = 2;
com.hedera.hapi.block.stream.Block block = 2;

/**
* An enumeration indicating the status of this request.
Expand Down Expand Up @@ -429,15 +462,15 @@ message SubscribeStreamResponse {
* <p>
* The block node server SHALL end the stream following this message.
*/
SubscribeStreamResponse.ResponseCode status = 1;
ResponseCode status = 1;

/**
* A stream response item containing a single `BlockItem`.
* <p>
* The full response SHALL consist of many `block_item` messages
* followed by a single `status` message.
*/
stream.BlockItem block_item = 2;
com.hedera.hapi.block.stream.BlockItem block_item = 2;
}

/**
Expand Down Expand Up @@ -545,7 +578,7 @@ message StateSnapshotResponse {
* This code SHALL indicate a successful call, or the detailed
* reason for failure.
*/
StateSnapshotResponse.ResponseCode status = 1;
ResponseCode status = 1;

/**
* A block number.
Expand Down
46 changes: 46 additions & 0 deletions block/stream/block_header.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,52 @@ message BlockHeader {
* `start_running_hash` that does not match the block hash of the previous
* block and MAY assume the block stream has encountered data loss,
* data corruption, or unauthorized modification.
* <p>
* The process for computing a block hash is somewhat complex, and involves
* creating a "virtual" merkle tree to obtain the root merkle hash of
* that virtual tree.<br/>
* The merkle tree SHALL have a 4 part structure with 2 internal nodes,
* structured in a strictly binary tree.
* <ul>
* <li>The merkle tree root SHALL be the parent of both
* internal nodes.
* <ol>
* <li>The first "internal" node SHALL be the parent of the
* two "left-most" nodes.
* <ol>
* <li>The first leaf MUST be the previous block hash, and is a
* single 48-byte value.</li>
* <li>The second leaf MUST be the root of a, strictly binary,
* merkle tree composed of all "input" block items in
* the block.<br/>
* Input items SHALL be transactions, system transactions,
* and events.<br/>
* Leaf nodes in this subtree SHALL be ordered in the
* same order that the block items are encountered
* in the stream.</li>
* </ol>
* </li>
* <li>The second "internal" node SHALL be the parent of the
* two "right-most" nodes.</li>
* <ol>
* <li>The third leaf MUST be the root of a, strictly binary,
* merkle tree composed of all "output" block items in
* the block.<br/>
* Output items SHALL be transaction result, transaction
* output, and state changes.<br/>
* Leaf nodes in this subtree SHALL be ordered in the
* same order that the block items are encountered
* in the stream.</li>
* <li>The fourth leaf MUST be the merkle tree root hash for
* network state at the end of the block, and is a single
* 48-byte value.</li>
* </ol>
* </li>
* </ol>
* </li>
* <li>The block hash SHALL be the SHA-384 hash calculated for the root
* of this merkle tree.</li>
* </ul>
*/
bytes start_running_hash = 3;

Expand Down
30 changes: 15 additions & 15 deletions block/stream/block_item.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
*
* This structure here MUST support a stream of block items with no enclosing
* message.<br/>
* The design SHOULD be reasonable if used in a gRPC bidirectional streaming
* RPC similar to
* Implementations SHOULD behave in a reasonable manner if used in a gRPC
* bidirectional streaming RPC similar to
* `rpc processBlocks(stream BlockItem) returns (stream Acknowledgement);`.
*
* ### Keywords
Expand Down Expand Up @@ -47,10 +47,10 @@ import "transaction.proto";
import "stream/block_header.proto";
import "stream/block_state_proof.proto";
import "stream/event_metadata.proto";
import "stream/state_changes.proto";
import "stream/system_transaction.proto";
import "stream/transaction_output.proto";
import "stream/transaction_result.proto";
import "stream/output/state_changes.proto";
import "stream/input/system_transaction.proto";
import "stream/output/transaction_output.proto";
import "stream/output/transaction_result.proto";

/**
* A single item within a block stream.
Expand All @@ -65,7 +65,7 @@ import "stream/transaction_result.proto";
* - A `transaction` SHALL be followed by a `transaction_result`.
* - A `transaction_result` MAY be followed by a `transaction_output`.
* - A `transaction_result` (or a `transaction_output`, if present) MAY be
* followed by a `state_changes`.
* followed by one or more `state_changes`.
*
* This forms the following required sequence for each block, which is then
* repeated within the block stream, indefinitely. Note that there is no
Expand All @@ -82,7 +82,7 @@ import "stream/transaction_result.proto";
* transaction
* transaction_result
* (optional) transaction_output
* (optional) state_changes
* (optional) repeated state_changes
* }
* }
* state_proof
Expand Down Expand Up @@ -113,7 +113,7 @@ message BlockItem {
* <p>
* This item MAY be removed as the platform removes the concept.
*/
SystemTransaction system_transaction = 3;
com.hedera.hapi.block.stream.input.SystemTransaction system_transaction = 3;

/**
* A raw network transaction.<br/>
Expand All @@ -130,7 +130,7 @@ message BlockItem {
* This item MAY be redacted in some circumstances, and SHALL be
* replaced with a `filtered_item` if removed.
*/
TransactionResult transaction_result = 5;
com.hedera.hapi.block.stream.output.TransactionResult transaction_result = 5;

/**
* A transaction output.
Expand All @@ -141,7 +141,7 @@ message BlockItem {
* in state changes, then this item MUST be present after the
* `transaction_result` for that transaction.
*/
TransactionOutput transaction_output = 6;
com.hedera.hapi.block.stream.output.TransactionOutput transaction_output = 6;

/**
* A set of state changes.
Expand All @@ -151,7 +151,7 @@ message BlockItem {
* The source of these state changes SHALL be described by the
* `reason` enumeration.
*/
StateChanges state_changes = 7;
com.hedera.hapi.block.stream.output.StateChanges state_changes = 7;

/**
* A block state proof.<br/>
Expand Down Expand Up @@ -186,9 +186,9 @@ message BlockItem {
*
* > Note:
* >> This may change. We are exploring options that may change this value to
* >> work with, merkle hashing rather than a running hash chain; the change
* >> is expected to improve some forms of verification and resolve edge
* >> cases with running hashes.
* >> work with merkle hashing rather than a simple running hash chain; the
* >> change is expected to improve some forms of verification and resolve
* >> edge cases with simple running hashes.
*
* Items of this type SHALL NOT be present in the full (unfiltered) block
* stream.<br/>
Expand Down
23 changes: 19 additions & 4 deletions block/stream/block_state_proof.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,20 @@ option java_multiple_files = true;
/**
* A state proof for the `BlockInfo` singleton in network state.
*
* > Important Note
* >> This design is very much IN FLUX and should not be relied upon.
* >> Significant changes are known to be needed to support efficient
* >> computation of an effective state proof while also supporting
* >> reasonably efficient validation of that state proof in an
* >> EVM-compatible smart contract execution environment.
* >
* >> The major items of concern include inefficient computation of 512-bit
* >> values required for SHA2-384 hashes (and all SHA3 hashes), signature
* >> verification of BLS signatures using the BLS12 family of curves, and
* >> possible needs for zero-knowledge proofs, including SNARKS based on
* >> the KZG commitments already present for Ethereum and future STARKS
* >> based on quantum-resistant hash functions.
*
* This message SHALL offer a state proof for the `BlockInfo` singleton at
* the end of a block. The information in the `BlockInfo` singleton SHALL be
* used to validate the full content of the most recent block, and, with
Expand Down Expand Up @@ -92,8 +106,8 @@ message BlockStateProof {
/**
* A state signature.
*
* This value is a cryptographic signature (RSA_DSS, edDSA, or ECDSA) over the
* hash at the root of the network state merkle tree.
* This value is a cryptographic signature (TSS_BLS, RSA_DSS, edDSA, or ECDSA)
* over the hash at the root of the network state merkle tree.
* Each such signature SHALL serve as assertion by a particular node that the
* given hash is the actual root hash of the network merkle state as observed
* by that node at the time the signature is generated.
Expand All @@ -102,7 +116,7 @@ message BlockStateProof {
* >> Do we need time (e.g. consensus time or round)? The address book varies
* >> over time, and we require, here, the address book _as of_ the moment
* >> when the signature was generated, not necessarily the address book
* >> at some later instant.
* >> at some later instant. TSS_BLS may resolve this issue...
*/
message NodeSignature {
/**
Expand All @@ -113,7 +127,8 @@ message NodeSignature {
* This value MUST be generated by signing the SHA2-384 hash at the root of
* the merkle tree network state.<br/>
* This value MUST be generated using the current accepted signature
* algorithm. The current signature algorithm is RSA_DSS.
* algorithm. The current signature algorithm is RSA_DSS (TSS_BLS is a
* high probability near-future replacement).
*/
bytes signature = 1;

Expand Down
6 changes: 3 additions & 3 deletions block/stream/system_transaction.proto → block/stream/input/system_transaction.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
*/
syntax = "proto3";

package com.hedera.hapi.block.stream;
package com.hedera.hapi.block.stream.input;

/*
* Copyright (C) 2024 Hedera Hashgraph, LLC
Expand All @@ -46,8 +46,8 @@ package com.hedera.hapi.block.stream;
* limitations under the License.
*/

option java_package = "com.hedera.hapi.block.stream";
// <<<pbj.java_package = "com.hedera.hapi.block.stream">>> This comment is special code for setting PBJ Compiler java package
option java_package = "com.hedera.hapi.block.stream.input";
// <<<pbj.java_package = "com.hedera.hapi.block.stream.input">>> This comment is special code for setting PBJ Compiler java package
option java_multiple_files = true;

/**
Expand Down
6 changes: 3 additions & 3 deletions block/stream/consensus_service.proto → block/stream/output/consensus_service.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
*/
syntax = "proto3";

package com.hedera.hapi.block.stream;
package com.hedera.hapi.block.stream.output;

/*
* Copyright (C) 2024 Hedera Hashgraph, LLC
Expand All @@ -35,8 +35,8 @@ package com.hedera.hapi.block.stream;
* limitations under the License.
*/

option java_package = "com.hedera.hapi.block.stream";
// <<<pbj.java_package = "com.hedera.hapi.block.stream">>> This comment is special code for setting PBJ Compiler java package
option java_package = "com.hedera.hapi.block.stream.output";
// <<<pbj.java_package = "com.hedera.hapi.block.stream.output">>> This comment is special code for setting PBJ Compiler java package
option java_multiple_files = true;

/**
Expand Down
6 changes: 3 additions & 3 deletions block/stream/crypto_service.proto → block/stream/output/crypto_service.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
*/
syntax = "proto3";

package com.hedera.hapi.block.stream;
package com.hedera.hapi.block.stream.output;

/*
* Copyright (C) 2024 Hedera Hashgraph, LLC
Expand All @@ -34,8 +34,8 @@ package com.hedera.hapi.block.stream;
* limitations under the License.
*/

option java_package = "com.hedera.hapi.block.stream";
// <<<pbj.java_package = "com.hedera.hapi.block.stream">>> This comment is special code for setting PBJ Compiler java package
option java_package = "com.hedera.hapi.block.stream.output";
// <<<pbj.java_package = "com.hedera.hapi.block.stream.output">>> This comment is special code for setting PBJ Compiler java package
option java_multiple_files = true;

import "custom_fees.proto";
Expand Down
Loading

0 comments on commit 55e7b70

Please sign in to comment.