From 21cf61ea8f339c36c276003844f002e70f491741 Mon Sep 17 00:00:00 2001 From: dapplion <35266934+dapplion@users.noreply.github.com> Date: Thu, 15 Dec 2022 11:28:20 +0700 Subject: [PATCH 01/10] De-duplicate range references in BlobSidecarsByRange --- specs/deneb/p2p-interface.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specs/deneb/p2p-interface.md b/specs/deneb/p2p-interface.md index 107c90b9a9..51180f8a2d 100644 --- a/specs/deneb/p2p-interface.md +++ b/specs/deneb/p2p-interface.md @@ -256,18 +256,18 @@ The request MUST be encoded as an SSZ-container. The response MUST consist of zero or more `response_chunk`. Each _successful_ `response_chunk` MUST contain a single `BlobSidecar` payload. -Clients MUST keep a record of signed blobs sidecars seen on the epoch range -`[max(current_epoch - MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS, DENEB_FORK_EPOCH), current_epoch]` +Let `blob_serve_range` be `[max(current_epoch - MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS, DENEB_FORK_EPOCH), current_epoch]`. +Clients MUST keep a record of signed blob sidecars seen on the epoch range `blob_serve_range` where `current_epoch` is defined by the current wall-clock time, and clients MUST support serving requests of blobs on this range. -Peers that are unable to reply to blob sidecar requests within the `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` -epoch range SHOULD respond with error code `3: ResourceUnavailable`. +Peers that are unable to reply to blob sidecar requests within the +range `blob_serve_range` SHOULD respond with error code `3: ResourceUnavailable`. Such peers that are unable to successfully reply to this range of requests MAY get descored or disconnected at any time. *Note*: The above requirement implies that nodes that start from a recent weak subjectivity checkpoint -MUST backfill the local blobs database to at least epoch `current_epoch - MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` +MUST backfill the local blobs database to at least the range `blob_serve_range` to be fully compliant with `BlobSidecarsByRange` requests. *Note*: Although clients that bootstrap from a weak subjectivity checkpoint can begin From f5e00848d0f74cd440b996ae716d58947b868173 Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Wed, 7 Jun 2023 11:59:51 +0600 Subject: [PATCH 02/10] Remove assert from is_data_available --- specs/deneb/fork-choice.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/deneb/fork-choice.md b/specs/deneb/fork-choice.md index 283e76b1df..8b3c224236 100644 --- a/specs/deneb/fork-choice.md +++ b/specs/deneb/fork-choice.md @@ -42,8 +42,7 @@ def is_data_available(beacon_block_root: Root, blob_kzg_commitments: Sequence[KZ if isinstance(blobs, str) or isinstance(proofs, str): return True - assert verify_blob_kzg_proof_batch(blobs, blob_kzg_commitments, proofs) - return True + return verify_blob_kzg_proof_batch(blobs, blob_kzg_commitments, proofs) ``` ## Updated fork-choice handlers From a547d47264e40adca4b06fce1954e22ba84e0c27 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 7 Jun 2023 17:45:39 +0800 Subject: [PATCH 03/10] specially mark the EIP4844 changes --- specs/deneb/beacon-chain.md | 38 ++++++++++++----------- specs/deneb/fork-choice.md | 4 ++- specs/deneb/fork.md | 6 ++-- specs/deneb/light-client/fork.md | 4 +-- specs/deneb/light-client/full-node.md | 2 +- specs/deneb/light-client/sync-protocol.md | 2 +- specs/deneb/p2p-interface.md | 14 +++++++-- specs/deneb/polynomial-commitments.md | 2 +- specs/deneb/validator.md | 2 ++ 9 files changed, 44 insertions(+), 30 deletions(-) diff --git a/specs/deneb/beacon-chain.md b/specs/deneb/beacon-chain.md index 489ae6151c..5c3b4e8a4d 100644 --- a/specs/deneb/beacon-chain.md +++ b/specs/deneb/beacon-chain.md @@ -41,16 +41,15 @@ ## Introduction -This upgrade adds blobs to the beacon chain as part of Deneb. This is an extension of the Capella upgrade. - -The blob transactions are packed into the execution payload by the EL/builder with their corresponding blobs being independently transmitted and are limited by `MAX_DATA_GAS_PER_BLOCK // DATA_GAS_PER_BLOB`. However the CL limit is independently defined by `MAX_BLOBS_PER_BLOCK`. +Deneb is a consensus-layer upgrade containing a number of features. Including: +* [EIP-4844](https://eips.ethereum.org/EIPS/eip-4844): Shard Blob Transactions scale data-availability of Ethereum in a simple, forwards-compatible manner. ## Custom types | Name | SSZ equivalent | Description | | - | - | - | -| `VersionedHash` | `Bytes32` | | -| `BlobIndex` | `uint64` | | +| `VersionedHash` | `Bytes32` | [New in Deneb:EIP4844] | +| `BlobIndex` | `uint64` | [New in Deneb:EIP4844] | ## Constants @@ -73,8 +72,11 @@ The blob transactions are packed into the execution payload by the EL/builder wi | Name | Value | Description | | - | - | - | -| `MAX_BLOB_COMMITMENTS_PER_BLOCK` | `uint64(2**12)` (= 4096) | hardfork independent fixed theoretical limit same as `LIMIT_BLOBS_PER_TX` (see EIP 4844) | -| `MAX_BLOBS_PER_BLOCK` | `uint64(2**2)` (= 4) | Maximum number of blobs in a single block limited by `MAX_BLOB_COMMITMENTS_PER_BLOCK` | +| `MAX_BLOB_COMMITMENTS_PER_BLOCK` | `uint64(2**12)` (= 4096) | [New in Deneb:EIP4844] hardfork independent fixed theoretical limit same as `LIMIT_BLOBS_PER_TX` (see EIP 4844) | +| `MAX_BLOBS_PER_BLOCK` | `uint64(2**2)` (= 4) | [New in Deneb:EIP4844] Maximum number of blobs in a single block limited by `MAX_BLOB_COMMITMENTS_PER_BLOCK` | + +*Note*: The blob transactions are packed into the execution payload by the EL/builder with their corresponding blobs being independently transmitted +and are limited by `MAX_DATA_GAS_PER_BLOCK // DATA_GAS_PER_BLOB`. However the CL limit is independently defined by `MAX_BLOBS_PER_BLOCK`. ## Configuration @@ -100,9 +102,9 @@ class BeaconBlockBody(Container): voluntary_exits: List[SignedVoluntaryExit, MAX_VOLUNTARY_EXITS] sync_aggregate: SyncAggregate # Execution - execution_payload: ExecutionPayload # [Modified in Deneb] + execution_payload: ExecutionPayload # [Modified in Deneb:EIP4844] bls_to_execution_changes: List[SignedBLSToExecutionChange, MAX_BLS_TO_EXECUTION_CHANGES] - blob_kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK] # [New in Deneb] + blob_kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK] # [New in Deneb:EIP4844] ``` #### `ExecutionPayload` @@ -126,8 +128,8 @@ class ExecutionPayload(Container): block_hash: Hash32 # Hash of execution block transactions: List[Transaction, MAX_TRANSACTIONS_PER_PAYLOAD] withdrawals: List[Withdrawal, MAX_WITHDRAWALS_PER_PAYLOAD] - data_gas_used: uint64 # [New in Deneb] - excess_data_gas: uint64 # [New in Deneb] + data_gas_used: uint64 # [New in Deneb:EIP4844] + excess_data_gas: uint64 # [New in Deneb:EIP4844] ``` #### `ExecutionPayloadHeader` @@ -151,8 +153,8 @@ class ExecutionPayloadHeader(Container): block_hash: Hash32 # Hash of execution block transactions_root: Root withdrawals_root: Root - data_gas_used: uint64 # [New in Deneb] - excess_data_gas: uint64 # [New in Deneb] + data_gas_used: uint64 # [New in Deneb:EIP4844] + excess_data_gas: uint64 # [New in Deneb:EIP4844] ``` ## Helper functions @@ -205,7 +207,7 @@ def verify_and_notify_new_payload(self: ExecutionEngine, if not self.is_valid_block_hash(new_payload_request.execution_payload): return False - # [New in Deneb] + # [New in Deneb:EIP4844] if not self.is_valid_versioned_hashes(new_payload_request): return False @@ -232,11 +234,11 @@ def process_execution_payload(state: BeaconState, body: BeaconBlockBody, executi # Verify timestamp assert payload.timestamp == compute_timestamp_at_slot(state, state.slot) - # [New in Deneb] Verify commitments are under limit + # [New in Deneb:EIP4844] Verify commitments are under limit assert len(body.blob_kzg_commitments) <= MAX_BLOBS_PER_BLOCK # Verify the execution payload is valid - # [Modified in Deneb] Pass `versioned_hashes` to Execution Engine + # [Modified in Deneb:EIP4844] Pass `versioned_hashes` to Execution Engine versioned_hashes = [kzg_commitment_to_versioned_hash(commitment) for commitment in body.blob_kzg_commitments] assert execution_engine.verify_and_notify_new_payload( NewPayloadRequest(execution_payload=payload, versioned_hashes=versioned_hashes) @@ -259,8 +261,8 @@ def process_execution_payload(state: BeaconState, body: BeaconBlockBody, executi block_hash=payload.block_hash, transactions_root=hash_tree_root(payload.transactions), withdrawals_root=hash_tree_root(payload.withdrawals), - data_gas_used=payload.data_gas_used, # [New in Deneb] - excess_data_gas=payload.excess_data_gas, # [New in Deneb] + data_gas_used=payload.data_gas_used, # [New in Deneb:EIP4844] + excess_data_gas=payload.excess_data_gas, # [New in Deneb:EIP4844] ) ``` diff --git a/specs/deneb/fork-choice.md b/specs/deneb/fork-choice.md index 283e76b1df..60e05f8268 100644 --- a/specs/deneb/fork-choice.md +++ b/specs/deneb/fork-choice.md @@ -25,6 +25,8 @@ This is the modification of the fork choice accompanying the Deneb upgrade. #### `is_data_available` +[New in Deneb:EIP4844] + The implementation of `is_data_available` will become more sophisticated during later scaling upgrades. Initially, verification requires every verifying actor to retrieve all matching `Blob`s and `KZGProof`s, and validate them with `verify_blob_kzg_proof_batch`. @@ -76,7 +78,7 @@ def on_block(store: Store, signed_block: SignedBeaconBlock) -> None: ) assert store.finalized_checkpoint.root == finalized_checkpoint_block - # [New in Deneb] + # [New in Deneb:EIP4844] # Check if blob data is available # If not, this block MAY be queued and subsequently considered when blob data becomes available assert is_data_available(hash_tree_root(block), block.body.blob_kzg_commitments) diff --git a/specs/deneb/fork.md b/specs/deneb/fork.md index 7fc80275d6..ffced6a59a 100644 --- a/specs/deneb/fork.md +++ b/specs/deneb/fork.md @@ -83,8 +83,8 @@ def upgrade_to_deneb(pre: capella.BeaconState) -> BeaconState: block_hash=pre.latest_execution_payload_header.block_hash, transactions_root=pre.latest_execution_payload_header.transactions_root, withdrawals_root=pre.latest_execution_payload_header.withdrawals_root, - data_gas_used=uint64(0), # [New in Deneb] - excess_data_gas=uint64(0), # [New in Deneb] + data_gas_used=uint64(0), # [New in Deneb:EIP4844] + excess_data_gas=uint64(0), # [New in Deneb:EIP4844] ) post = BeaconState( # Versioning @@ -126,7 +126,7 @@ def upgrade_to_deneb(pre: capella.BeaconState) -> BeaconState: current_sync_committee=pre.current_sync_committee, next_sync_committee=pre.next_sync_committee, # Execution-layer - latest_execution_payload_header=latest_execution_payload_header, # [Modified in Deneb] + latest_execution_payload_header=latest_execution_payload_header, # [Modified in Deneb:EIP4844] # Withdrawals next_withdrawal_index=pre.next_withdrawal_index, next_withdrawal_validator_index=pre.next_withdrawal_validator_index, diff --git a/specs/deneb/light-client/fork.md b/specs/deneb/light-client/fork.md index 2fd410fc56..f4fd1b396b 100644 --- a/specs/deneb/light-client/fork.md +++ b/specs/deneb/light-client/fork.md @@ -41,8 +41,8 @@ def upgrade_lc_header_to_deneb(pre: capella.LightClientHeader) -> LightClientHea block_hash=pre.execution.block_hash, transactions_root=pre.execution.transactions_root, withdrawals_root=pre.execution.withdrawals_root, - data_gas_used=uint64(0), # [New in Deneb] - excess_data_gas=uint64(0), # [New in Deneb] + data_gas_used=uint64(0), # [New in Deneb:EIP4844] + excess_data_gas=uint64(0), # [New in Deneb:EIP4844] ), execution_branch=pre.execution_branch, ) diff --git a/specs/deneb/light-client/full-node.md b/specs/deneb/light-client/full-node.md index 69ba0d3bab..876a6c2580 100644 --- a/specs/deneb/light-client/full-node.md +++ b/specs/deneb/light-client/full-node.md @@ -47,7 +47,7 @@ def block_to_light_client_header(block: SignedBeaconBlock) -> LightClientHeader: withdrawals_root=hash_tree_root(payload.withdrawals), ) - # [New in Deneb] + # [New in Deneb:EIP4844] if epoch >= DENEB_FORK_EPOCH: execution_header.data_gas_used = payload.data_gas_used execution_header.excess_data_gas = payload.excess_data_gas diff --git a/specs/deneb/light-client/sync-protocol.md b/specs/deneb/light-client/sync-protocol.md index b7b9b5ab1f..38909ddbf5 100644 --- a/specs/deneb/light-client/sync-protocol.md +++ b/specs/deneb/light-client/sync-protocol.md @@ -66,7 +66,7 @@ def get_lc_execution_root(header: LightClientHeader) -> Root: def is_valid_light_client_header(header: LightClientHeader) -> bool: epoch = compute_epoch_at_slot(header.beacon.slot) - # [New in Deneb] + # [New in Deneb:EIP4844] if epoch < DENEB_FORK_EPOCH: if header.execution.data_gas_used != uint64(0) or header.execution.excess_data_gas != uint64(0): return False diff --git a/specs/deneb/p2p-interface.md b/specs/deneb/p2p-interface.md index caf2b5baea..e9123c3096 100644 --- a/specs/deneb/p2p-interface.md +++ b/specs/deneb/p2p-interface.md @@ -50,6 +50,8 @@ The specification of these changes continues in the same format as the network s #### `BlobSidecar` +[New in Deneb:EIP4844] + ```python class BlobSidecar(Container): block_root: Root @@ -64,6 +66,8 @@ class BlobSidecar(Container): #### `SignedBlobSidecar` +[New in Deneb:EIP4844] + ```python class SignedBlobSidecar(Container): message: BlobSidecar @@ -72,6 +76,8 @@ class SignedBlobSidecar(Container): #### `BlobIdentifier` +[New in Deneb:EIP4844] + ```python class BlobIdentifier(Container): block_root: Root @@ -107,7 +113,7 @@ The new topics along with the type of the `data` field of a gossipsub message ar | Name | Message Type | | - | - | -| `blob_sidecar_{subnet_id}` | `SignedBlobSidecar` (new) | +| `blob_sidecar_{subnet_id}` | `SignedBlobSidecar` [New in Deneb:EIP4844] | ##### Global topics @@ -124,6 +130,8 @@ New validation: ###### `blob_sidecar_{subnet_id}` +[New in Deneb:EIP4844] + This topic is used to propagate signed blob sidecars, where each blob index maps to some `subnet_id`. The following validations MUST pass before forwarding the `signed_blob_sidecar` on the network, assuming the alias `sidecar = signed_blob_sidecar.message`: @@ -191,7 +199,7 @@ No more than `MAX_REQUEST_BLOCKS_DENEB` may be requested at a time. **Protocol ID:** `/eth2/beacon_chain/req/blob_sidecars_by_root/1/` -New in deneb. +[New in Deneb:EIP4844] The `` field is calculated as `context = compute_fork_digest(fork_version, genesis_validators_root)`: @@ -240,7 +248,7 @@ Clients MAY limit the number of blocks and sidecars in the response. **Protocol ID:** `/eth2/beacon_chain/req/blob_sidecars_by_range/1/` -New in deneb. +[New in Deneb:EIP4844] The `` field is calculated as `context = compute_fork_digest(fork_version, genesis_validators_root)`: diff --git a/specs/deneb/polynomial-commitments.md b/specs/deneb/polynomial-commitments.md index e800d8cc27..8a26b9ade5 100644 --- a/specs/deneb/polynomial-commitments.md +++ b/specs/deneb/polynomial-commitments.md @@ -336,7 +336,7 @@ def evaluate_polynomial_in_evaluation_form(polynomial: Polynomial, ### KZG -KZG core functions. These are also defined in Deneb execution specs. +KZG core functions. These are also defined in Deneb execution specs for EIP-4844. #### `blob_to_kzg_commitment` diff --git a/specs/deneb/validator.md b/specs/deneb/validator.md index ae5f266737..5af7c4938a 100644 --- a/specs/deneb/validator.md +++ b/specs/deneb/validator.md @@ -98,6 +98,8 @@ All validator responsibilities remain unchanged other than those noted below. ##### Blob KZG commitments +[New in Deneb:EIP4844] + 1. After retrieving the execution payload from the execution engine as specified in Capella, use the `payload_id` to retrieve `blobs`, `blob_kzg_commitments`, and `blob_kzg_proofs` via `get_payload(payload_id).blobs_bundle`. From b7e21d6caf94980318f4892d35096c2061b10f2c Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 8 Jun 2023 15:05:46 +0800 Subject: [PATCH 04/10] PR feedback and make the marks italic --- specs/deneb/beacon-chain.md | 8 ++++---- specs/deneb/fork-choice.md | 2 +- specs/deneb/p2p-interface.md | 12 ++++++------ specs/deneb/polynomial-commitments.md | 4 ++-- specs/deneb/validator.md | 10 +++++++--- 5 files changed, 20 insertions(+), 16 deletions(-) diff --git a/specs/deneb/beacon-chain.md b/specs/deneb/beacon-chain.md index 5c3b4e8a4d..41ce76bd2f 100644 --- a/specs/deneb/beacon-chain.md +++ b/specs/deneb/beacon-chain.md @@ -48,8 +48,8 @@ Deneb is a consensus-layer upgrade containing a number of features. Including: | Name | SSZ equivalent | Description | | - | - | - | -| `VersionedHash` | `Bytes32` | [New in Deneb:EIP4844] | -| `BlobIndex` | `uint64` | [New in Deneb:EIP4844] | +| `VersionedHash` | `Bytes32` | *[New in Deneb:EIP4844]* | +| `BlobIndex` | `uint64` | *[New in Deneb:EIP4844]* | ## Constants @@ -72,8 +72,8 @@ Deneb is a consensus-layer upgrade containing a number of features. Including: | Name | Value | Description | | - | - | - | -| `MAX_BLOB_COMMITMENTS_PER_BLOCK` | `uint64(2**12)` (= 4096) | [New in Deneb:EIP4844] hardfork independent fixed theoretical limit same as `LIMIT_BLOBS_PER_TX` (see EIP 4844) | -| `MAX_BLOBS_PER_BLOCK` | `uint64(2**2)` (= 4) | [New in Deneb:EIP4844] Maximum number of blobs in a single block limited by `MAX_BLOB_COMMITMENTS_PER_BLOCK` | +| `MAX_BLOB_COMMITMENTS_PER_BLOCK` | `uint64(2**12)` (= 4096) | *[New in Deneb:EIP4844]* hardfork independent fixed theoretical limit same as `LIMIT_BLOBS_PER_TX` (see EIP 4844) | +| `MAX_BLOBS_PER_BLOCK` | `uint64(2**2)` (= 4) | *[New in Deneb:EIP4844]* Maximum number of blobs in a single block limited by `MAX_BLOB_COMMITMENTS_PER_BLOCK` | *Note*: The blob transactions are packed into the execution payload by the EL/builder with their corresponding blobs being independently transmitted and are limited by `MAX_DATA_GAS_PER_BLOCK // DATA_GAS_PER_BLOB`. However the CL limit is independently defined by `MAX_BLOBS_PER_BLOCK`. diff --git a/specs/deneb/fork-choice.md b/specs/deneb/fork-choice.md index 60e05f8268..cc7373c093 100644 --- a/specs/deneb/fork-choice.md +++ b/specs/deneb/fork-choice.md @@ -25,7 +25,7 @@ This is the modification of the fork choice accompanying the Deneb upgrade. #### `is_data_available` -[New in Deneb:EIP4844] +*[New in Deneb:EIP4844]* The implementation of `is_data_available` will become more sophisticated during later scaling upgrades. Initially, verification requires every verifying actor to retrieve all matching `Blob`s and `KZGProof`s, and validate them with `verify_blob_kzg_proof_batch`. diff --git a/specs/deneb/p2p-interface.md b/specs/deneb/p2p-interface.md index e9123c3096..0a61ef800c 100644 --- a/specs/deneb/p2p-interface.md +++ b/specs/deneb/p2p-interface.md @@ -50,7 +50,7 @@ The specification of these changes continues in the same format as the network s #### `BlobSidecar` -[New in Deneb:EIP4844] +*[New in Deneb:EIP4844]* ```python class BlobSidecar(Container): @@ -66,7 +66,7 @@ class BlobSidecar(Container): #### `SignedBlobSidecar` -[New in Deneb:EIP4844] +*[New in Deneb:EIP4844]* ```python class SignedBlobSidecar(Container): @@ -76,7 +76,7 @@ class SignedBlobSidecar(Container): #### `BlobIdentifier` -[New in Deneb:EIP4844] +*[New in Deneb:EIP4844]* ```python class BlobIdentifier(Container): @@ -130,7 +130,7 @@ New validation: ###### `blob_sidecar_{subnet_id}` -[New in Deneb:EIP4844] +*[New in Deneb:EIP4844]* This topic is used to propagate signed blob sidecars, where each blob index maps to some `subnet_id`. @@ -199,7 +199,7 @@ No more than `MAX_REQUEST_BLOCKS_DENEB` may be requested at a time. **Protocol ID:** `/eth2/beacon_chain/req/blob_sidecars_by_root/1/` -[New in Deneb:EIP4844] +*[New in Deneb:EIP4844]* The `` field is calculated as `context = compute_fork_digest(fork_version, genesis_validators_root)`: @@ -248,7 +248,7 @@ Clients MAY limit the number of blocks and sidecars in the response. **Protocol ID:** `/eth2/beacon_chain/req/blob_sidecars_by_range/1/` -[New in Deneb:EIP4844] +*[New in Deneb:EIP4844]* The `` field is calculated as `context = compute_fork_digest(fork_version, genesis_validators_root)`: diff --git a/specs/deneb/polynomial-commitments.md b/specs/deneb/polynomial-commitments.md index 8a26b9ade5..94299aaa2f 100644 --- a/specs/deneb/polynomial-commitments.md +++ b/specs/deneb/polynomial-commitments.md @@ -49,7 +49,7 @@ ## Introduction -This document specifies basic polynomial operations and KZG polynomial commitment operations as they are needed for the Deneb specification. The implementations are not optimized for performance, but readability. All practical implementations should optimize the polynomial operations. +This document specifies basic polynomial operations and KZG polynomial commitment operations that are essential for the implementation of the EIP-4844 feature in the Deneb specification. The implementations are not optimized for performance, but readability. All practical implementations should optimize the polynomial operations. Functions flagged as "Public method" MUST be provided by the underlying KZG library as public functions. All other functions are private functions used internally by the KZG library. @@ -336,7 +336,7 @@ def evaluate_polynomial_in_evaluation_form(polynomial: Polynomial, ### KZG -KZG core functions. These are also defined in Deneb execution specs for EIP-4844. +KZG core functions. These are also defined in Deneb execution specs. #### `blob_to_kzg_commitment` diff --git a/specs/deneb/validator.md b/specs/deneb/validator.md index 5af7c4938a..b53c0a042d 100644 --- a/specs/deneb/validator.md +++ b/specs/deneb/validator.md @@ -46,12 +46,14 @@ Please see related Beacon Chain doc before continuing and use them as a referenc | Name | Value | Unit | | - | - | :-: | -| `BLOB_SIDECAR_SUBNET_COUNT` | `4` | The number of blob sidecar subnets used in the gossipsub protocol. | +| `BLOB_SIDECAR_SUBNET_COUNT` | `4` | *[New in Deneb:EIP4844]* The number of blob sidecar subnets used in the gossipsub protocol. | ## Helpers ### `BlobsBundle` +*[New in Deneb:EIP4844]* + ```python @dataclass class BlobsBundle(object): @@ -67,7 +69,7 @@ class BlobsBundle(object): class GetPayloadResponse(object): execution_payload: ExecutionPayload block_value: uint256 - blobs_bundle: BlobsBundle + blobs_bundle: BlobsBundle # [New in Deneb:EIP4844] ``` ## Protocol @@ -98,7 +100,7 @@ All validator responsibilities remain unchanged other than those noted below. ##### Blob KZG commitments -[New in Deneb:EIP4844] +*[New in Deneb:EIP4844]* 1. After retrieving the execution payload from the execution engine as specified in Capella, use the `payload_id` to retrieve `blobs`, `blob_kzg_commitments`, and `blob_kzg_proofs` @@ -107,6 +109,8 @@ via `get_payload(payload_id).blobs_bundle`. #### Constructing the `SignedBlobSidecar`s +*[New in Deneb:EIP4844]* + To construct a `SignedBlobSidecar`, a `signed_blob_sidecar` is defined with the necessary context for block and sidecar proposal. ##### Sidecar From 241e52a20230a970a156b7d9501d9a98ed8954c1 Mon Sep 17 00:00:00 2001 From: Lion - dapplion <35266934+dapplion@users.noreply.github.com> Date: Thu, 8 Jun 2023 10:35:03 +0300 Subject: [PATCH 05/10] Whisk (SSLE) with Curdleproofs - rebased (#3342) * Introduce consensus code for Whisk * polish, simplify, clean up (~100 fewer lines) @asn-d6: As discussed, I fixed a few bugs along the way but likely also introduced some bugs :) * minor cleanups and fixes * simplify is_k_commitment_unique * Update beacon-chain.md * Update beacon-chain.md * Initialize `k` in `get_validator_from_deposit()` * minor cleanups * Update beacon-chain.md * Create beacon-chain.md This PR changes the Whisk tracker format to be of the form `(r * pubkey, r * BLS_GT_GENERATOR)` instead of `(r * k * BLS_G1_GENERATOR, r * BLS_G1_GENERATOR)`. This allows for non-interactive tracker registrations from validator pubkeys, removing ~50 lines the code. It also significantly reduces the amount of state overhead. This PR also removes permutation commitments, though those can be easily readded if deemed necessary. * A couple of fixes to the no-registration simplification @asn-d6: Readded a consistency check for `IsValidWhiskOpeningProof` (involving `pubkey` instead of `k_commitment`). * remove unused helpers * use Mary's suggested tracker * Update beacon-chain.md * Revert G_t element optimization This needs its own ethresearch post, and some additional analysis to see if we can do the shuffle ZKP in the allowed timeframe. This reverts commit 8517acabfc1dbb1a35789e6170b5db0bb2c19c9a. * Implement new shuffling strategy Ditch the Feistel logic and instead have each shuffler pick the row they shuffle using their RANDAO reveal. * Curdleproofs edits * working whisk eth2spec * working whisk dummy test * add more boilerplate set up code * rebase constants * Implement even newer and simplified shuffling strategy This commit further simplifies 0faef30fc131d1b471b63a7f16772eeeef548ef8 by removing the entire squareshuffle. The latest version of https://eprint.iacr.org/2022/560 proposes that each shuffler picks random indices from the entire candidate set instead of organizing validators into a square. * Move to _features * remove dummy test * Run doctoc * Change Whisk's previous fork to Capella instead of Bellatrix. Make linter happier. * Fix lint * Fix pylint * Fix mypy issues * Clean-up get_beacon_proposer_index * Fix doc headers * Fix capella link * Update apply_deposit * Rename process_shuffled_trackers --------- Co-authored-by: George Kadianakis Co-authored-by: Justin Co-authored-by: Nalin Bhardwaj Co-authored-by: Hsiao-Wei Wang --- .gitignore | 1 + Makefile | 2 +- setup.py | 39 ++- specs/_features/whisk/beacon-chain.md | 482 ++++++++++++++++++++++++++ specs/_features/whisk/fork.md | 138 ++++++++ 5 files changed, 656 insertions(+), 6 deletions(-) create mode 100644 specs/_features/whisk/beacon-chain.md create mode 100644 specs/_features/whisk/fork.md diff --git a/.gitignore b/.gitignore index 82026c27bd..c7012d51b3 100644 --- a/.gitignore +++ b/.gitignore @@ -21,6 +21,7 @@ tests/core/pyspec/eth2spec/bellatrix/ tests/core/pyspec/eth2spec/capella/ tests/core/pyspec/eth2spec/deneb/ tests/core/pyspec/eth2spec/eip6110/ +tests/core/pyspec/eth2spec/whisk/ # coverage reports .htmlcov diff --git a/Makefile b/Makefile index ab5521663a..5c25d59d3c 100644 --- a/Makefile +++ b/Makefile @@ -34,7 +34,7 @@ MARKDOWN_FILES = $(wildcard $(SPEC_DIR)/*/*.md) \ $(wildcard $(SPEC_DIR)/_features/*/*/*.md) \ $(wildcard $(SSZ_DIR)/*.md) -ALL_EXECUTABLE_SPECS = phase0 altair bellatrix capella deneb eip6110 +ALL_EXECUTABLE_SPECS = phase0 altair bellatrix capella deneb eip6110 whisk # The parameters for commands. Use `foreach` to avoid listing specs again. COVERAGE_SCOPE := $(foreach S,$(ALL_EXECUTABLE_SPECS), --cov=eth2spec.$S.$(TEST_PRESET_TYPE)) PYLINT_SCOPE := $(foreach S,$(ALL_EXECUTABLE_SPECS), ./eth2spec/$S) diff --git a/setup.py b/setup.py index c9041010d1..f4d0c87197 100644 --- a/setup.py +++ b/setup.py @@ -48,6 +48,7 @@ def installPackage(package: str): CAPELLA = 'capella' DENEB = 'deneb' EIP6110 = 'eip6110' +WHISK = 'whisk' # The helper functions that are used when defining constants @@ -733,9 +734,31 @@ def imports(cls, preset_name: str): ''' +# +# WhiskSpecBuilder +# +class WhiskSpecBuilder(CapellaSpecBuilder): + fork: str = WHISK + + @classmethod + def imports(cls, preset_name: str): + return super().imports(preset_name) + f''' +from eth2spec.capella import {preset_name} as capella +''' + + @classmethod + def hardcoded_custom_type_dep_constants(cls, spec_object) -> str: + # Necessary for custom types `WhiskShuffleProof` and `WhiskTrackerProof` + constants = { + 'WHISK_MAX_SHUFFLE_PROOF_SIZE': spec_object.constant_vars['WHISK_MAX_SHUFFLE_PROOF_SIZE'].value, + 'WHISK_MAX_OPENING_PROOF_SIZE': spec_object.constant_vars['WHISK_MAX_OPENING_PROOF_SIZE'].value, + } + return {**super().hardcoded_custom_type_dep_constants(spec_object), **constants} + + spec_builders = { builder.fork: builder - for builder in (Phase0SpecBuilder, AltairSpecBuilder, BellatrixSpecBuilder, CapellaSpecBuilder, DenebSpecBuilder, EIP6110SpecBuilder) + for builder in (Phase0SpecBuilder, AltairSpecBuilder, BellatrixSpecBuilder, CapellaSpecBuilder, DenebSpecBuilder, EIP6110SpecBuilder, WhiskSpecBuilder) } @@ -1045,7 +1068,7 @@ def finalize_options(self): if len(self.md_doc_paths) == 0: print("no paths were specified, using default markdown file paths for pyspec" " build (spec fork: %s)" % self.spec_fork) - if self.spec_fork in (PHASE0, ALTAIR, BELLATRIX, CAPELLA, DENEB, EIP6110): + if self.spec_fork in (PHASE0, ALTAIR, BELLATRIX, CAPELLA, DENEB, EIP6110, WHISK): self.md_doc_paths = """ specs/phase0/beacon-chain.md specs/phase0/fork-choice.md @@ -1053,7 +1076,7 @@ def finalize_options(self): specs/phase0/weak-subjectivity.md specs/phase0/p2p-interface.md """ - if self.spec_fork in (ALTAIR, BELLATRIX, CAPELLA, DENEB, EIP6110): + if self.spec_fork in (ALTAIR, BELLATRIX, CAPELLA, DENEB, EIP6110, WHISK): self.md_doc_paths += """ specs/altair/light-client/full-node.md specs/altair/light-client/light-client.md @@ -1065,7 +1088,7 @@ def finalize_options(self): specs/altair/validator.md specs/altair/p2p-interface.md """ - if self.spec_fork in (BELLATRIX, CAPELLA, DENEB, EIP6110): + if self.spec_fork in (BELLATRIX, CAPELLA, DENEB, EIP6110, WHISK): self.md_doc_paths += """ specs/bellatrix/beacon-chain.md specs/bellatrix/fork.md @@ -1074,7 +1097,7 @@ def finalize_options(self): specs/bellatrix/p2p-interface.md sync/optimistic.md """ - if self.spec_fork in (CAPELLA, DENEB, EIP6110): + if self.spec_fork in (CAPELLA, DENEB, EIP6110, WHISK): self.md_doc_paths += """ specs/capella/light-client/fork.md specs/capella/light-client/full-node.md @@ -1104,6 +1127,11 @@ def finalize_options(self): specs/_features/eip6110/beacon-chain.md specs/_features/eip6110/fork.md """ + if self.spec_fork == WHISK: + self.md_doc_paths += """ + specs/_features/whisk/beacon-chain.md + specs/_features/whisk/fork.md + """ if len(self.md_doc_paths) == 0: raise Exception('no markdown files specified, and spec fork "%s" is unknown', self.spec_fork) @@ -1259,5 +1287,6 @@ def run(self): "lru-dict==1.1.8", MARKO_VERSION, "py_arkworks_bls12381==0.3.4", + "curdleproofs @ git+https://github.com/nalinbhardwaj/curdleproofs.pie@master#egg=curdleproofs&subdirectory=curdleproofs", ] ) diff --git a/specs/_features/whisk/beacon-chain.md b/specs/_features/whisk/beacon-chain.md new file mode 100644 index 0000000000..7d2ee078fe --- /dev/null +++ b/specs/_features/whisk/beacon-chain.md @@ -0,0 +1,482 @@ +# Whisk -- The Beacon Chain + +**Notice**: This document is a work-in-progress for researchers and implementers. + +## Table of contents + + + + + +- [Introduction](#introduction) +- [Constants](#constants) +- [Cryptography](#cryptography) + - [BLS](#bls) + - [Curdleproofs and opening proofs](#curdleproofs-and-opening-proofs) +- [Epoch processing](#epoch-processing) + - [`WhiskTracker`](#whisktracker) + - [`Validator`](#validator) + - [`BeaconState`](#beaconstate) +- [Block processing](#block-processing) + - [Block header](#block-header) + - [`BeaconBlock`](#beaconblock) + - [Whisk](#whisk) + - [`BeaconBlockBody`](#beaconblockbody) + - [Deposits](#deposits) + - [`get_beacon_proposer_index`](#get_beacon_proposer_index) + + + + +## Introduction + +This document details the beacon chain additions and changes of to support the Whisk SSLE, + +*Note:* This specification is built upon [Capella](../../capella/beacon-chain.md) and is under active development. + +## Constants + +| Name | Value | Description | +| ---------------------------------- | -------------------------- | ----------------------------------------------------------- | +| `WHISK_CANDIDATE_TRACKERS_COUNT` | `uint64(2**14)` (= 16,384) | number of candidate trackers | +| `WHISK_PROPOSER_TRACKERS_COUNT` | `uint64(2**13)` (= 8,192) | number of proposer trackers | +| `WHISK_EPOCHS_PER_SHUFFLING_PHASE` | `Epoch(2**8)` (= 256) | epochs per shuffling phase | +| `WHISK_VALIDATORS_PER_SHUFFLE` | `uint64(2**7)` (= 128) | number of validators shuffled per shuffle step | +| `WHISK_PROPOSER_SELECTION_GAP` | `Epoch(2)` | gap between proposer selection and the block proposal phase | +| `WHISK_MAX_SHUFFLE_PROOF_SIZE` | `uint64(2**15)` | max size of a shuffle proof | +| `WHISK_MAX_OPENING_PROOF_SIZE` | `uint64(2**10)` | max size of a opening proof | + +| Name | Value | +| ---------------------------------- | -------------------------- | +| `DOMAIN_WHISK_CANDIDATE_SELECTION` | `DomainType('0x07000000')` | +| `DOMAIN_WHISK_SHUFFLE` | `DomainType('0x07100000')` | +| `DOMAIN_WHISK_PROPOSER_SELECTION` | `DomainType('0x07200000')` | + +## Cryptography + +### BLS + +| Name | SSZ equivalent | Description | +| ------------------- | ---------------------------------------- | ----------------------------- | +| `BLSFieldElement` | `uint256` | BLS12-381 scalar | +| `BLSG1Point` | `Bytes48` | compressed BLS12-381 G1 point | +| `WhiskShuffleProof` | `ByteList[WHISK_MAX_SHUFFLE_PROOF_SIZE]` | Serialized shuffle proof | +| `WhiskTrackerProof` | `ByteList[WHISK_MAX_OPENING_PROOF_SIZE]` | Serialized tracker proof | + +*Note*: A subgroup check MUST be performed when deserializing a `BLSG1Point` for use in any of the functions below. + +```python +def BLSG1ScalarMultiply(scalar: BLSFieldElement, point: BLSG1Point) -> BLSG1Point: + return bls.G1_to_bytes48(bls.multiply(bls.bytes48_to_G1(point), scalar)) +``` + +```python +def bytes_to_bls_field(b: Bytes32) -> BLSFieldElement: + """ + Convert bytes to a BLS field scalar. The output is not uniform over the BLS field. + TODO: Deneb will introduces this helper too. Should delete it once it's rebased to post-Deneb. + """ + field_element = int.from_bytes(b, ENDIANNESS) + assert field_element < BLS_MODULUS + return BLSFieldElement(field_element) +``` + +| Name | Value | +| ------------------ | ------------------------------------------------------------------------------- | +| `BLS_G1_GENERATOR` | `bls.G1_to_bytes48(bls.G1)` | +| `BLS_MODULUS` | `52435875175126190479447740508185965837690552500527637822603658699938581184513` | + +### Curdleproofs and opening proofs + +Note that Curdleproofs (Whisk Shuffle Proofs), the tracker opening proofs and all related data structures and verifier code (along with tests) is specified in [curdleproofs.pie](https://github.com/nalinbhardwaj/curdleproofs.pie/tree/verifier-only) repository. + +```python +def IsValidWhiskShuffleProof(pre_shuffle_trackers: Sequence[WhiskTracker], + post_shuffle_trackers: Sequence[WhiskTracker], + M: BLSG1Point, + shuffle_proof: WhiskShuffleProof) -> bool: + """ + Verify `post_shuffle_trackers` is a permutation of `pre_shuffle_trackers`. + Defined in https://github.com/nalinbhardwaj/curdleproofs.pie/tree/verifier-only. + """ + # pylint: disable=unused-argument + return True +``` + +```python +def IsValidWhiskOpeningProof(tracker: WhiskTracker, + k_commitment: BLSG1Point, + tracker_proof: WhiskTrackerProof) -> bool: + """ + Verify knowledge of `k` such that `tracker.k_r_G == k * tracker.r_G` and `k_commitment == k * BLS_G1_GENERATOR`. + Defined in https://github.com/nalinbhardwaj/curdleproofs.pie/tree/verifier-only. + """ + # pylint: disable=unused-argument + return True +``` + +## Epoch processing + +### `WhiskTracker` + +```python +class WhiskTracker(Container): + r_G: BLSG1Point # r * G + k_r_G: BLSG1Point # k * r * G +``` + +### `Validator` + +```python +class Validator(Container): + pubkey: BLSPubkey + withdrawal_credentials: Bytes32 # Commitment to pubkey for withdrawals + effective_balance: Gwei # Balance at stake + slashed: boolean + # Status epochs + activation_eligibility_epoch: Epoch # When criteria for activation were met + activation_epoch: Epoch + exit_epoch: Epoch + withdrawable_epoch: Epoch # When validator can withdraw funds + whisk_tracker: WhiskTracker # Whisk tracker (r * G, k * r * G) [New in Whisk] + whisk_k_commitment: BLSG1Point # Whisk k commitment k * BLS_G1_GENERATOR [New in Whisk] +``` + +### `BeaconState` + +```python +class BeaconState(Container): + # Versioning + genesis_time: uint64 + genesis_validators_root: Root + slot: Slot + fork: Fork + # History + latest_block_header: BeaconBlockHeader + block_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT] + state_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT] + historical_roots: List[Root, HISTORICAL_ROOTS_LIMIT] # Frozen in Capella, replaced by historical_summaries + # Eth1 + eth1_data: Eth1Data + eth1_data_votes: List[Eth1Data, EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH] + eth1_deposit_index: uint64 + # Registry + validators: List[Validator, VALIDATOR_REGISTRY_LIMIT] # [Modified in Whisk] + balances: List[Gwei, VALIDATOR_REGISTRY_LIMIT] + # Randomness + randao_mixes: Vector[Bytes32, EPOCHS_PER_HISTORICAL_VECTOR] + # Slashings + slashings: Vector[Gwei, EPOCHS_PER_SLASHINGS_VECTOR] # Per-epoch sums of slashed effective balances + # Participation + previous_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT] + current_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT] + # Finality + justification_bits: Bitvector[JUSTIFICATION_BITS_LENGTH] # Bit set for every recent justified epoch + previous_justified_checkpoint: Checkpoint + current_justified_checkpoint: Checkpoint + finalized_checkpoint: Checkpoint + # Inactivity + inactivity_scores: List[uint64, VALIDATOR_REGISTRY_LIMIT] + # Sync + current_sync_committee: SyncCommittee + next_sync_committee: SyncCommittee + # Execution + latest_execution_payload_header: ExecutionPayloadHeader + # Withdrawals + next_withdrawal_index: WithdrawalIndex + next_withdrawal_validator_index: ValidatorIndex + # Deep history valid from Capella onwards + historical_summaries: List[HistoricalSummary, HISTORICAL_ROOTS_LIMIT] + whisk_candidate_trackers: Vector[WhiskTracker, WHISK_CANDIDATE_TRACKERS_COUNT] # [New in Whisk] + whisk_proposer_trackers: Vector[WhiskTracker, WHISK_PROPOSER_TRACKERS_COUNT] # [New in Whisk] +``` + +```python +def select_whisk_trackers(state: BeaconState, epoch: Epoch) -> None: + # Select proposer trackers from candidate trackers + proposer_seed = get_seed(state, epoch - WHISK_PROPOSER_SELECTION_GAP, DOMAIN_WHISK_PROPOSER_SELECTION) + for i in range(WHISK_PROPOSER_TRACKERS_COUNT): + index = compute_shuffled_index(uint64(i), uint64(len(state.whisk_candidate_trackers)), proposer_seed) + state.whisk_proposer_trackers[i] = state.whisk_candidate_trackers[index] + + # Select candidate trackers from active validator trackers + active_validator_indices = get_active_validator_indices(state, epoch) + for i in range(WHISK_CANDIDATE_TRACKERS_COUNT): + seed = hash(get_seed(state, epoch, DOMAIN_WHISK_CANDIDATE_SELECTION) + uint_to_bytes(i)) + candidate_index = compute_proposer_index(state, active_validator_indices, seed) # sample by effective balance + state.whisk_candidate_trackers[i] = state.validators[candidate_index].whisk_tracker +``` + +```python +def process_whisk_updates(state: BeaconState) -> None: + next_epoch = Epoch(get_current_epoch(state) + 1) + if next_epoch % WHISK_EPOCHS_PER_SHUFFLING_PHASE == 0: # select trackers at the start of shuffling phases + select_whisk_trackers(state, next_epoch) +``` + +```python +def process_epoch(state: BeaconState) -> None: + process_justification_and_finalization(state) + process_inactivity_updates(state) + process_rewards_and_penalties(state) + process_registry_updates(state) + process_slashings(state) + process_eth1_data_reset(state) + process_effective_balance_updates(state) + process_slashings_reset(state) + process_randao_mixes_reset(state) + process_historical_summaries_update(state) + process_participation_flag_updates(state) + process_sync_committee_updates(state) + process_whisk_updates(state) # [New in Whisk] +``` + +## Block processing + +### Block header + +#### `BeaconBlock` + +```python +class BeaconBlock(Container): + slot: Slot + proposer_index: ValidatorIndex + parent_root: Root + state_root: Root + body: BeaconBlockBody + whisk_opening_proof: WhiskTrackerProof # [New in Whisk] +``` + +```python +def process_whisk_opening_proof(state: BeaconState, block: BeaconBlock) -> None: + tracker = state.whisk_proposer_trackers[state.slot % WHISK_PROPOSER_TRACKERS_COUNT] + k_commitment = state.validators[block.proposer_index].whisk_k_commitment + assert IsValidWhiskOpeningProof(tracker, k_commitment, block.whisk_opening_proof) +``` + +Removed `assert block.proposer_index == get_beacon_proposer_index(state)` check in Whisk. + +```python +def process_block_header(state: BeaconState, block: BeaconBlock) -> None: + # Verify that the slots match + assert block.slot == state.slot + # Verify that the block is newer than latest block header + assert block.slot > state.latest_block_header.slot + + # # Verify that proposer index is the correct index + # assert block.proposer_index == get_beacon_proposer_index(state) + + # Verify that the parent matches + assert block.parent_root == hash_tree_root(state.latest_block_header) + # Cache current block as the new latest block + state.latest_block_header = BeaconBlockHeader( + slot=block.slot, + proposer_index=block.proposer_index, + parent_root=block.parent_root, + state_root=Bytes32(), # Overwritten in the next process_slot call + body_root=hash_tree_root(block.body), + ) + + # Verify proposer is not slashed + proposer = state.validators[block.proposer_index] + assert not proposer.slashed + process_whisk_opening_proof(state, block) # [New in Whisk] +``` + +### Whisk + +#### `BeaconBlockBody` + +```python +class BeaconBlockBody(capella.BeaconBlockBody): + randao_reveal: BLSSignature + eth1_data: Eth1Data # Eth1 data vote + graffiti: Bytes32 # Arbitrary data + # Operations + proposer_slashings: List[ProposerSlashing, MAX_PROPOSER_SLASHINGS] + attester_slashings: List[AttesterSlashing, MAX_ATTESTER_SLASHINGS] + attestations: List[Attestation, MAX_ATTESTATIONS] + deposits: List[Deposit, MAX_DEPOSITS] + voluntary_exits: List[SignedVoluntaryExit, MAX_VOLUNTARY_EXITS] + sync_aggregate: SyncAggregate + # Execution + execution_payload: ExecutionPayload + # Capella operations + bls_to_execution_changes: List[SignedBLSToExecutionChange, MAX_BLS_TO_EXECUTION_CHANGES] + whisk_post_shuffle_trackers: Vector[WhiskTracker, WHISK_VALIDATORS_PER_SHUFFLE] # [New in Whisk] + whisk_shuffle_proof: WhiskShuffleProof # [New in Whisk] + whisk_shuffle_proof_M_commitment: BLSG1Point # [New in Whisk] + whisk_registration_proof: WhiskTrackerProof # [New in Whisk] + whisk_tracker: WhiskTracker # [New in Whisk] + whisk_k_commitment: BLSG1Point # [New in Whisk] +``` + +```python +def get_shuffle_indices(randao_reveal: BLSSignature) -> Sequence[uint64]: + """ + Given a `randao_reveal` return the list of indices that got shuffled from the entire candidate set + """ + indices = [] + for i in WHISK_VALIDATORS_PER_SHUFFLE: + # XXX ensure we are not suffering from modulo bias + shuffle_index = uint256(hash(randao_reveal + uint_to_bytes(i))) % WHISK_CANDIDATE_TRACKERS_COUNT + indices.append(shuffle_index) + + return indices +``` + +```python +def process_shuffled_trackers(state: BeaconState, body: BeaconBlockBody) -> None: + # Check the shuffle proof + shuffle_indices = get_shuffle_indices(body.randao_reveal) + pre_shuffle_trackers = [state.whisk_candidate_trackers[i] for i in shuffle_indices] + post_shuffle_trackers = body.whisk_post_shuffle_trackers + + shuffle_epoch = get_current_epoch(state) % WHISK_EPOCHS_PER_SHUFFLING_PHASE + if shuffle_epoch + WHISK_PROPOSER_SELECTION_GAP + 1 >= WHISK_EPOCHS_PER_SHUFFLING_PHASE: + # Require unchanged trackers during cooldown + assert pre_shuffle_trackers == post_shuffle_trackers + else: + # Require shuffled trackers during shuffle + assert IsValidWhiskShuffleProof( + pre_shuffle_trackers, + post_shuffle_trackers, + body.whisk_shuffle_proof_M_commitment, + body.whisk_shuffle_proof, + ) + + # Shuffle candidate trackers + for i, shuffle_index in enumerate(shuffle_indices): + state.whisk_candidate_trackers[shuffle_index] = post_shuffle_trackers[i] +``` + +```python +def is_k_commitment_unique(state: BeaconState, k_commitment: BLSG1Point) -> bool: + return all([validator.whisk_k_commitment != k_commitment for validator in state.validators]) +``` + +```python +def process_whisk(state: BeaconState, body: BeaconBlockBody) -> None: + process_shuffled_trackers(state, body) + + # Overwrite all validator Whisk fields (first Whisk proposal) or just the permutation commitment (next proposals) + proposer = state.validators[get_beacon_proposer_index(state)] + if proposer.whisk_tracker.r_G == BLS_G1_GENERATOR: # first Whisk proposal + assert body.whisk_tracker.r_G != BLS_G1_GENERATOR + assert is_k_commitment_unique(state, body.whisk_k_commitment) + assert IsValidWhiskOpeningProof( + body.whisk_tracker, + body.whisk_k_commitment, + body.whisk_registration_proof, + ) + proposer.whisk_tracker = body.whisk_tracker + proposer.whisk_k_commitment = body.whisk_k_commitment + else: # next Whisk proposals + assert body.whisk_registration_proof == WhiskTrackerProof() + assert body.whisk_tracker == WhiskTracker() + assert body.whisk_k_commitment == BLSG1Point() + assert body.whisk_shuffle_proof_M_commitment == BLSG1Point() +``` + +```python +def process_block(state: BeaconState, block: BeaconBlock) -> None: + process_block_header(state, block) + if is_execution_enabled(state, block.body): + process_withdrawals(state, block.body.execution_payload) + process_execution_payload(state, block.body.execution_payload, EXECUTION_ENGINE) + process_randao(state, block.body) + process_eth1_data(state, block.body) + process_operations(state, block.body) + process_sync_aggregate(state, block.body.sync_aggregate) + process_whisk(state, block.body) # [New in Whisk] +``` + +### Deposits + +```python +def get_unique_whisk_k(state: BeaconState, validator_index: ValidatorIndex) -> BLSFieldElement: + counter = 0 + while True: + # hash `validator_index || counter` + k = BLSFieldElement(bytes_to_bls_field(hash(uint_to_bytes(validator_index) + uint_to_bytes(uint64(counter))))) + if is_k_commitment_unique(state, BLSG1ScalarMultiply(k, BLS_G1_GENERATOR)): + return k # unique by trial and error + counter += 1 +``` + +```python +def get_initial_commitments(k: BLSFieldElement) -> Tuple[BLSG1Point, WhiskTracker]: + return ( + BLSG1ScalarMultiply(k, BLS_G1_GENERATOR), + WhiskTracker(r_G=BLS_G1_GENERATOR, k_r_G=BLSG1ScalarMultiply(k, BLS_G1_GENERATOR)) + ) +``` + +```python +def get_validator_from_deposit_whisk( + state: BeaconState, + pubkey: BLSPubkey, + withdrawal_credentials: Bytes32, + amount: uint64 +) -> Validator: + effective_balance = min(amount - amount % EFFECTIVE_BALANCE_INCREMENT, MAX_EFFECTIVE_BALANCE) + k = get_unique_whisk_k(state, ValidatorIndex(len(state.validators))) + whisk_k_commitment, whisk_tracker = get_initial_commitments(k) + + validator = Validator( + pubkey=pubkey, + withdrawal_credentials=withdrawal_credentials, + activation_eligibility_epoch=FAR_FUTURE_EPOCH, + activation_epoch=FAR_FUTURE_EPOCH, + exit_epoch=FAR_FUTURE_EPOCH, + withdrawable_epoch=FAR_FUTURE_EPOCH, + effective_balance=effective_balance, + # Whisk fields + whisk_tracker=whisk_tracker, + whisk_k_commitment=whisk_k_commitment, + ) + return validator +``` + +```python +def apply_deposit(state: BeaconState, + pubkey: BLSPubkey, + withdrawal_credentials: Bytes32, + amount: uint64, + signature: BLSSignature) -> None: + validator_pubkeys = [validator.pubkey for validator in state.validators] + if pubkey not in validator_pubkeys: + # Verify the deposit signature (proof of possession) which is not checked by the deposit contract + deposit_message = DepositMessage( + pubkey=pubkey, + withdrawal_credentials=withdrawal_credentials, + amount=amount, + ) + domain = compute_domain(DOMAIN_DEPOSIT) # Fork-agnostic domain since deposits are valid across forks + signing_root = compute_signing_root(deposit_message, domain) + # Initialize validator if the deposit signature is valid + if bls.Verify(pubkey, signing_root, signature): + index = get_index_for_new_validator(state) + validator = get_validator_from_deposit_whisk(state, pubkey, withdrawal_credentials, amount) + set_or_append_list(state.validators, index, validator) + set_or_append_list(state.balances, index, amount) + # [New in Altair] + set_or_append_list(state.previous_epoch_participation, index, ParticipationFlags(0b0000_0000)) + set_or_append_list(state.current_epoch_participation, index, ParticipationFlags(0b0000_0000)) + set_or_append_list(state.inactivity_scores, index, uint64(0)) + else: + # Increase balance by deposit amount + index = ValidatorIndex(validator_pubkeys.index(pubkey)) + increase_balance(state, index, amount) +``` + +### `get_beacon_proposer_index` + +```python +def get_beacon_proposer_index(state: BeaconState) -> ValidatorIndex: + """ + Return the beacon proposer index at the current slot. + """ + assert state.latest_block_header.slot == state.slot # sanity check `process_block_header` has been called + return state.latest_block_header.proposer_index +``` diff --git a/specs/_features/whisk/fork.md b/specs/_features/whisk/fork.md new file mode 100644 index 0000000000..189d8c5b32 --- /dev/null +++ b/specs/_features/whisk/fork.md @@ -0,0 +1,138 @@ +# Whisk -- Fork Logic + +**Notice**: This document is a work-in-progress for researchers and implementers. + +## Table of contents + + + + + +- [Introduction](#introduction) +- [Configuration](#configuration) +- [Fork to Whisk](#fork-to-whisk) + + + + +## Introduction + +This document describes the process of Whisk upgrade. + + +``` +""" + WHISK_FORK_EPOCH + | cooldown + | | || + v vsvv + --+~~~~~~~~~~~~~~~~~~~~~----+- + shuffling ^ + | + | + proposer selection + candidate selection +""" +``` + +## Configuration + +Warning: this configuration is not definitive. + +| Name | Value | +| -------------------- | ----------------------- | +| `WHISK_FORK_VERSION` | `Version('0x05000000')` | +| `WHISK_FORK_EPOCH` | `Epoch(18446744073709551615)` **TBD** | + +## Fork to Whisk + +If `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == WHISK_FORK_EPOCH`, an irregular state change is made to upgrade to Whisk. `WHISK_FORK_EPOCH` must be a multiple of `WHISK_RUN_DURATION_IN_EPOCHS`. + +The upgrade occurs after the completion of the inner loop of `process_slots` that sets `state.slot` equal to `WHISK_FORK_EPOCH * SLOTS_PER_EPOCH`. + +This ensures that we drop right into the beginning of the shuffling phase but without `process_whisk_epoch()` triggering for this Whisk run. Hence we handle all the setup ourselves in `upgrade_to_whisk()` below. + +```python +def whisk_candidate_selection(state: BeaconState, epoch: Epoch) -> None: + # TODO + # pylint: disable=unused-argument + pass +``` + +```python +def whisk_proposer_selection(state: BeaconState, epoch: Epoch) -> None: + # TODO + # pylint: disable=unused-argument + pass +``` + +```python +def upgrade_to_whisk(pre: bellatrix.BeaconState) -> BeaconState: + epoch = bellatrix.get_current_epoch(pre) + post = BeaconState( + # Versioning + genesis_time=pre.genesis_time, + genesis_validators_root=pre.genesis_validators_root, + slot=pre.slot, + fork=Fork( + previous_version=pre.fork.current_version, + current_version=WHISK_FORK_VERSION, + epoch=epoch, + ), + # History + latest_block_header=pre.latest_block_header, + block_roots=pre.block_roots, + state_roots=pre.state_roots, + historical_roots=pre.historical_roots, + # Eth1 + eth1_data=pre.eth1_data, + eth1_data_votes=pre.eth1_data_votes, + eth1_deposit_index=pre.eth1_deposit_index, + # Registry + validators=[], + balances=pre.balances, + # Randomness + randao_mixes=pre.randao_mixes, + # Slashings + slashings=pre.slashings, + # Participation + previous_epoch_participation=pre.previous_epoch_participation, + current_epoch_participation=pre.current_epoch_participation, + # Finality + justification_bits=pre.justification_bits, + previous_justified_checkpoint=pre.previous_justified_checkpoint, + current_justified_checkpoint=pre.current_justified_checkpoint, + finalized_checkpoint=pre.finalized_checkpoint, + # Inactivity + inactivity_scores=pre.inactivity_Scores, + ) + + # Initialize all validators with predictable commitments + for val_index, pre_validator in enumerate(pre.validators): + whisk_commitment, whisk_tracker = get_initial_commitments(get_unique_whisk_k(post, ValidatorIndex(val_index))) + + post_validator = Validator( + pubkey=pre_validator.pubkey, + withdrawal_credentials=pre_validator.withdrawal_credentials, + effective_balance=pre_validator.effective_balance, + slashed=pre_validator.slashed, + activation_eligibility_epoch=pre_validator.activation_eligibility_epoch, + activation_epoch=pre_validator.activation_epoch, + exit_epoch=pre_validator.exit_epoch, + withdrawable_epoch=pre_validator.withdrawable_epoch, + whisk_commitment=whisk_commitment, + whisk_tracker=whisk_tracker, + ) + post.validators.append(post_validator) + + # Do a candidate selection followed by a proposer selection so that we have proposers for the upcoming day + # Use an old epoch when selecting candidates so that we don't get the same seed as in the next candidate selection + whisk_candidate_selection(post, epoch - WHISK_PROPOSER_SELECTION_GAP - 1) + whisk_proposer_selection(post, epoch) + + # Do a final round of candidate selection. + # We need it so that we have something to shuffle over the upcoming shuffling phase. + whisk_candidate_selection(post, epoch) + + return post +``` From 68bb5ee5079ababf7be4cf7b6a1e0a9cada09f43 Mon Sep 17 00:00:00 2001 From: dapplion <35266934+dapplion@users.noreply.github.com> Date: Thu, 8 Jun 2023 15:00:49 +0300 Subject: [PATCH 06/10] Ensure make lint resolves to the right pylint and mypy versions --- Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 5c25d59d3c..6c852a1e94 100644 --- a/Makefile +++ b/Makefile @@ -146,8 +146,8 @@ codespell: lint: pyspec . venv/bin/activate; cd $(PY_SPEC_DIR); \ flake8 --config $(LINTER_CONFIG_FILE) ./eth2spec \ - && pylint --rcfile $(LINTER_CONFIG_FILE) $(PYLINT_SCOPE) \ - && mypy --config-file $(LINTER_CONFIG_FILE) $(MYPY_SCOPE) + && python -m pylint --rcfile $(LINTER_CONFIG_FILE) $(PYLINT_SCOPE) \ + && python -m mypy --config-file $(LINTER_CONFIG_FILE) $(MYPY_SCOPE) lint_generators: pyspec . venv/bin/activate; cd $(TEST_GENERATORS_DIR); \ From efd2429b3f14f7bccaad342a75fdf843b21c0951 Mon Sep 17 00:00:00 2001 From: Lion - dapplion <35266934+dapplion@users.noreply.github.com> Date: Fri, 9 Jun 2023 11:28:06 +0300 Subject: [PATCH 07/10] Move whisk_opening_proof to block body (#3408) --- specs/_features/whisk/beacon-chain.md | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/specs/_features/whisk/beacon-chain.md b/specs/_features/whisk/beacon-chain.md index 7d2ee078fe..bad8188378 100644 --- a/specs/_features/whisk/beacon-chain.md +++ b/specs/_features/whisk/beacon-chain.md @@ -19,7 +19,6 @@ - [`BeaconState`](#beaconstate) - [Block processing](#block-processing) - [Block header](#block-header) - - [`BeaconBlock`](#beaconblock) - [Whisk](#whisk) - [`BeaconBlockBody`](#beaconblockbody) - [Deposits](#deposits) @@ -235,23 +234,11 @@ def process_epoch(state: BeaconState) -> None: ### Block header -#### `BeaconBlock` - -```python -class BeaconBlock(Container): - slot: Slot - proposer_index: ValidatorIndex - parent_root: Root - state_root: Root - body: BeaconBlockBody - whisk_opening_proof: WhiskTrackerProof # [New in Whisk] -``` - ```python def process_whisk_opening_proof(state: BeaconState, block: BeaconBlock) -> None: tracker = state.whisk_proposer_trackers[state.slot % WHISK_PROPOSER_TRACKERS_COUNT] k_commitment = state.validators[block.proposer_index].whisk_k_commitment - assert IsValidWhiskOpeningProof(tracker, k_commitment, block.whisk_opening_proof) + assert IsValidWhiskOpeningProof(tracker, k_commitment, block.body.whisk_opening_proof) ``` Removed `assert block.proposer_index == get_beacon_proposer_index(state)` check in Whisk. @@ -303,6 +290,8 @@ class BeaconBlockBody(capella.BeaconBlockBody): execution_payload: ExecutionPayload # Capella operations bls_to_execution_changes: List[SignedBLSToExecutionChange, MAX_BLS_TO_EXECUTION_CHANGES] + # Whisk + whisk_opening_proof: WhiskTrackerProof # [New in Whisk] whisk_post_shuffle_trackers: Vector[WhiskTracker, WHISK_VALIDATORS_PER_SHUFFLE] # [New in Whisk] whisk_shuffle_proof: WhiskShuffleProof # [New in Whisk] whisk_shuffle_proof_M_commitment: BLSG1Point # [New in Whisk] From 7b69f176e2e836ccff5b58f655b39b9c583f00cd Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 9 Jun 2023 08:54:16 -0600 Subject: [PATCH 08/10] bump VERSION.txt --- tests/core/pyspec/eth2spec/VERSION.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/core/pyspec/eth2spec/VERSION.txt b/tests/core/pyspec/eth2spec/VERSION.txt index 2a6fa03c08..54dd11e46b 100644 --- a/tests/core/pyspec/eth2spec/VERSION.txt +++ b/tests/core/pyspec/eth2spec/VERSION.txt @@ -1 +1 @@ -1.4.0-alpha.1 +1.4.0-alpha.2 From 45f3432019336b5cb1196037e6f5d92a6dc5aeba Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 9 Jun 2023 23:10:00 +0800 Subject: [PATCH 09/10] Update `MAX_BLOBS_PER_BLOCK` to `6` and add Deneb networking configs to yaml files (#3410) --- configs/mainnet.yaml | 8 ++++++++ configs/minimal.yaml | 8 ++++++++ presets/mainnet/deneb.yaml | 4 ++-- presets/minimal/deneb.yaml | 4 ++-- specs/deneb/beacon-chain.md | 2 +- specs/deneb/fork-choice.md | 3 ++- .../test/deneb/unittests/test_config_invariants.py | 8 ++++++++ 7 files changed, 31 insertions(+), 6 deletions(-) diff --git a/configs/mainnet.yaml b/configs/mainnet.yaml index baf489739e..80a240185b 100644 --- a/configs/mainnet.yaml +++ b/configs/mainnet.yaml @@ -124,3 +124,11 @@ ATTESTATION_SUBNET_COUNT: 64 ATTESTATION_SUBNET_EXTRA_BITS: 0 # ceillog2(ATTESTATION_SUBNET_COUNT) + ATTESTATION_SUBNET_EXTRA_BITS ATTESTATION_SUBNET_PREFIX_BITS: 6 + +# Deneb +# `2**7` (=128) +MAX_REQUEST_BLOCKS_DENEB: 128 +# MAX_REQUEST_BLOCKS_DENEB * MAX_BLOBS_PER_BLOCK +MAX_REQUEST_BLOB_SIDECARS: 768 +# `2**12` (= 4096 epochs, ~18 days) +MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 4096 diff --git a/configs/minimal.yaml b/configs/minimal.yaml index 43f1fc83fa..82a813a9b1 100644 --- a/configs/minimal.yaml +++ b/configs/minimal.yaml @@ -125,3 +125,11 @@ ATTESTATION_SUBNET_COUNT: 64 ATTESTATION_SUBNET_EXTRA_BITS: 0 # ceillog2(ATTESTATION_SUBNET_COUNT) + ATTESTATION_SUBNET_EXTRA_BITS ATTESTATION_SUBNET_PREFIX_BITS: 6 + +# Deneb +# `2**7` (=128) +MAX_REQUEST_BLOCKS_DENEB: 128 +# MAX_REQUEST_BLOCKS_DENEB * MAX_BLOBS_PER_BLOCK +MAX_REQUEST_BLOB_SIDECARS: 768 +# `2**12` (= 4096 epochs, ~18 days) +MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 4096 diff --git a/presets/mainnet/deneb.yaml b/presets/mainnet/deneb.yaml index 10c5025ed8..23889fd18e 100644 --- a/presets/mainnet/deneb.yaml +++ b/presets/mainnet/deneb.yaml @@ -6,5 +6,5 @@ FIELD_ELEMENTS_PER_BLOB: 4096 # `uint64(2**12)` (= 4096) MAX_BLOB_COMMITMENTS_PER_BLOCK: 4096 -# `uint64(2**2)` (= 4) -MAX_BLOBS_PER_BLOCK: 4 +# `uint64(6)` +MAX_BLOBS_PER_BLOCK: 6 diff --git a/presets/minimal/deneb.yaml b/presets/minimal/deneb.yaml index 91120f9dae..e21d387773 100644 --- a/presets/minimal/deneb.yaml +++ b/presets/minimal/deneb.yaml @@ -6,5 +6,5 @@ FIELD_ELEMENTS_PER_BLOB: 4 # [customized] MAX_BLOB_COMMITMENTS_PER_BLOCK: 16 -# `uint64(2**2)` (= 4) -MAX_BLOBS_PER_BLOCK: 4 +# `uint64(6)` +MAX_BLOBS_PER_BLOCK: 6 diff --git a/specs/deneb/beacon-chain.md b/specs/deneb/beacon-chain.md index 41ce76bd2f..148dc96d8f 100644 --- a/specs/deneb/beacon-chain.md +++ b/specs/deneb/beacon-chain.md @@ -73,7 +73,7 @@ Deneb is a consensus-layer upgrade containing a number of features. Including: | Name | Value | Description | | - | - | - | | `MAX_BLOB_COMMITMENTS_PER_BLOCK` | `uint64(2**12)` (= 4096) | *[New in Deneb:EIP4844]* hardfork independent fixed theoretical limit same as `LIMIT_BLOBS_PER_TX` (see EIP 4844) | -| `MAX_BLOBS_PER_BLOCK` | `uint64(2**2)` (= 4) | *[New in Deneb:EIP4844]* Maximum number of blobs in a single block limited by `MAX_BLOB_COMMITMENTS_PER_BLOCK` | +| `MAX_BLOBS_PER_BLOCK` | `uint64(6)` | *[New in Deneb:EIP4844]* maximum number of blobs in a single block limited by `MAX_BLOB_COMMITMENTS_PER_BLOCK` | *Note*: The blob transactions are packed into the execution payload by the EL/builder with their corresponding blobs being independently transmitted and are limited by `MAX_DATA_GAS_PER_BLOCK // DATA_GAS_PER_BLOB`. However the CL limit is independently defined by `MAX_BLOBS_PER_BLOCK`. diff --git a/specs/deneb/fork-choice.md b/specs/deneb/fork-choice.md index c679062594..bbc7fa0f89 100644 --- a/specs/deneb/fork-choice.md +++ b/specs/deneb/fork-choice.md @@ -36,7 +36,8 @@ The block MUST NOT be considered valid until all valid `Blob`s have been downloa def is_data_available(beacon_block_root: Root, blob_kzg_commitments: Sequence[KZGCommitment]) -> bool: # `retrieve_blobs_and_proofs` is implementation and context dependent # It returns all the blobs for the given block root, and raises an exception if not available - # Note: the p2p network does not guarantee sidecar retrieval outside of `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` + # Note: the p2p network does not guarantee sidecar retrieval outside of + # `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` blobs, proofs = retrieve_blobs_and_proofs(beacon_block_root) # For testing, `retrieve_blobs_and_proofs` returns ("TEST", "TEST"). diff --git a/tests/core/pyspec/eth2spec/test/deneb/unittests/test_config_invariants.py b/tests/core/pyspec/eth2spec/test/deneb/unittests/test_config_invariants.py index 13a54225ed..75955ed161 100644 --- a/tests/core/pyspec/eth2spec/test/deneb/unittests/test_config_invariants.py +++ b/tests/core/pyspec/eth2spec/test/deneb/unittests/test_config_invariants.py @@ -10,3 +10,11 @@ @single_phase def test_length(spec): assert spec.MAX_BLOBS_PER_BLOCK < spec.MAX_BLOB_COMMITMENTS_PER_BLOCK + + +@with_deneb_and_later +@spec_test +@single_phase +def test_networking(spec): + assert spec.MAX_BLOBS_PER_BLOCK < spec.MAX_BLOB_COMMITMENTS_PER_BLOCK + assert spec.config.MAX_REQUEST_BLOB_SIDECARS == spec.config.MAX_REQUEST_BLOCKS_DENEB * spec.MAX_BLOBS_PER_BLOCK From 5576d0e6850b0ddbffd1ec122841b35e31e66fee Mon Sep 17 00:00:00 2001 From: Paul Harris Date: Sat, 10 Jun 2023 01:12:38 +1000 Subject: [PATCH 10/10] Moved configuration into network preset instead of constants. (#3394) * Moved configuration into network preset instead of constants. Now that `MAX_CHUNK_SIZE` and `GOSSIP_MAX_SIZE` are in configuration, we no longer need separate constants to represent them in the spec when they change in Bellatrix. I've changed the usage, and put the values into the presets, but I'm not sure if I've updated the descriptions in the best way... This is following on from the work in #3375 where a number of constants got moved into configuration, so we no longer need these constants to be separately represented, they can simply be updated in presets. * Update presets/minimal/bellatrix.yaml Co-authored-by: Hsiao-Wei Wang * Update presets/mainnet/bellatrix.yaml Co-authored-by: Hsiao-Wei Wang * Moved preset items into the correct section and updated TOC. It looked like the items listed in configuration about the max size and chunk size were no longer needed since we're updating preset values now and the preset changes seem to only be listed in the changes at the top. * review feedback * hopefully correct this time! Moved the 2 fields from configs into presets completely as suggested. * WIP - changing back to being in config and updating the phase 0 value... I think this should be close but want to see what's outstanding. * fix intellij's formatting of table. * more fixes --------- Co-authored-by: Hsiao-Wei Wang --- configs/mainnet.yaml | 8 ++++---- configs/minimal.yaml | 8 ++++---- presets/mainnet/phase0.yaml | 2 +- presets/minimal/phase0.yaml | 2 +- specs/bellatrix/p2p-interface.md | 28 ++-------------------------- specs/phase0/p2p-interface.md | 4 ++-- 6 files changed, 14 insertions(+), 38 deletions(-) diff --git a/configs/mainnet.yaml b/configs/mainnet.yaml index 80a240185b..fe405cb38a 100644 --- a/configs/mainnet.yaml +++ b/configs/mainnet.yaml @@ -98,16 +98,16 @@ DEPOSIT_CONTRACT_ADDRESS: 0x00000000219ab540356cBB839Cbe05303d7705Fa # Networking # --------------------------------------------------------------- -# `2**20` (= 1048576, 1 MiB) -GOSSIP_MAX_SIZE: 1048576 +# `10 * 2**20` (= 10485760, 10 MiB) +GOSSIP_MAX_SIZE: 10485760 # `2**10` (= 1024) MAX_REQUEST_BLOCKS: 1024 # `2**8` (= 256) EPOCHS_PER_SUBNET_SUBSCRIPTION: 256 # `MIN_VALIDATOR_WITHDRAWABILITY_DELAY + CHURN_LIMIT_QUOTIENT // 2` (= 33024, ~5 months) MIN_EPOCHS_FOR_BLOCK_REQUESTS: 33024 -# `2**20` (=1048576, 1 MiB) -MAX_CHUNK_SIZE: 1048576 +# `10 * 2**20` (=10485760, 10 MiB) +MAX_CHUNK_SIZE: 10485760 # 5s TTFB_TIMEOUT: 5 # 10s diff --git a/configs/minimal.yaml b/configs/minimal.yaml index 82a813a9b1..dcc834963e 100644 --- a/configs/minimal.yaml +++ b/configs/minimal.yaml @@ -99,16 +99,16 @@ DEPOSIT_CONTRACT_ADDRESS: 0x1234567890123456789012345678901234567890 # Networking # --------------------------------------------------------------- -# `2**20` (= 1048576, 1 MiB) -GOSSIP_MAX_SIZE: 1048576 +# `10 * 2**20` (= 10485760, 10 MiB) +GOSSIP_MAX_SIZE: 10485760 # `2**10` (= 1024) MAX_REQUEST_BLOCKS: 1024 # `2**8` (= 256) EPOCHS_PER_SUBNET_SUBSCRIPTION: 256 # [customized] `MIN_VALIDATOR_WITHDRAWABILITY_DELAY + CHURN_LIMIT_QUOTIENT // 2` (= 272) MIN_EPOCHS_FOR_BLOCK_REQUESTS: 272 -# `2**20` (=1048576, 1 MiB) -MAX_CHUNK_SIZE: 1048576 +# `10 * 2**20` (=10485760, 10 MiB) +MAX_CHUNK_SIZE: 10485760 # 5s TTFB_TIMEOUT: 5 # 10s diff --git a/presets/mainnet/phase0.yaml b/presets/mainnet/phase0.yaml index 02bc96c8cd..00133ba369 100644 --- a/presets/mainnet/phase0.yaml +++ b/presets/mainnet/phase0.yaml @@ -85,4 +85,4 @@ MAX_ATTESTATIONS: 128 # 2**4 (= 16) MAX_DEPOSITS: 16 # 2**4 (= 16) -MAX_VOLUNTARY_EXITS: 16 +MAX_VOLUNTARY_EXITS: 16 \ No newline at end of file diff --git a/presets/minimal/phase0.yaml b/presets/minimal/phase0.yaml index e7028f5a42..d9a6a2b6c0 100644 --- a/presets/minimal/phase0.yaml +++ b/presets/minimal/phase0.yaml @@ -85,4 +85,4 @@ MAX_ATTESTATIONS: 128 # 2**4 (= 16) MAX_DEPOSITS: 16 # 2**4 (= 16) -MAX_VOLUNTARY_EXITS: 16 +MAX_VOLUNTARY_EXITS: 16 \ No newline at end of file diff --git a/specs/bellatrix/p2p-interface.md b/specs/bellatrix/p2p-interface.md index b8b3a11d6b..7d80d40a83 100644 --- a/specs/bellatrix/p2p-interface.md +++ b/specs/bellatrix/p2p-interface.md @@ -14,7 +14,6 @@ Readers should understand the Phase 0 and Altair documents and use them as a bas - [Warning](#warning) - [Modifications in Bellatrix](#modifications-in-bellatrix) - - [Configuration](#configuration) - [The gossip domain: gossipsub](#the-gossip-domain-gossipsub) - [Topics and messages](#topics-and-messages) - [Global topics](#global-topics) @@ -41,15 +40,6 @@ Refer to the note in the [validator guide](./validator.md) for further details. ## Modifications in Bellatrix -### Configuration - -This section outlines modifications constants that are used in this spec. - -| Name | Value | Description | -|---|---|---| -| `GOSSIP_MAX_SIZE_BELLATRIX` | `10 * 2**20` (= 10,485,760, 10 MiB) | The maximum allowed size of uncompressed gossip messages starting at Bellatrix upgrade. | -| `MAX_CHUNK_SIZE_BELLATRIX` | `10 * 2**20` (= 10,485,760, 10 MiB) | The maximum allowed size of uncompressed req/resp chunked responses starting at Bellatrix upgrade. | - ### The gossip domain: gossipsub Some gossip meshes are upgraded in Bellatrix to support upgraded types. @@ -61,11 +51,6 @@ All topics remain stable except the beacon block topic which is updated with the The specification around the creation, validation, and dissemination of messages has not changed from the Phase 0 and Altair documents unless explicitly noted here. -Starting at Bellatrix upgrade, each gossipsub [message](https://github.com/libp2p/go-libp2p-pubsub/blob/master/pb/rpc.proto#L17-L24) -has a maximum size of `GOSSIP_MAX_SIZE_BELLATRIX`. -Clients MUST reject (fail validation) messages that are over this size limit. -Likewise, clients MUST NOT emit or propagate messages larger than this limit. - The derivation of the `message-id` remains stable. The new topics along with the type of the `data` field of a gossipsub message are given in this table: @@ -130,10 +115,6 @@ down-scoring or disconnection. Request and Response remain unchanged unless explicitly noted here. -Starting at Bellatrix upgrade, -a global maximum uncompressed byte size of `MAX_CHUNK_SIZE_BELLATRIX` MUST be applied to all method response chunks -regardless of type specific bounds that *MUST* also be respected. - Bellatrix fork-digest is introduced to the `context` enum to specify Bellatrix block type. Per `context = compute_fork_digest(fork_version, genesis_validators_root)`: @@ -171,17 +152,12 @@ Per `context = compute_fork_digest(fork_version, genesis_validators_root)`: With the addition of `ExecutionPayload` to `BeaconBlock`s, there is a dynamic field -- `transactions` -- which can validly exceed the `GOSSIP_MAX_SIZE` limit (1 MiB) put in -place at Phase 0. At the `GAS_LIMIT` (~30M) currently seen on mainnet in 2021, a single transaction +place at Phase 0, so GOSSIP_MAX_SIZE has increased to 10 Mib on the network. +At the `GAS_LIMIT` (~30M) currently seen on mainnet in 2021, a single transaction filled entirely with data at a cost of 16 gas per byte can create a valid `ExecutionPayload` of ~2 MiB. Thus we need a size limit to at least account for current mainnet conditions. -Geth currently has a [max gossip message size](https://github.com/ethereum/go-ethereum/blob/3ce9f6d96f38712f5d6756e97b59ccc20cc403b3/eth/protocols/eth/protocol.go#L49) of 10 MiB. -To support backward compatibility with this previously defined network limit, -we adopt `GOSSIP_MAX_SIZE_BELLATRIX` of 10 MiB for maximum gossip sizes at the -point of Bellatrix and beyond. Note, that clients SHOULD still reject objects -that exceed their maximum theoretical bounds which in most cases is less than `GOSSIP_MAX_SIZE_BELLATRIX`. - Note, that due to additional size induced by the `BeaconBlock` contents (e.g. proposer signature, operations lists, etc) this does reduce the theoretical max valid `ExecutionPayload` (and `transactions` list) size as diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index de3e0e529c..c0d18b08f3 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -189,11 +189,11 @@ This section outlines configurations that are used in this spec. | Name | Value | Description | |---|---|---| -| `GOSSIP_MAX_SIZE` | `2**20` (= 1048576, 1 MiB) | The maximum allowed size of uncompressed gossip messages. | +| `GOSSIP_MAX_SIZE` | `10 * 2**20` (= 10485760, 10 MiB) | The maximum allowed size of uncompressed gossip messages. | | `MAX_REQUEST_BLOCKS` | `2**10` (= 1024) | Maximum number of blocks in a single request | | `EPOCHS_PER_SUBNET_SUBSCRIPTION` | `2**8` (= 256) | Number of epochs on a subnet subscription (~27 hours) | | `MIN_EPOCHS_FOR_BLOCK_REQUESTS` | `MIN_VALIDATOR_WITHDRAWABILITY_DELAY + CHURN_LIMIT_QUOTIENT // 2` (= 33024, ~5 months) | The minimum epoch range over which a node must serve blocks | -| `MAX_CHUNK_SIZE` | `2**20` (=1048576, 1 MiB) | The maximum allowed size of uncompressed req/resp chunked responses. | +| `MAX_CHUNK_SIZE` | `10 * 2**20` (=10485760, 10 MiB) | The maximum allowed size of uncompressed req/resp chunked responses. | | `TTFB_TIMEOUT` | `5` | The maximum duration in **seconds** to wait for first byte of request response (time-to-first-byte). | | `RESP_TIMEOUT` | `10` | The maximum duration in **seconds** for complete response transfer. | | `ATTESTATION_PROPAGATION_SLOT_RANGE` | `32` | The maximum number of slots during which an attestation can be propagated. |