Implement EIP-4844 (proto danksharding)

[benjaminion]

Specification

Everything that builds on top of previous fork should build on top of Capella, so it is strongly suggested to wait Capella classes to be defined before introducing Eip4844 versions.

Decoupling EIP-4844 blobs

• Implement Blob Decoupling #6822

backlog

• Specify the number of sidecar subnets #7167
• Use engine_newPayloadV3 to pass versioned_hashes to EL for validation #7203
• Implement unit tests in DatabaseTest covering 4844 enabled tx #6668
• Optimize interface to Kzg library Implement a wrapper around c-kzg jni bindings #6520 (we should minimize unnecessary data copy)
  block fetcher)
• Add support of kzg reference tests #6969

Nice to have

• BlobSidecar related Store tests #7946
• Restore DebugDbCommand validate-blob-sidecars #7947
• Add LRU cache for BlobSidecars #7369
• Add new blob related structures in JsonProviderPropertyTest
• Package consistency (blobs vs blob and blocks vs block)
• make MIN_EPOCHS_FOR_BLOBS_SIDECARS_REQUESTS overridable #6395
• Issues under the KZG library (https://github.com/ConsenSys/jc-kzg-4844/issues)

Tasks below are prior to the blobs decoupling spec change, so may not be required

Configuration

• Define new constants #6377
• Define new types #6378
• Define new presets #6379
• Support trusted setup configuration #6382
• Rename EIP4844 Milestone to Deneb #6608

SSZ Containers and schema

• Add\update containers #6383
• expose schema definitions for Eip4844 fork #6388
• change excess_blobs: uint64 to excess_data_gas: uint256 #6479

Spec

• Implement MiscHelpersEip4844 #6391
• Implement SpecConfigEip4844 #6401
• Implement BlockProcessorEip4844 #6392
• Implement SpecLogicEip4844 #6440
  • BlindBlockUtilEip4844
    • SignedBeaconBlockBlinderEip4844
    • SignedBeaconBlockUnblinderEip4844
• Implement Eip4844StateUpgrade #6454
• blockProcessor should no more initiate data availability check #6673

Validator coordinator

• Implement Block Production #6411

Storage

• update Storage and KvStore classes to support BlobsSidecar lifecycle #6586
• implement BlobsSidecar lifecycle. #6629
• Implement blobs related Store methods #6633
• Implement retrieveSignedBlockAndBlobsSidecar in TestStoreImpl #6724

context:

• expected minimum space requirements (with current mainnet params) for the worst case scenario (maximum blob capacity):
  FIELD_ELEMENTS_PER_BLOB * BLSFieldElement * MAX_BLOBS_PER_BLOCK * MIN_EPOCHS_FOR_BLOBS_SIDECARS_REQUESTS * SLOTS_PER_EPOCH = 4096*32 (131KB: max blob size) * 16 (2MB max blobs sidecar) * 4096 * 32 = 256GiB

Update: MAX_BLOBS_PER_BLOCK is now 4 so space requirements goes down to 64GiB

• we need to serve blobs:

  • by block root from the last finalized blobs to the most recent (to responds with a coupled SignedBeaconBlockAndBlobsSidecar)

  • by slot range for p2p (still need do be decided if coupled or decoupled from blocks)

Other things to consider:

• check "Additional notes" from here: [EIP-4844] Implement BlobsManager #6636

Sync

• Implement Forward Sync for EIP-4844 #6669
• Extend RecentBlockFetcherService for EIP-4844 #6691

HistoricalBlockSyncService

• Implement blob sidecar backfill. This becomes critical since we can consider our head valid only if all blobs within MIN_EPOCHS_FOR_BLOBS_SIDECARS_REQUESTS have been downloaded (verified against the corresponding block)

We have to bare in mind that we have to run validate_blobs_sidecar against the alleged coupled block to make sure that they actually couple.

Networking

• gossip
  • Implement beacon_block_and_blobs_sidecar topic #6460
  • implement beacon_block_and_blobs_sidecar gossip validation #6473
• request\response
  • Implement BeaconBlocksByRange v2
  • Implement BlobsSidecarsByRange v1 #6571
  • Implement BlobsSidecarsByRange validation #6584
  • Throttle BlobsSidecarsByRange requests #6585
  • Add BlobsSidecarsByRange Integration test #6602
  • Implement BeaconBlockAndBlobsSidecarByRoot v1 #6609
  • BeaconBlocksByRootV2 is replaced by BeaconBlockAndBlobsSidecarByRootV1 when 4844 activates. We have to continue using BeaconBlocksByRootV2 until 4844 fork finalizes, to cover the case in which we want to lookup blocks from previous fork. (EIP4844: Add block and sidecar retrival by root ethereum/consensus-specs#3089) (covered in Extend RecentBlockFetcherService for EIP-4844 #6691)
  • Add BeaconBlockAndBlobsSidecarByRoot Integration test #6638

interesting point from Lion: ethereum/consensus-specs#3113

Beacon API

• Expose new Eip4844 Block and State in Beacon API #6710

Engine API

• Implement engine_getBlobsBundleV1 in engine API #6387

KZG Crypto library

• Create java binding for c-kzg library #6389
• Add c-kzg java binding dependency in infrastructure crypto module #6470

backlog

• EIP-4844 tests #6422
• Follow EIP4844: Update MAX_BLOBS_PER_BLOCK to 4 ethereum/consensus-specs#3145 preset changes when merged
• Extend ChainBuilder to support EIP-4844 #6634
• Enable 4844 reference tests
• Enable Capella operations when it will hit EIP-4844 reference test (should be v1.3.0). It was disabled in our code and it's enabled finally in spec.
• change tx type from 0x05 to 0x03 deneb: switch blob tx type to 0x03 ethereum/consensus-specs#3317

[benjaminion]

Implementation notes for consensus clients.

[benjaminion]

4844 Readiness Checklist

[lucassaldanha]

@tbenr Are we happy to close this epic now?

[mehdi-aouadi]

@lucassaldanha there is only #7946 left, will close the epic once it's merged

[mehdi-aouadi]

All done now 🥳