feat: add devnet 5 support#378
Draft
MegaRedHand wants to merge 7 commits into
Draft
Conversation
…361) ## 🗒️ Description / Motivation Ports the typed two-level multi-signature envelope introduced by contributor commit [`anshalshukla/leanSpec@0ab09dd`](anshalshukla/leanSpec@0ab09dd) ("dummy type 1 and type 2 aggregation with block proofs") to ethlambda: - `TypeOneMultiSignature` — single-message N-signer proof; replaces `AggregatedSignatureProof` on the `SignedAggregatedAttestation` gossip wire. - `TypeTwoMultiSignature` — merged multi-message proof binding every per-attestation Type-1 plus a singleton proposer Type-1 over the block root. - `SignedBlock.signature: BlockSignatures` → `SignedBlock.proof: ByteListMiB` carrying the SSZ-encoded merged Type-2. The upstream commit is WIP (verify functions are explicit stubs, not yet in canonical `leanethereum/leanSpec`). ethlambda leads the wire-shape migration so the type plumbing is in place when canonical absorbs the refactor and real `lean_multisig` bindings land. **Opening as draft until canonical catches up.** ## What Changed Landed as three commits, one per phase. Each phase compiled and passed `make test` independently. ### Phase 1 — `f2d0fb5` — additive type plumbing - `crates/common/types/src/block.rs` — added `TypeOneInfo`, `TypeOneMultiSignature`, `TypeOneInfos` (SSZ-list limit `MAX_ATTESTATIONS_DATA + 1`), `TypeTwoMultiSignature`, and `BytecodeClaim` (typed alias for `H256`, placeholder until `lean_multisig` defines the trusted evaluation). - SSZ round-trip + capacity unit tests. - Pure additive: no consumers yet. ### Phase 2 — `18a60b5` — gossip-layer pipeline - `crates/common/types/src/attestation.rs` — `SignedAggregatedAttestation.proof` → `TypeOneMultiSignature`. - `crates/blockchain/src/aggregation.rs` — `AggregatedGroupOutput.proof`, `aggregate_job`, `resolve_child_pubkeys`, `select_proofs_greedily` all carry/read Type-1. - `crates/storage/src/store.rs` — `PayloadEntry.proofs: Vec<TypeOneMultiSignature>`; subsumption logic reads `info.participants`. - Block-builder helpers (`compact_attestations`, `extend_proofs_greedily`, `build_block`) operate on Type-1 throughout. - Temporary `to_legacy` / `from_legacy` boundary at block assembly + block-body ingestion so `SignedBlock` wire stayed legacy through Phase 2. ### Phase 3 — `fc9ce1f` — block wire + storage - `SignedBlock.signature: BlockSignatures` → `SignedBlock.proof: ByteListMiB`. Legacy `BlockSignatures` / `AttestationSignatures` / `AggregatedSignatureProof` removed. - `crates/blockchain/src/lib.rs::propose_block` wraps the proposer XMSS as a singleton Type-1, calls `aggregate_type_2`, SSZ-encodes the merged proof, and stashes it on `SignedBlock.proof`. - `crates/blockchain/src/store.rs::verify_signatures` rewritten as a structural-only check (mirrors upstream `verify_type_2` stub): decode the merged proof, assert `info.len() == attestations.len() + 1`, validate per-attestation `(message, slot, participants)` alignment and the trailing proposer entry; no per-Type-1 crypto. - `crates/storage/src/store.rs::write_signed_block` / `get_signed_block` now store `ByteListMiB` blobs in the existing `BlockSignatures` column family (renaming deferred to avoid a CF migration). - `aggregate_type_2` is a no-crypto stub today: it preserves the full `TypeOneInfos` metadata list but leaves `proof: ByteListMiB::default()`. Real merging arrives when `lean_multisig` exposes a merged-proof primitive — the existing `aggregate_proofs` only handles single-message merging. - Test fixtures regenerated from canonical leanSpec (`make leanSpec/fixtures`). The regen also cleared three pre-existing forkchoice spec failures on `main` (`AttestationTooFarInFuture` ×2, `AggregateVerificationFailed(InvalidProof)` on `test_valid_gossip_aggregated_attestation`) — they were stale-fixture artifacts. ## Correctness / Behavior Guarantees **Verified at gossip:** `on_gossip_aggregated_attestation` continues to run real `ethlambda_crypto::verify_aggregated_signature` on every `SignedAggregatedAttestation`. Invalid aggregates are rejected at the gossip boundary just like before. **Block-level becomes structural:** Block-level verification no longer crypto-verifies the merged proof. The merged proof bytes can't be split client-side (the type-2 merging primitive doesn't exist in `lean-multisig` yet — the existing `aggregate_proofs` is single-message only). `verify_signatures` enforces: - `info.len() == attestations.len() + 1`, - each `info[i]` matches the corresponding `block.body.attestations[i]` on `participants`, `slot`, and `message`, - the trailing `info[N]` has `message == block_root`, `slot == block.slot`, single-bit `participants` set to `block.proposer_index`, - all participant indices fit within the validator registry. This is the conscious "mirror upstream stubs" trade-off agreed during planning. When `lean_multisig` ships a real `verify_type_2`, the structural stub is swapped for the real call. **Block-body ingestion preserves fork-choice LMD GHOST inputs:** since the merged proof can't be split, `process_new_block` inserts info-only Type-1 entries (real `(message, slot, participants)`, empty proof bytes) into the payload buffer. `extract_latest_known_attestations` works unchanged. Empty-bytes entries never get fed back into `aggregate_proofs` (that path is only hit when multiple proofs share the same `AttestationData`, in which case at least one came from gossip with real bytes). **Storage:** Table name kept (`BlockSignatures`) to avoid a RocksDB CF migration; doc comment updated. Renaming to `Table::BlockProof` is a follow-up. **Skipped tests, all behind `TODO(type1-type2)`:** - `ssz_spectests.rs`: `SignedBlock`, `BlockSignatures`, `AggregatedSignatureProof`, `SignedAggregatedAttestation` — on-disk SSZ bytes still use the legacy schema since canonical leanSpec hasn't absorbed the refactor. - `signature_spectests.rs`: `test_invalid_proposer_signature` — relies on block-level proposer-signature crypto, which is now a structural stub. Attempted to bump `LEAN_SPEC_COMMIT_HASH` to `anshalshukla/leanSpec@0ab09dd` to regenerate fixtures against the new schema. Reverted: the upstream testing harness in that commit (`leanSpec/packages/testing/src/consensus_testing/keys.py`) still imports `AttestationSignatures`, which the same commit removes — `fill` crashes on module load. Documented in a `NOTE(type1-type2)` in the Makefile. ## Tests Added / Run - Added: SSZ round-trip and capacity unit tests for the new Type-1/Type-2 containers in `crates/common/types/src/block.rs`. - Updated: `verify_signatures_rejects_participants_mismatch`, `build_block_caps_attestation_data_entries`, `on_block_rejects_duplicate_attestation_data`, the `compact_attestations` and `extend_proofs_greedily` tests, all `forkchoice_spectests.rs` step builders, `signature_types.rs` fixture converter, and the `rpc::test_get_latest_finalized_block` test — all rebuilt to construct the new merged-proof shape. - Verified locally: - `make fmt` — clean - `cargo clippy --workspace --all-targets -- -D warnings` — clean - `cargo test --workspace --release` — green (84 forkchoice spec tests, 7 signature spec tests with 1 expected skip, all unit tests pass) ## Related Issues / PRs - Upstream commit being ported: [`anshalshukla/leanSpec@0ab09dd`](anshalshukla/leanSpec@0ab09dd) - Follow-ups when canonical absorbs the refactor: - Swap the structural `verify_type_2` stub for the real `lean_multisig` primitive. - Revert `LEAN_SPEC_COMMIT_HASH` skip markers in `ssz_spectests.rs` and `signature_spectests.rs`. - Consider renaming `Table::BlockSignatures` → `Table::BlockProof`. ## ✅ Verification Checklist - [x] Ran `make fmt` — clean - [x] Ran `make lint` (clippy with `-D warnings`) — clean - [x] Ran `cargo test --workspace --release` — all passing --------- Co-authored-by: Tomás Grüner <47506558+MegaRedHand@users.noreply.github.com>
# Conflicts: # crates/blockchain/src/store.rs # crates/common/test-fixtures/src/fork_choice.rs # crates/net/p2p/src/req_resp/handlers.rs # crates/net/rpc/src/lib.rs # crates/storage/src/store.rs
…370) Branched from #378 ## Summary - Bump `lean-multisig` / `leansig_wrapper` to devnet5 HEAD (`0242c909`) and rewrite `ethlambda-crypto` on the new Type-1 / Type-2 API. - Align the on-wire block proof with [leanSpec PR #717](leanEthereum/leanSpec#717) — `SignedBlock.proof` carries the SSZ-encoded `TypeTwoMultiSignature { proof: ByteList512KiB }` container, which collapses to `[4-byte LE offset = 4][type2_wire]` on the wire. - Port leanSpec PR #717's `SyncService._deconstruct_block_into_store` to the actor: imported blocks are SNARK-split per attestation, merged with local partial Type-1s, and (on aggregators) re-published on gossip. - Re-fixture against leanSpec `d9d2e67` (just past PR #717) and migrate the prod_scheme key JSON shape so signature and forkchoice spec tests cover the real cryptographic verifier end-to-end. ## Branch commit list | sha | what | |--|--| | `1cd80dd` | Crate-level integration: new Type-1 / Type-2 wrappers, real merge in `propose_block`, real `verify_type_2` in `verify_block_signatures`. | | `2c9dec0` | First pass at PR #717 envelope: `TypeOneInfo { participants, proof }`, `TypeTwoMultiSignature { info, proof }`, drop `bytecode_claim`. `split_type_2_signature(index)` → `split_type_2_by_message(message)`. | | `5361136` | Strip per-component Type-1 bytes when packing the Type-2 envelope — real Type-1s are ~225 KiB, N+1 copies blow the (old) 1 MiB `ByteListMiB` cap. | | `3199e7d` → `70c7cdb` | Experimental `--crypto-merge-t1-into-t2` flag, **reverted**. The merge runs synchronously on the actor today; moving it off-thread is a follow-up. | | `604ea4c` | Plan B: flatten `TypeOneMultiSignature` to `{ participants, proof }`, delete `TypeOneInfo` / `TypeOneInfos` / Rust `TypeTwoMultiSignature` wrappers, rename `ByteListMiB` → `ByteList512KiB`. | | `cc3df59` | Plan A: `reaggregate_from_block` module + actor hook. Caps `MAX_REAGGREGATIONS_PER_BLOCK = 4`, skips attestations behind the store's justified checkpoint, runs only when the chain is in sync. Aggregator-only republish on gossip. | | `4238a94` | Bump pinned `LEAN_SPEC_COMMIT_HASH` to `d9d2e67`, switch fixture generation to `--fork Lstar --scheme=test/prod`, port fixture parsers to the PR #717 schema (`signedBlock.proof.data` blob, `attestation.proof.proof.data` for gossip aggregates). | | `961aba4` | Restore the thin SSZ container header in front of the merged proof: `SignedBlock::wrap_merged_proof` / `merged_proof_bytes` helpers; lower `MAX_ATTESTATIONS_DATA` from 16 to 8 to match leanSpec PR #717. | ## Crypto crate API | function | wraps | notes | |--|--|--| | `aggregate_signatures(pks, sigs, msg, slot)` | `aggregate_type_1([], raw_xmss, …)` | Type-1 from raw XMSS only | | `aggregate_mixed(children, raw_pks, raw_sigs, msg, slot)` | `aggregate_type_1(children, raw_xmss, …)` | mixed Type-1 children + raw XMSS | | `aggregate_proofs(children, msg, slot)` | `aggregate_type_1(children, [], …)` | recursive Type-1 merge | | `verify_aggregated_signature(proof, pks, msg, slot)` | `verify_type_1` | Type-1 SNARK verify + explicit binding check | | `merge_type_1s_into_type_2(parts)` | `merge_many_type_1` | bundle N Type-1s into a Type-2 | | `verify_type_2_signature(proof_bytes, pks_per_component, expected_bindings)` | `verify_type_2` | Type-2 SNARK verify + per-component binding check; takes `&[u8]` after envelope strip | | `split_type_2_by_message(proof_bytes, pks_per_component, message)` | `split_type_2` (after locating index by message) | disaggregate to one Type-1; mirrors leanSpec `split_by_msg` | Type-1 / Type-2 proof bytes are `compress_without_pubkeys()` form throughout. `verify_type_2_signature` and `split_type_2_by_message` take `&[u8]` so callers feed the raw bytes (post-envelope-strip) directly. ## Wire format ``` TypeOneMultiSignature { participants: AggregationBits, proof: ByteList512KiB } SignedBlock.proof bytes: [4-byte LE offset = 4][raw lean-multisig Type-2 wire] ``` The 4-byte prefix is the SSZ Container-with-one-varlen-field offset header — the spec's `TypeTwoMultiSignature { proof: ByteList512KiB }` container. `SignedBlock::merged_proof_bytes()` / `SignedBlock::wrap_merged_proof()` keep the magic number off the call sites. No Rust struct for `TypeTwoMultiSignature` — per-component participants come from `block.body.attestations[i].aggregation_bits` and `block.proposer_index`, not the envelope. `MAX_ATTESTATIONS_DATA = 8` (down from 16, matching leanSpec PR #717). The merged Type-2 binds `MAX_ATTESTATIONS_DATA + 1 = 9` components, within lean-multisig's `MAX_RECURSIONS = 16`. ## Reaggregate-from-block New `crates/blockchain/src/reaggregate.rs`. After `process_block` succeeds and the chain is in sync, the actor: 1. Selects up to 4 attestations whose target outruns the store's justified checkpoint AND whose participants extend the local coverage. 2. `split_type_2_by_message`-splits each one out of the block's merged Type-2 proof. 3. Merges with locally-held partial Type-1s via `aggregate_proofs`. 4. Writes the combined proof into `latest_new_aggregated_payloads`. Aggregator-role nodes republish on gossip. 5 unit tests cover the candidate-selection rules without paying SNARK cost (target-slot gate, participant subset gate, hard cap, priority ordering). Each `split_type_2_by_message` runs a fresh SNARK; it currently executes synchronously on the actor thread. Moving to an off-thread worker mirroring `aggregation::run_aggregation_worker` is a natural follow-up if profiling shows it bleeding into the slot budget. ## Test coverage | Suite | Result | |---|---| | `signature_spectests` | 13 / 13 | | `forkchoice_spectests` | 84 / 84 | | `stf_spectests` | 35 / 35 | | `ssz_spectests` | 51 / 51 | | `test_driver_e2e` (Hive) | 8 / 8 | | `ethlambda-blockchain` unit | 24 / 24 (incl. 5 new reaggregate-from-block) | | Workspace total | 419 / 0 | Fixture regeneration: `LEAN_SPEC_COMMIT_HASH = d9d2e67`, generated with `make leanSpec/fixtures` (`uv run fill --fork Lstar --scheme=prod -o fixtures`). The prod_scheme key JSON shape upstream still has the pre-#725 flat layout (`attestation_public` / `attestation_secret` at top level) which the post-#725 `keys.py:395` no longer reads; this branch carries a one-shot local migration to the nested shape (`attestation_keypair.public_key` etc.) so the prod-scheme fixture filler runs. A small upstream PR converting the 12 `prod_scheme/*.json` files would let us drop the local step. ## Devnet validation Pending — re-run on a multi-node devnet now that `verify_type_2` actually executes on the import path. Expected to surface latency cliffs that the `--crypto-merge-t1-into-t2` flag previously hid.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🗒️ Description / Motivation
What Changed
Correctness / Behavior Guarantees
Tests Added / Run
Related Issues / PRs
✅ Verification Checklist
make fmt— cleanmake lint(clippy with-D warnings) — cleancargo test --workspace --release— all passing