DA API Payload to Item in mempool #634

bacv · 2024-04-16T17:55:27Z

Payload to Item conversion when appending to the mempool. This change aims to support da certificate conversion into VID and insertion into a mempool. Receive Certificate as payload, insert VID as item.

danielSanchezQ · 2024-04-17T08:30:05Z

nomos-core/src/da/auth/mod.rs

+pub trait Signer {
+    fn sign(&self, message: &[u8]) -> Vec<u8>;
+}
+
+pub trait Verifier {
+    fn verify(&self, message: &[u8], sig: &[u8]) -> bool;
+}


What is this used for? I am not really sure we need this.

This will be removed as CertificateVerifier will be replaced with implementation specific verifier, but this trait was used to constraint KeyProvider:
https://github.com/logos-co/nomos-node/pull/634/files/10675e02d5317f853cc1ceb0bd9bca1d25353004#diff-bd4a1028aaa1f642e81508d453bb409463a5e4bbe6e05e8131a9e6403d756a69R7-R11

danielSanchezQ · 2024-04-19T11:20:19Z

nomos-services/data-availability/src/auth/mod.rs

@@ -0,0 +1,11 @@
+pub mod mock;


We should probably feature gate this (not necessarely in this PR) (?)

We have more mocks in core unrelated to DA, we could have one PR to feature gate them all.

danielSanchezQ · 2024-04-19T11:21:12Z

nomos-services/data-availability/src/auth/mod.rs

+pub trait DaAuth {
+    type Settings: Clone;
+
+    fn new(settings: Self::Settings) -> Self;
+}


I don't get the main purpose of DaAuth

Some parts of previous verification proposal are still in this PR. As discussed, the verification will be defined by the certificate implementation and only performed in the da mempool. DaAuth and other structures will be removed. I still need to add more changes to this, will ping reviewers once the draft is converted to a proper PR.

danielSanchezQ · 2024-04-19T11:22:19Z

nomos-services/mempool/src/backend/mod.rs

+    fn add_item<I: Into<Self::Item>>(
+        &mut self,
+        key: Self::Key,
+        item: I,
+    ) -> Result<(), MempoolError>;


This is in general a good abstraction actually. Really nice.

danielSanchezQ · 2024-04-19T11:23:22Z

nomos-core/src/da/certificate/mock.rs

+impl MockAttestation {
+    pub fn new(voter: &[u8; 32], signature: &[u8]) -> Self {
+        MockAttestation {
+            voter: *voter,
+            signature: signature.to_vec(),
+        }
+    }
+}


As well probably feature gated? Or do we plan to remove this once we have the actual stuff in place?

Added new da protocols and types

* Removed old kzg rs modules * Added new kzgrs core module * Implemented bytes_to_polynomial and tests * Use coefficient form * Refactor evaluations into method * Use domain elements instead of roots of unity in tests * Fix encoding and test * Clippy happy * Add comments * Implement polynomial commitment * Implement proof generation * Sketch fn signature for verification * implement proof verification * Implemented verification and tests * Return evaluations from bytes_to_polynomial as well * Use modular le bytes * Implement rs encode/decode * Implement decoding tests * Implement decode using lagrange * Cleanup imports

bacv · 2024-04-22T12:23:57Z

@danielSanchezQ @youngjoon-lee @zeegomo Marking PR as ready to be reviewed and merged into da-protocol-v1 branch.

The main objective of this PR was to allow da mempool receive a Certificate via the network and covert it to VID when adding to the item pool. Certificate verification will be added in other PR.

danielSanchezQ · 2024-04-22T15:12:20Z

Nice to see tests green again!!! 🚀

danielSanchezQ

I may be misslead for the (probably) unupdated PR title. But I cannot locate where does the verification happens (in the da mempool?).

danielSanchezQ · 2024-04-22T15:57:42Z

nomos-da/full-replication/src/lib.rs

+        let mut signatures = Vec::new();
+        for attestation in &self.attestations {
+            signatures.extend_from_slice(attestation.signature());
+        }


If attestations are in different order it will produce different signature. I think this should sort the signatures first so it is deterministic (?)

Added sorting, will expand in the PR for certificate verification.

nomos-da/full-replication/src/lib.rs

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

bacv · 2024-04-22T16:48:29Z

I may be misslead for the (probably) unupdated PR title. But I cannot locate where does the verification happens (in the da mempool?).

I've updated the PR description, at the moment code focuses on the conversion; Verification will be added in the next PR.

nomos-da/full-replication/src/lib.rs

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

danielSanchezQ · 2024-04-23T08:35:02Z

nomos-da/full-replication/src/lib.rs

+    fn metadata(&self) -> Option<(Self::AppId, Self::Index)> {
+        Some((self.metadata.app_id, self.metadata.index))


What happens if a certificate has no metadata? or when do this happens? Just trying to understand the Option in the trait 🤔

I made it optional in the FullReplication implementation assuming that when the zone is receiving attestations, it might want to hold a partial certificate without metadata assigned. Verification on the node side should fail if metadata is not attached. We might improve this with the da-protocol-v1 certificate.

I'm not sure that is the correct approach. I feel that if something do not have the metadata it should just simply not be able to implement the trait. But yeah, lets revisit this later on.

danielSanchezQ

🎸 We are almost there! Looking good.

* Base cleaning of da to new traits/structure Added new da protocols and types * DA: KZG+RS core (#632) * Removed old kzg rs modules * Added new kzgrs core module * Implemented bytes_to_polynomial and tests * Use coefficient form * Refactor evaluations into method * Use domain elements instead of roots of unity in tests * Fix encoding and test * Clippy happy * Add comments * Implement polynomial commitment * Implement proof generation * Sketch fn signature for verification * implement proof verification * Implemented verification and tests * Return evaluations from bytes_to_polynomial as well * Use modular le bytes * Implement rs encode/decode * Implement decoding tests * Implement decode using lagrange * Cleanup imports * Reduce abstraction for certificate and vid metadata * Allow payload to mempool as long as it converts into item * Da Certificate verifier * Add mock certificate for core tests * Mempool payload verification * Integrate mock verifiers for tx and certs * Detach verification from cert and tx * Seperate payload and item in mempools * Mempools in integration tests * Remove old cert verifier * Network payload to item constraints in da mempool * Update nomos-da/full-replication/src/lib.rs Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> * Sort attestations for cert signature * Update nomos-da/full-replication/src/lib.rs Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> --------- Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>

* Base cleaning of da to new traits/structure Added new da protocols and types * DA: KZG+RS core (#632) * Removed old kzg rs modules * Added new kzgrs core module * Implemented bytes_to_polynomial and tests * Use coefficient form * Refactor evaluations into method * Use domain elements instead of roots of unity in tests * Fix encoding and test * Clippy happy * Add comments * Implement polynomial commitment * Implement proof generation * Sketch fn signature for verification * implement proof verification * Implemented verification and tests * Return evaluations from bytes_to_polynomial as well * Use modular le bytes * Implement rs encode/decode * Implement decoding tests * Implement decode using lagrange * Cleanup imports * Da: v1 encoder (#633) * Added new kzgrs core module * Implemented bytes_to_polynomial and tests * Use coefficient form * Refactor evaluations into method * Use domain elements instead of roots of unity in tests * Fix encoding and test * Clippy happy * Add comments * Implement polynomial commitment * Implement proof generation * Sketch fn signature for verification * implement proof verification * Implemented verification and tests * Return evaluations from bytes_to_polynomial as well * Use modular le bytes * Implement rs encode/decode * Implement decoding tests * Implement decode using lagrange * Added chunksmatrix * Implement encoder with chunkify * Added missing files * Implement commit row commitments * Implement compute elements (row) proofs * Fix lib and types exposures * Implement encoder main methods * Implement encode method * Implement chunkify test Fix related bugs * Implement compute row kzg commitments Fix related bugs * Implement rs encode rows test Fix related bugs Refactored API * Implement row proofs tests Fix fieldelement encoding/decoding bug * Implement aggregated commitment test Implement aggregated column proofs test * Cleanup * Fix deps * Fix tests * Fix chunk too big test * Da: v1 verifier (#635) * Fix encoding and test * Implement commit row commitments * Implemented dablob * Implement verifier new Implement verifier check column * Clippy cleanup * Implement verifier * Implemented verify column test * Implemented full verify test * DA API Payload to Item in mempool (#634) * Base cleaning of da to new traits/structure Added new da protocols and types * DA: KZG+RS core (#632) * Removed old kzg rs modules * Added new kzgrs core module * Implemented bytes_to_polynomial and tests * Use coefficient form * Refactor evaluations into method * Use domain elements instead of roots of unity in tests * Fix encoding and test * Clippy happy * Add comments * Implement polynomial commitment * Implement proof generation * Sketch fn signature for verification * implement proof verification * Implemented verification and tests * Return evaluations from bytes_to_polynomial as well * Use modular le bytes * Implement rs encode/decode * Implement decoding tests * Implement decode using lagrange * Cleanup imports * Reduce abstraction for certificate and vid metadata * Allow payload to mempool as long as it converts into item * Da Certificate verifier * Add mock certificate for core tests * Mempool payload verification * Integrate mock verifiers for tx and certs * Detach verification from cert and tx * Seperate payload and item in mempools * Mempools in integration tests * Remove old cert verifier * Network payload to item constraints in da mempool * Update nomos-da/full-replication/src/lib.rs Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> * Sort attestations for cert signature * Update nomos-da/full-replication/src/lib.rs Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> --------- Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com> * DA API Certificate verification (#641) * Redo certificate verification in mempool * FullReplication verifier params provider * Integrate da params provider into the node * DA API Indexer service (#643) * Base cleaning of da to new traits/structure Added new da protocols and types * Remove da availability crate completely * Scaffold for da storage service * Indexer service responsible for storage and blockchain subscription * Handle index related ops only * Acquire storage and consensus relays * Indexer trait * wip: storage adapter * Use storage adapter instead of storage * Add consensus adapter trait for block subscriptions * Consensus block subscriber adapter * Use metadata from da core in indexer * Update nomos-services/data-availability/indexer/src/lib.rs Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> * Update nomos-services/data-availability/indexer/src/lib.rs Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> * Use std::ops::Range for da queries * Return certificate metadata * Da storage adapter methods --------- Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com> * Reuse evaluations when computing proofs (#647) * Reuse precomputed evaluations instead of evaluation polynomial for each proof * kzgrs benchmarks * Clippy happy * DA API indexer implementation (#644) * Rocksb adapter in da indexer * Handle da service messages * Remove indexer trait, use storage directly in the indexer service * Return unordered indexes range * Load blob by vid from file * Use VID in consensus * Change full replication index type to byte array * Change payload to cert and item to vid where required * Service integration tests for indexer * Feature gate rocksdb backend * Propagate range response send error * FRIndex to Index * VID to VidCertificate * Pass blobs file dir via settings * Da v1 multiple proofs bench (#648) * Parallel proof generation bench * Added bench comment * Modify domain to fit exact sizes * Fix domain in benches * Force parallelization features in lib * DA: Implement base structure for verifier service (#627) * Base cleaning of da to new traits/structure Added new da protocols and types * Implement base structure for verifier service * Added comments and todo! * Cleanup imports * Size of VidCert in full replication * Nomos Da Verifier service crate * Extension replaced with metadata * Fix DaIndexer service name * Storage adapter trait in verifier * Manage lifecycle and messages in verifier * Blob trait in core * Common nomos da storage crate * Use updated nomos da storage in indexer * Verifier storage adapter * Libp2p adaper for verifier * Kzgrs backend in verifier service * Fix fmt * Clippy happy --------- Co-authored-by: Gusto <bacvinka@gmail.com> * DA Verifier service integration tests (#650) * Base cleaning of da to new traits/structure Added new da protocols and types * Implement base structure for verifier service * Added comments and todo! * Cleanup imports * Size of VidCert in full replication * Nomos Da Verifier service crate * Extension replaced with metadata * Storage adapter trait in verifier * Manage lifecycle and messages in verifier * Common nomos da storage crate * Use updated nomos da storage in indexer * Verifier storage adapter * Libp2p adaper for verifier * Kzgrs backend in verifier service * Fix fmt * Data availability tests module * Return attestation in service msg response * Common definitions for da tests * Serde for kzgrs proofs and commitments * Da verifier integration test * WIP nomos-core in kzgrs backend * Kzgrs blob to common module * Add client zone to verifier test and check if attestations are created * Cleanup and comments * Use libp2p only for verifier and indexer service tests * Lint in da tests * Simplify da blob serialization * Remove attester from nomos-core attestation * Verifier backend error --------- Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com> * DA Kzgrs Backend Certificate implementation (#651) * Kzgrs backend certificate definition * Encoded data to certificate test * Nomos da domain specific tag * Handle errors in da certificate creation * Add nomos core traits to da cert * Derive ordering traits for da index * Add failure test cases to kzgrs certificate * Da v1 benches expand (#658) * Update benches with more cases * Expand benches * Added parallel feature * Fix test comment Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com> * Remove outdated comment --------- Co-authored-by: Gusto <bacvinka@gmail.com> Co-authored-by: gusto <bacv@users.noreply.github.com> Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>

bacv added Mempool da labels Apr 16, 2024

bacv requested review from danielSanchezQ and youngjoon-lee April 16, 2024 17:55

bacv self-assigned this Apr 16, 2024

danielSanchezQ reviewed Apr 17, 2024

View reviewed changes

bacv force-pushed the da-api-vid-mempool branch from 10675e0 to 1618be8 Compare April 17, 2024 11:50

bacv force-pushed the da-protocol-v1 branch from 6616469 to aebdd5c Compare April 18, 2024 13:25

bacv force-pushed the da-api-vid-mempool branch from b535632 to 3cbbb9c Compare April 19, 2024 11:05

danielSanchezQ reviewed Apr 19, 2024

View reviewed changes

bacv force-pushed the da-api-vid-mempool branch from 5eca951 to 66adf7e Compare April 19, 2024 13:30

danielSanchezQ and others added 13 commits April 22, 2024 15:20

Base cleaning of da to new traits/structure

23e096f

Added new da protocols and types

Reduce abstraction for certificate and vid metadata

4a99d7f

Allow payload to mempool as long as it converts into item

7f4586d

Da Certificate verifier

e7d1f53

Add mock certificate for core tests

93f5418

Mempool payload verification

850f603

Integrate mock verifiers for tx and certs

92c6686

Detach verification from cert and tx

a6c9cda

Seperate payload and item in mempools

fe015b8

Mempools in integration tests

ff97095

Remove old cert verifier

1a76f4f

Network payload to item constraints in da mempool

bf7b90a

bacv force-pushed the da-api-vid-mempool branch from 07ae5fa to bf7b90a Compare April 22, 2024 12:20

bacv marked this pull request as ready for review April 22, 2024 12:20

bacv requested a review from zeegomo April 22, 2024 12:24

danielSanchezQ reviewed Apr 22, 2024

View reviewed changes

Update nomos-da/full-replication/src/lib.rs

bffac0e

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

bacv force-pushed the da-api-vid-mempool branch from 5613858 to bffac0e Compare April 22, 2024 16:22

Sort attestations for cert signature

3f8775b

bacv changed the title ~~DA API mempool VID verification~~ DA API Payload to Item in mempool Apr 22, 2024

danielSanchezQ reviewed Apr 22, 2024

View reviewed changes

nomos-da/full-replication/src/lib.rs Outdated Show resolved Hide resolved

Update nomos-da/full-replication/src/lib.rs

ca77211

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

bacv force-pushed the da-api-vid-mempool branch from 612ce7b to ca77211 Compare April 22, 2024 17:12

danielSanchezQ reviewed Apr 23, 2024

View reviewed changes

danielSanchezQ approved these changes Apr 23, 2024

View reviewed changes

bacv merged commit 4b67475 into da-protocol-v1 Apr 23, 2024
9 of 11 checks passed

bacv deleted the da-api-vid-mempool branch April 23, 2024 09:32

bacv mentioned this pull request Apr 25, 2024

DA API Certificate verification #641

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DA API Payload to Item in mempool #634

DA API Payload to Item in mempool #634

bacv commented Apr 16, 2024 •

edited

danielSanchezQ Apr 17, 2024

bacv Apr 17, 2024

danielSanchezQ Apr 19, 2024

bacv Apr 22, 2024

danielSanchezQ Apr 19, 2024

bacv Apr 19, 2024

danielSanchezQ Apr 19, 2024

danielSanchezQ Apr 19, 2024

bacv commented Apr 22, 2024

danielSanchezQ commented Apr 22, 2024

danielSanchezQ left a comment

danielSanchezQ Apr 22, 2024

bacv Apr 22, 2024

bacv commented Apr 22, 2024

danielSanchezQ Apr 23, 2024

bacv Apr 23, 2024

danielSanchezQ Apr 23, 2024

danielSanchezQ left a comment

		fn metadata(&self) -> Option<(Self::AppId, Self::Index)> {
		Some((self.metadata.app_id, self.metadata.index))

DA API Payload to Item in mempool #634

DA API Payload to Item in mempool #634

Conversation

bacv commented Apr 16, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bacv commented Apr 22, 2024

danielSanchezQ commented Apr 22, 2024

danielSanchezQ left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bacv commented Apr 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielSanchezQ left a comment

Choose a reason for hiding this comment

bacv commented Apr 16, 2024 •

edited