DA API Certificate verification #641

bacv · 2024-04-25T04:57:30Z

Certificate verification in the mempool (follow-up to #634). Tests will be added as the Indexer service nears completion.

danielSanchezQ

Another chunk of generics for the collection 🪙

danielSanchezQ · 2024-04-25T09:16:10Z

nomos-services/mempool/src/verify/mod.rs

+    // TODO: Payload verification parameters most likely will come from another Overwatch service.
+    // Once it's decided, update the `new` method to require service relay as parameter.
+    fn new(settings: Self::Settings) -> Self;
+
+    async fn get_parameters(&self, payload: &Self::Payload) -> Self::Parameters;


Yes, this is my main concern. It is gonna be difficult to be generic over this settings if by settings we need to include relays to other services. The best way would be to change the fn signature to include an overwatch handler. But then we are introducing a dependency on overwatch that we may not want. We have to think about this. I have some other ideas like passing a channel that gets updated everytime the settings change (as they probably are not static, they may change over time). And then drop the overwatch into a getter async task. Probably not the best explanation neither.

TLDR: Lets go ahead with this, then change it further ahead when we see the flaws clearly.

In other services we have settings separated from the relay and I think it works well, this is example from mempool network adapter:

nomos-node/nomos-services/mempool/src/network/mod.rs

Lines 21 to 24 in 69c88d9

async fn new(

settings: Self::Settings,

network_relay: OutboundRelay<<NetworkService<Self::Backend> as ServiceData>::Message>,

) -> Self;

I was mostly referring to this setup being missing in that TODO comment. But if you have an idea how we can improve this - lets discuss it.

Yes, but for other services settings are static. In the case for verification in our DA protocol, we need an updated list of registered DA nodes. That list comes from approved registrations in blocks, which changes over time (slots/epochs tbd). Which invalidates any static pre-configuration.

danielSanchezQ · 2024-04-25T09:16:55Z

nomos-services/mempool/src/verify/mod.rs

@@ -0,0 +1,12 @@
+#[async_trait::async_trait]
+pub trait MempoolVerificationProvider {


If this is intended for DaMempool lets specify in the name. If its intended to be generic over mempools lets keep it.

This definition is in mempool/verify/mod.rs, I'm pretty sure that we'll want some kind of verification for transactions? Da specific implementation is in mempool/da/verify/fullreplication.rs.

* Redo certificate verification in mempool * FullReplication verifier params provider * Integrate da params provider into the node

* Base cleaning of da to new traits/structure Added new da protocols and types * DA: KZG+RS core (#632) * Removed old kzg rs modules * Added new kzgrs core module * Implemented bytes_to_polynomial and tests * Use coefficient form * Refactor evaluations into method * Use domain elements instead of roots of unity in tests * Fix encoding and test * Clippy happy * Add comments * Implement polynomial commitment * Implement proof generation * Sketch fn signature for verification * implement proof verification * Implemented verification and tests * Return evaluations from bytes_to_polynomial as well * Use modular le bytes * Implement rs encode/decode * Implement decoding tests * Implement decode using lagrange * Cleanup imports * Da: v1 encoder (#633) * Added new kzgrs core module * Implemented bytes_to_polynomial and tests * Use coefficient form * Refactor evaluations into method * Use domain elements instead of roots of unity in tests * Fix encoding and test * Clippy happy * Add comments * Implement polynomial commitment * Implement proof generation * Sketch fn signature for verification * implement proof verification * Implemented verification and tests * Return evaluations from bytes_to_polynomial as well * Use modular le bytes * Implement rs encode/decode * Implement decoding tests * Implement decode using lagrange * Added chunksmatrix * Implement encoder with chunkify * Added missing files * Implement commit row commitments * Implement compute elements (row) proofs * Fix lib and types exposures * Implement encoder main methods * Implement encode method * Implement chunkify test Fix related bugs * Implement compute row kzg commitments Fix related bugs * Implement rs encode rows test Fix related bugs Refactored API * Implement row proofs tests Fix fieldelement encoding/decoding bug * Implement aggregated commitment test Implement aggregated column proofs test * Cleanup * Fix deps * Fix tests * Fix chunk too big test * Da: v1 verifier (#635) * Fix encoding and test * Implement commit row commitments * Implemented dablob * Implement verifier new Implement verifier check column * Clippy cleanup * Implement verifier * Implemented verify column test * Implemented full verify test * DA API Payload to Item in mempool (#634) * Base cleaning of da to new traits/structure Added new da protocols and types * DA: KZG+RS core (#632) * Removed old kzg rs modules * Added new kzgrs core module * Implemented bytes_to_polynomial and tests * Use coefficient form * Refactor evaluations into method * Use domain elements instead of roots of unity in tests * Fix encoding and test * Clippy happy * Add comments * Implement polynomial commitment * Implement proof generation * Sketch fn signature for verification * implement proof verification * Implemented verification and tests * Return evaluations from bytes_to_polynomial as well * Use modular le bytes * Implement rs encode/decode * Implement decoding tests * Implement decode using lagrange * Cleanup imports * Reduce abstraction for certificate and vid metadata * Allow payload to mempool as long as it converts into item * Da Certificate verifier * Add mock certificate for core tests * Mempool payload verification * Integrate mock verifiers for tx and certs * Detach verification from cert and tx * Seperate payload and item in mempools * Mempools in integration tests * Remove old cert verifier * Network payload to item constraints in da mempool * Update nomos-da/full-replication/src/lib.rs Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> * Sort attestations for cert signature * Update nomos-da/full-replication/src/lib.rs Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> --------- Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com> * DA API Certificate verification (#641) * Redo certificate verification in mempool * FullReplication verifier params provider * Integrate da params provider into the node * DA API Indexer service (#643) * Base cleaning of da to new traits/structure Added new da protocols and types * Remove da availability crate completely * Scaffold for da storage service * Indexer service responsible for storage and blockchain subscription * Handle index related ops only * Acquire storage and consensus relays * Indexer trait * wip: storage adapter * Use storage adapter instead of storage * Add consensus adapter trait for block subscriptions * Consensus block subscriber adapter * Use metadata from da core in indexer * Update nomos-services/data-availability/indexer/src/lib.rs Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> * Update nomos-services/data-availability/indexer/src/lib.rs Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> * Use std::ops::Range for da queries * Return certificate metadata * Da storage adapter methods --------- Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com> * Reuse evaluations when computing proofs (#647) * Reuse precomputed evaluations instead of evaluation polynomial for each proof * kzgrs benchmarks * Clippy happy * DA API indexer implementation (#644) * Rocksb adapter in da indexer * Handle da service messages * Remove indexer trait, use storage directly in the indexer service * Return unordered indexes range * Load blob by vid from file * Use VID in consensus * Change full replication index type to byte array * Change payload to cert and item to vid where required * Service integration tests for indexer * Feature gate rocksdb backend * Propagate range response send error * FRIndex to Index * VID to VidCertificate * Pass blobs file dir via settings * Da v1 multiple proofs bench (#648) * Parallel proof generation bench * Added bench comment * Modify domain to fit exact sizes * Fix domain in benches * Force parallelization features in lib * DA: Implement base structure for verifier service (#627) * Base cleaning of da to new traits/structure Added new da protocols and types * Implement base structure for verifier service * Added comments and todo! * Cleanup imports * Size of VidCert in full replication * Nomos Da Verifier service crate * Extension replaced with metadata * Fix DaIndexer service name * Storage adapter trait in verifier * Manage lifecycle and messages in verifier * Blob trait in core * Common nomos da storage crate * Use updated nomos da storage in indexer * Verifier storage adapter * Libp2p adaper for verifier * Kzgrs backend in verifier service * Fix fmt * Clippy happy --------- Co-authored-by: Gusto <bacvinka@gmail.com> * DA Verifier service integration tests (#650) * Base cleaning of da to new traits/structure Added new da protocols and types * Implement base structure for verifier service * Added comments and todo! * Cleanup imports * Size of VidCert in full replication * Nomos Da Verifier service crate * Extension replaced with metadata * Storage adapter trait in verifier * Manage lifecycle and messages in verifier * Common nomos da storage crate * Use updated nomos da storage in indexer * Verifier storage adapter * Libp2p adaper for verifier * Kzgrs backend in verifier service * Fix fmt * Data availability tests module * Return attestation in service msg response * Common definitions for da tests * Serde for kzgrs proofs and commitments * Da verifier integration test * WIP nomos-core in kzgrs backend * Kzgrs blob to common module * Add client zone to verifier test and check if attestations are created * Cleanup and comments * Use libp2p only for verifier and indexer service tests * Lint in da tests * Simplify da blob serialization * Remove attester from nomos-core attestation * Verifier backend error --------- Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com> * DA Kzgrs Backend Certificate implementation (#651) * Kzgrs backend certificate definition * Encoded data to certificate test * Nomos da domain specific tag * Handle errors in da certificate creation * Add nomos core traits to da cert * Derive ordering traits for da index * Add failure test cases to kzgrs certificate * Da v1 benches expand (#658) * Update benches with more cases * Expand benches * Added parallel feature * Fix test comment Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com> * Remove outdated comment --------- Co-authored-by: Gusto <bacvinka@gmail.com> Co-authored-by: gusto <bacv@users.noreply.github.com> Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>

bacv added 3 commits April 24, 2024 20:27

Redo certificate verification in mempool

5c04696

FullReplication verifier params provider

5d7ddda

Integrate da params provider into the node

69c88d9

bacv added Mempool da labels Apr 25, 2024

bacv requested review from danielSanchezQ and youngjoon-lee April 25, 2024 04:57

bacv self-assigned this Apr 25, 2024

danielSanchezQ approved these changes Apr 25, 2024

View reviewed changes

bacv merged commit a289390 into da-protocol-v1 Apr 25, 2024
10 of 12 checks passed

bacv deleted the da-api-cert-verify branch April 25, 2024 10:01

bacv added a commit that referenced this pull request May 13, 2024

DA API Certificate verification (#641)

a4eb739

* Redo certificate verification in mempool * FullReplication verifier params provider * Integrate da params provider into the node

bacv added a commit that referenced this pull request Jun 12, 2024

DA API Certificate verification (#641)

41c71dd

* Redo certificate verification in mempool * FullReplication verifier params provider * Integrate da params provider into the node

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DA API Certificate verification #641

DA API Certificate verification #641

bacv commented Apr 25, 2024

danielSanchezQ left a comment

danielSanchezQ Apr 25, 2024

bacv Apr 25, 2024

danielSanchezQ Apr 25, 2024

danielSanchezQ Apr 25, 2024

bacv Apr 25, 2024

	async fn new(
	settings: Self::Settings,
	network_relay: OutboundRelay<<NetworkService<Self::Backend> as ServiceData>::Message>,
	) -> Self;

		@@ -0,0 +1,12 @@
		#[async_trait::async_trait]
		pub trait MempoolVerificationProvider {

DA API Certificate verification #641

DA API Certificate verification #641

Conversation

bacv commented Apr 25, 2024

danielSanchezQ left a comment

Choose a reason for hiding this comment

danielSanchezQ Apr 25, 2024

Choose a reason for hiding this comment

bacv Apr 25, 2024

Choose a reason for hiding this comment

danielSanchezQ Apr 25, 2024

Choose a reason for hiding this comment

danielSanchezQ Apr 25, 2024

Choose a reason for hiding this comment

bacv Apr 25, 2024

Choose a reason for hiding this comment