DA: Implement base structure for verifier service #627
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bacv
force-pushed
the
da-protocol-v1
branch
2 times, most recently
from
ffdcd4e
to
3a06e8f
Compare
bacv
force-pushed
the
verifier-service-sketch
branch
from
aba1cd7
to
f29c052
Compare
bacv
force-pushed
the
verifier-service-sketch
branch
from
ab10ca3
to
b7104bd
Compare
Added new da protocols and types
bacv
force-pushed
the
verifier-service-sketch
branch
from
b7104bd
to
5cbd7c1
Compare
bacv
force-pushed
the
verifier-service-sketch
branch
from
61a5075
to
66016f8
Compare
bacv
force-pushed
the
verifier-service-sketch
branch
from
a697658
to
d5ef6d1
Compare
|
@danielSanchezQ please review, I can't add you on GH because you've created the initial PR. |
danielSanchezQ
commented
May 15, 2024
nomos-da/kzgrs-backend/src/common.rs
Outdated
| @@ -54,6 +62,10 @@ impl Column { | |||
| pub fn len(&self) -> usize { | |||
| self.0.len() | |||
| } | |||
| #[must_use] | |||
There was a problem hiding this comment.
Why must use? (it is repeated in the other methods as well)
There was a problem hiding this comment.
It was a suggestion from clippy: https://rust-lang.github.io/rust-clippy/master/index.html#/len_without_is_empty
Thought that the must_use annotation was mandatory, but it's not - removed.
Comment on lines
+149
to
+150
| // an updated list of the available nodes list, as it needs his own index coming from the | ||
| // position of his bls public key landing in the above-mentioned list. |
There was a problem hiding this comment.
It need to update itself. This list need to be read from new blocks if it change.
There was a problem hiding this comment.
(This is your comment) I've also added another one where the verifier is initialized. I'll work on "params provider" service after basic integration between services suceeds.
Comment on lines
+1
to
+107
| use crate::network::NetworkAdapter; | ||
| use nomos_core::wire; | ||
| use nomos_network::backends::libp2p::{Command, Event, EventKind, Libp2p, Message, TopicHash}; | ||
| use nomos_network::{NetworkMsg, NetworkService}; | ||
| use overwatch_rs::services::relay::OutboundRelay; | ||
| use overwatch_rs::services::ServiceData; | ||
| use serde::de::DeserializeOwned; | ||
| use serde::Serialize; | ||
| use tokio_stream::wrappers::BroadcastStream; | ||
| use tokio_stream::StreamExt; | ||
| use tracing::debug; | ||
|
|
||
| pub const NOMOS_DA_TOPIC: &str = "NomosDa"; | ||
|
|
||
| pub struct Libp2pAdapter<B, A> { | ||
| network_relay: OutboundRelay<<NetworkService<Libp2p> as ServiceData>::Message>, | ||
| _blob: PhantomData<B>, | ||
| _attestation: PhantomData<A>, | ||
| } | ||
|
|
||
| impl<B, A> Libp2pAdapter<B, A> | ||
| where | ||
| B: Serialize + DeserializeOwned + Send + Sync + 'static, | ||
| A: Serialize + DeserializeOwned + Send + Sync + 'static, | ||
| { | ||
| async fn stream_for<E: DeserializeOwned>(&self) -> Box<dyn Stream<Item = E> + Unpin + Send> { | ||
| let topic_hash = TopicHash::from_raw(NOMOS_DA_TOPIC); | ||
| let (sender, receiver) = tokio::sync::oneshot::channel(); | ||
| self.network_relay | ||
| .send(NetworkMsg::Subscribe { | ||
| kind: EventKind::Message, | ||
| sender, | ||
| }) | ||
| .await | ||
| .expect("Network backend should be ready"); | ||
| let receiver = receiver.await.unwrap(); | ||
| Box::new(Box::pin(BroadcastStream::new(receiver).filter_map( | ||
| move |msg| match msg { | ||
| Ok(Event::Message(Message { topic, data, .. })) if topic == topic_hash => { | ||
| match wire::deserialize::<E>(&data) { | ||
| Ok(msg) => Some(msg), | ||
| Err(e) => { | ||
| debug!("Unrecognized message: {e}"); | ||
| None | ||
| } | ||
| } | ||
| } | ||
| _ => None, | ||
| }, | ||
| ))) | ||
| } | ||
|
|
||
| async fn send<E: Serialize>(&self, data: E) -> Result<(), DynError> { | ||
| let message = wire::serialize(&data)?.into_boxed_slice(); | ||
| self.network_relay | ||
| .send(NetworkMsg::Process(Command::Broadcast { | ||
| topic: NOMOS_DA_TOPIC.to_string(), | ||
| message, | ||
| })) | ||
| .await | ||
| .map_err(|(e, _)| Box::new(e) as DynError) | ||
| } | ||
| } | ||
|
|
||
| #[async_trait::async_trait] | ||
| impl<B, A> NetworkAdapter for Libp2pAdapter<B, A> | ||
| where | ||
| B: Serialize + DeserializeOwned + Send + Sync + 'static, | ||
| A: Serialize + DeserializeOwned + Send + Sync + 'static, | ||
| { | ||
| type Backend = Libp2p; | ||
| type Settings = (); | ||
| type Blob = B; | ||
| type Attestation = A; | ||
|
|
||
| async fn new( | ||
| _settings: Self::Settings, | ||
| network_relay: OutboundRelay<<NetworkService<Self::Backend> as ServiceData>::Message>, | ||
| ) -> Self { | ||
| network_relay | ||
| .send(NetworkMsg::Process(Command::Subscribe( | ||
| NOMOS_DA_TOPIC.to_string(), | ||
| ))) | ||
| .await | ||
| .expect("Network backend should be ready"); | ||
| Self { | ||
| network_relay, | ||
| _blob: Default::default(), | ||
| _attestation: Default::default(), | ||
| } | ||
| } | ||
|
|
||
| async fn blob_stream(&self) -> Box<dyn Stream<Item = Self::Blob> + Unpin + Send> { | ||
| self.stream_for::<Self::Blob>().await | ||
| } | ||
|
|
||
| async fn send_attestation(&self, attestation: Self::Attestation) -> Result<(), DynError> { | ||
| self.send(attestation).await | ||
| } | ||
| } |
There was a problem hiding this comment.
It is still not sure how would it go. (Probably the approach of the http service we used to have is closer to what we were thinking).
How it is right now may be a problem, how do a node filter what blob is for himself? If he receives all blobs there is no point in doing all the DA dance 😅
We do not have a DA dispersal protocol yet. But I would say we should start point2point.
Maybe we bring back the http endpoints for now? wdyt?
There was a problem hiding this comment.
Libp2p network adapter is reverted to the version as in previous da service implementation - it should be functional, but the plan for a short term is to use http api for tests and cli (until Da Dissemination protocol is defined).
I've udes what we had before here. I'm not planning to use it in tests, but will try when reworking cli app.
Http api will be used for now.
bacv
force-pushed
the
verifier-service-sketch
branch
from
82d06da
to
02189db
Compare
bacv
added a commit
that referenced
this pull request
Jun 12, 2024
* Base cleaning of da to new traits/structure Added new da protocols and types * Implement base structure for verifier service * Added comments and todo! * Cleanup imports * Size of VidCert in full replication * Nomos Da Verifier service crate * Extension replaced with metadata * Fix DaIndexer service name * Storage adapter trait in verifier * Manage lifecycle and messages in verifier * Blob trait in core * Common nomos da storage crate * Use updated nomos da storage in indexer * Verifier storage adapter * Libp2p adaper for verifier * Kzgrs backend in verifier service * Fix fmt * Clippy happy --------- Co-authored-by: Gusto <bacvinka@gmail.com>
bacv
added a commit
that referenced
this pull request
Jun 13, 2024
* Base cleaning of da to new traits/structure Added new da protocols and types * DA: KZG+RS core (#632) * Removed old kzg rs modules * Added new kzgrs core module * Implemented bytes_to_polynomial and tests * Use coefficient form * Refactor evaluations into method * Use domain elements instead of roots of unity in tests * Fix encoding and test * Clippy happy * Add comments * Implement polynomial commitment * Implement proof generation * Sketch fn signature for verification * implement proof verification * Implemented verification and tests * Return evaluations from bytes_to_polynomial as well * Use modular le bytes * Implement rs encode/decode * Implement decoding tests * Implement decode using lagrange * Cleanup imports * Da: v1 encoder (#633) * Added new kzgrs core module * Implemented bytes_to_polynomial and tests * Use coefficient form * Refactor evaluations into method * Use domain elements instead of roots of unity in tests * Fix encoding and test * Clippy happy * Add comments * Implement polynomial commitment * Implement proof generation * Sketch fn signature for verification * implement proof verification * Implemented verification and tests * Return evaluations from bytes_to_polynomial as well * Use modular le bytes * Implement rs encode/decode * Implement decoding tests * Implement decode using lagrange * Added chunksmatrix * Implement encoder with chunkify * Added missing files * Implement commit row commitments * Implement compute elements (row) proofs * Fix lib and types exposures * Implement encoder main methods * Implement encode method * Implement chunkify test Fix related bugs * Implement compute row kzg commitments Fix related bugs * Implement rs encode rows test Fix related bugs Refactored API * Implement row proofs tests Fix fieldelement encoding/decoding bug * Implement aggregated commitment test Implement aggregated column proofs test * Cleanup * Fix deps * Fix tests * Fix chunk too big test * Da: v1 verifier (#635) * Fix encoding and test * Implement commit row commitments * Implemented dablob * Implement verifier new Implement verifier check column * Clippy cleanup * Implement verifier * Implemented verify column test * Implemented full verify test * DA API Payload to Item in mempool (#634) * Base cleaning of da to new traits/structure Added new da protocols and types * DA: KZG+RS core (#632) * Removed old kzg rs modules * Added new kzgrs core module * Implemented bytes_to_polynomial and tests * Use coefficient form * Refactor evaluations into method * Use domain elements instead of roots of unity in tests * Fix encoding and test * Clippy happy * Add comments * Implement polynomial commitment * Implement proof generation * Sketch fn signature for verification * implement proof verification * Implemented verification and tests * Return evaluations from bytes_to_polynomial as well * Use modular le bytes * Implement rs encode/decode * Implement decoding tests * Implement decode using lagrange * Cleanup imports * Reduce abstraction for certificate and vid metadata * Allow payload to mempool as long as it converts into item * Da Certificate verifier * Add mock certificate for core tests * Mempool payload verification * Integrate mock verifiers for tx and certs * Detach verification from cert and tx * Seperate payload and item in mempools * Mempools in integration tests * Remove old cert verifier * Network payload to item constraints in da mempool * Update nomos-da/full-replication/src/lib.rs Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> * Sort attestations for cert signature * Update nomos-da/full-replication/src/lib.rs Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> --------- Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com> * DA API Certificate verification (#641) * Redo certificate verification in mempool * FullReplication verifier params provider * Integrate da params provider into the node * DA API Indexer service (#643) * Base cleaning of da to new traits/structure Added new da protocols and types * Remove da availability crate completely * Scaffold for da storage service * Indexer service responsible for storage and blockchain subscription * Handle index related ops only * Acquire storage and consensus relays * Indexer trait * wip: storage adapter * Use storage adapter instead of storage * Add consensus adapter trait for block subscriptions * Consensus block subscriber adapter * Use metadata from da core in indexer * Update nomos-services/data-availability/indexer/src/lib.rs Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> * Update nomos-services/data-availability/indexer/src/lib.rs Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> * Use std::ops::Range for da queries * Return certificate metadata * Da storage adapter methods --------- Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com> * Reuse evaluations when computing proofs (#647) * Reuse precomputed evaluations instead of evaluation polynomial for each proof * kzgrs benchmarks * Clippy happy * DA API indexer implementation (#644) * Rocksb adapter in da indexer * Handle da service messages * Remove indexer trait, use storage directly in the indexer service * Return unordered indexes range * Load blob by vid from file * Use VID in consensus * Change full replication index type to byte array * Change payload to cert and item to vid where required * Service integration tests for indexer * Feature gate rocksdb backend * Propagate range response send error * FRIndex to Index * VID to VidCertificate * Pass blobs file dir via settings * Da v1 multiple proofs bench (#648) * Parallel proof generation bench * Added bench comment * Modify domain to fit exact sizes * Fix domain in benches * Force parallelization features in lib * DA: Implement base structure for verifier service (#627) * Base cleaning of da to new traits/structure Added new da protocols and types * Implement base structure for verifier service * Added comments and todo! * Cleanup imports * Size of VidCert in full replication * Nomos Da Verifier service crate * Extension replaced with metadata * Fix DaIndexer service name * Storage adapter trait in verifier * Manage lifecycle and messages in verifier * Blob trait in core * Common nomos da storage crate * Use updated nomos da storage in indexer * Verifier storage adapter * Libp2p adaper for verifier * Kzgrs backend in verifier service * Fix fmt * Clippy happy --------- Co-authored-by: Gusto <bacvinka@gmail.com> * DA Verifier service integration tests (#650) * Base cleaning of da to new traits/structure Added new da protocols and types * Implement base structure for verifier service * Added comments and todo! * Cleanup imports * Size of VidCert in full replication * Nomos Da Verifier service crate * Extension replaced with metadata * Storage adapter trait in verifier * Manage lifecycle and messages in verifier * Common nomos da storage crate * Use updated nomos da storage in indexer * Verifier storage adapter * Libp2p adaper for verifier * Kzgrs backend in verifier service * Fix fmt * Data availability tests module * Return attestation in service msg response * Common definitions for da tests * Serde for kzgrs proofs and commitments * Da verifier integration test * WIP nomos-core in kzgrs backend * Kzgrs blob to common module * Add client zone to verifier test and check if attestations are created * Cleanup and comments * Use libp2p only for verifier and indexer service tests * Lint in da tests * Simplify da blob serialization * Remove attester from nomos-core attestation * Verifier backend error --------- Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com> * DA Kzgrs Backend Certificate implementation (#651) * Kzgrs backend certificate definition * Encoded data to certificate test * Nomos da domain specific tag * Handle errors in da certificate creation * Add nomos core traits to da cert * Derive ordering traits for da index * Add failure test cases to kzgrs certificate * Da v1 benches expand (#658) * Update benches with more cases * Expand benches * Added parallel feature * Fix test comment Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com> * Remove outdated comment --------- Co-authored-by: Gusto <bacvinka@gmail.com> Co-authored-by: gusto <bacv@users.noreply.github.com> Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added verifier service that expects nomos da protocol implementation as a backend.
Libp2p network adapter is reverted to the version as in previous da service implementation - it should be functional, but the plan for a short term is to use http api for tests and cli (until Da Dissemination protocol is defined).
Service integration test will be added after this PR is reviewd and merged.