Listen to chain events and update twin on requests #194

AbdelrahmanElawady · 2024-03-21T14:13:02Z

Description

This PR utilize listening to chain events to update relay cache with the latest twin information. For TwinStored and TwinUpdated events, the event listener get twin information from the event and update local cache. For twins that are neither update nor created after starting events listener they will be fetched from the chain for the first time a request is made but will remain in cache forever until a TwinUpdated is emitted then its data will be updated too.

The time difference between clients calling update_twin and the blocks finalizing led to response failures for up to 6 seconds. So, to mitigate this delay a new optional relays field was introduced to Envelope type. If set it will be considered the source of truth for which relays the twin is using and relays will update local cache based on that (even for federating messages).

This removes all delay for twin updates (if using new field in envelope) and the need for cache warmer.

Issues

[relay] chain cache bottleneck #189

AbdelrahmanElawady · 2024-03-21T14:15:18Z

still missing reconnecting to chain urls for rpc errors in a round robin way

AbdelrahmanElawady · 2024-03-24T10:17:25Z

Connection error handling was added and tested with gdb and calling manually shutdown() syscall to simulate chain closing connection.

muhamadazmy

In general this looks good but i have really few comments below.

muhamadazmy · 2024-03-25T08:58:08Z

proto/types.proto

@@ -71,4 +71,6 @@ message Envelope {
    bytes plain = 13;
    bytes cipher = 14;
  }
+
+  optional string relays = 17;


why this is just a single string, not a list of strings? it's better to send it as a []string imho

if we gonna re-introduced this again why not reuse the old federation field?
is that to keep compatibility with old peers that may interrupt it differently?
Also it is expected to have some seconds delay after switching the relay before it can receive messages.
it won't be bad as DNS propagation time which can take anywhere from a few hours up to 48 hours to propagate a new domain name for a website worldwide :D

so even if we didn't go with this path (adding the relays filed) we still just fine, how often node would switch its relays given that we are storing on-chain multiple relays per twin? is that even supported in zos nodes?

@muhamadazmy @AbdelrahmanElawady

yes, it is currently used to set the relay of twin destination in rmb-sdk-go here. so it would break compatibility with older versions.

also for the delay part, for clients that are using this new feature there should be no delay as cache will be updated with the requests made. for clients that are not using this field and changing relays there should be a delay up to 6 seconds until new blocks are produced.

This is already an improvement to the currently used relay warmer that adds a delay up to 10 minutes and previously before warmer it could take up to 60 seconds.

muhamadazmy · 2024-03-25T09:49:34Z

src/bins/rmb-relay.rs

+
+    let mut l = events::Listener::new(args.substrate, redis_cache).await?;
+    tokio::spawn(async move {
+        if let Err(e) = l.listen().await {


i see a problem here. If the listener failed for any reason (and returned an error here) It's not gonna be good for the system that the relay keeps running otherwise it will rely on out-dated date forever.

So what we need to do is:

Events listener failure is fatal and the entire system should then exit with an error

Event system never fails, it need to retry forever

If failure is persisted for sometime relay then should return errors to sender that they can't grantee deliver to destinations

I checked the listener code and it seems there are many points where the listener can return an error. for example cache.flush() but what if redis was temporary unavailable or restarting for some reason. This will cause the system to continue working but with a cache that will never be updated.

IMHO the best approach is to make the listener infailable. This can be accomplished but basically never give up on errors

muhamadazmy · 2024-03-25T10:01:12Z

src/events/events.rs

+    C: Cache<Twin> + Clone,
+{
+    pub async fn new(substrate_urls: Vec<String>, cache: C) -> Result<Self> {
+        let mut urls = LinkedList::new();


I am sure you can create a linked list directly from a Vec
will probably be something like

LinkedList::from_iter(substrate_urls)

just saying

muhamadazmy · 2024-03-25T10:13:32Z

src/events/events.rs

+        while let Some(block) = blocks_sub.next().await {
+            let events = block?.events().await?;
+            for evt in events.iter() {
+                let evt = evt?;


Why do u assume this is a connection error here. Although it can be encoding error (most probably) this mean that you are flushing the entire db, and reconnecting if you receive an bad event type

I am wondering if we better log this error and continue instead ? what do you think

muhamadazmy · 2024-03-25T10:15:03Z

src/events/mod.rs

@@ -0,0 +1,3 @@
+mod events;
+
+pub use events::Listener;


Check the warning from clippy i think since you don't need to use submodule instead just move the Listener to mod.rs directly

muhamadazmy · 2024-03-25T10:15:58Z

src/relay/api.rs

+            .await
+            .map_err(|err| HttpError::FailedToGetTwin(err.to_string()))?
+            .ok_or_else(|| HttpError::TwinNotFound(envelope.source.twin))?;
+        let envelope_relays = match RelayDomains::from_str(relays) {


check comments ont he proto buf schema

muhamadazmy · 2024-03-25T10:24:49Z

src/relay/api.rs

@@ -309,6 +331,25 @@ impl<M: Metrics, D: TwinDB> Stream<M, D> {
            .await?
            .ok_or_else(|| anyhow::Error::msg("unknown twin destination"))?;

+        if let Some(relays) = &envelope.relays {


I see the same code is repeating here and in federation as well. I mean this code is in the same file so I am sure you can create a function that u can reuse

Listen to chain events and update twin on requests

fae3a21

Add retry connections to chain

602adca

muhamadazmy requested changes Mar 25, 2024

View reviewed changes

AbdelrahmanElawady added 2 commits March 25, 2024 16:11

Make listener not fail on decode errors and db errors

d164d7a

Update twin relays

319af11

AbdelrahmanElawady force-pushed the development_listen_to_chain_events branch from 441e730 to 319af11 Compare March 26, 2024 08:34

AbdelrahmanElawady and others added 2 commits March 26, 2024 12:32

Use if let instead of match to check Err

deccc8e

run clippy

24498b7

muhamadazmy approved these changes Apr 5, 2024

View reviewed changes

muhamadazmy merged commit 610d871 into main Apr 5, 2024
1 check passed

muhamadazmy deleted the development_listen_to_chain_events branch April 5, 2024 09:25

This was referenced Apr 7, 2024

Update rmb-peer with the latest envelope data threefoldtech/tfgrid-sdk-ts#2539

Closed

Update rmb-peer with the latest envelope data threefoldtech/tfgrid-sdk-go#977

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Listen to chain events and update twin on requests #194

Listen to chain events and update twin on requests #194

AbdelrahmanElawady commented Mar 21, 2024

AbdelrahmanElawady commented Mar 21, 2024

AbdelrahmanElawady commented Mar 24, 2024

muhamadazmy left a comment

muhamadazmy Mar 25, 2024

sameh-farouk Mar 25, 2024 •

edited

Loading

AbdelrahmanElawady Mar 25, 2024

AbdelrahmanElawady Mar 25, 2024

muhamadazmy Mar 25, 2024

muhamadazmy Mar 25, 2024

muhamadazmy Mar 25, 2024

muhamadazmy Mar 25, 2024

muhamadazmy Mar 25, 2024

muhamadazmy Mar 25, 2024

Listen to chain events and update twin on requests #194

Listen to chain events and update twin on requests #194

Conversation

AbdelrahmanElawady commented Mar 21, 2024

Description

Issues

AbdelrahmanElawady commented Mar 21, 2024

AbdelrahmanElawady commented Mar 24, 2024

muhamadazmy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sameh-farouk Mar 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sameh-farouk Mar 25, 2024 •

edited

Loading