Static Sharding

[fryorcraken]

Planned start date:
Due date: 15 Oct 2023

Summary

Implement Static Sharding as specified in https://rfc.vac.dev/spec/51/#static-sharding

Acceptance Criteria

As an app dev, I can...

• specify the static shard I am using so that my node connects to nodes serving said shard
• use filter for a given static shard
• use lightpush for a given static shard
• use store for a given static shard
• use relay for a given static shard

Tasks

Taking into account #1430 (comment), where we assume that shards will not change mid-run; we can update the sequence diagram as:

sequenceDiagram
    participant Alice
    participant js-waku
    participant connection manager

    Alice->>js-waku: start (pass [`shard1`, `shard2`])
    js-waku->>connection manager: discovers & attempts connections to relevant peers
    Alice->>js-waku: send a subscription request that includes a pubsubtopic based on IDecoder
    js-waku->>js-waku: if the new pubsubtopic is different from initialisation, throw error
    js-waku->>js-waku: if the new pubsubtopic is one of the ones from initialisation, proceed
    js-waku->>connection manager: open a stream with one of the connected peers from the relevant shard
    js-waku->>Alice: return the subscription object 

this definitely narrows our scope of work (v1) but we can design our system as above discussed where we think of it being potentially scaled up to being used for run-time changes as well (v2?)

---

Tasks (v1)

Create

• take in pubsubTopics during the creation of node: feat(static-sharding)!: allow multiple pubSubTopics #1586
  • eg: const node = createRelayNode({ cluster: M, shards: [N, O] });

Discovery

• when ENR is decoded within discovery classes, find its rs/rsv and add that information in PeerStore for easier access feat(static-sharding): filter peer connections per shards #1626

ConnectionManager

• before peers are attempted to dial within the Manager, check if the peer corresponds to a relevant shard by reading its rs/rsv value & then dial them accordingly feat(static-sharding): filter peer connections per shards #1626

Protocols

• within individual protocols, when a request is sent, ensure that the pubsubTopic for that request is for a shard that the js-waku node was setup with during initialisation: feat(static-sharding)!: allow multiple pubSubTopics #1586
  • else, we throw an error

RAID (Risks, Assumptions, Issues and Dependencies)

[danisharora099]

some learnings:
js-waku being a client and not actually participating in discv5 discovery process, the implementation would be mostly limited to filtering out the right nodes from discovery protocols; in terms of peer exchange, the goal is to connect only with peers from the PX response that share the same pubsub topic or shard that the relay protocol is subscribed to, while discarding other peers received in the response

cc @fryorcraken keen to know your perspective on the above

[danisharora099]

note: is currently blocked on nwaku and go-waku as while they have the functionality implemented, they are not exposed to the user yet through a flag

[danisharora099]

Child issue: #1430

[fryorcraken]

some learnings: js-waku being a client and not actually participating in discv5 discovery process, the implementation would be mostly limited to filtering out the right nodes from discovery protocols; in terms of peer exchange, the goal is to connect only with peers from the PX response that share the same pubsub topic or shard that the relay protocol is subscribed to, while discarding other peers received in the response

cc @fryorcraken keen to know your perspective on the above

that and we also need to implement the API to enable static and naming sharding on filter, light push, store and relay. Which is the discussion I saw on Discord.

[fryorcraken]

Note this is needed for the "community preview" web portal of Status. Cc @danisharora099 @chair28980 @John-44 @prichodko @felicio

[John-44]

Note this is needed for the "community preview" web portal of Status. Cc @danisharora099 @chair28980 @John-44 @prichodko @felicio

and Community Channel Preview web pages

[danisharora099]

some notes for design:

•  specify the static shard I am using so that my node connects to nodes serving said shard
  • allow ability to users to subscribe/unsubscribe to shards
    • a new API that allows for sub/unsub
      • currently we only allow consumers to set a pubsubTopic when they wanna start a node beforehand
    • look into where we can maintain this state (preferably)
  • allow connectivity of peers that support a particular shard
    • maintain peers that are discovered but dont belong in a relevant shard for later
      • let's do this as it costs little-to-nothing for us, but acts as a potential big ROI for library consumers

[danisharora099]

@weboko and I discussed how the user flow might look like for subscription & unsubscription from the user's end where we assume Alice will pass the shard during the initialisation of the node:

sequenceDiagram
    participant Alice
    participant js-waku
    participant connection manager

    Alice->>js-waku: start (pass [`shard1`, `shard2`])
    js-waku->>connection manager: discovers & attempts connections to relevant peers
    connection manager->>connection manager: cache peers for irrelevant static shards
    Alice->>js-waku: subscribe to a new pubsubtopic (`shard3`) based on IDecoder (that includes pubsubTopic)
    js-waku->>js-waku: if the new IDecoder is for static sharding (following topic name convention)
    js-waku->>connection manager: attempt dial
    connection manager->>connection manager: check if we already have a connected peer for this shard & use this peer
    connection manager->>connection manager: else, loop over discovered peers and attempt a dial to new peer
    js-waku->>Alice: if no peers are found for that shard -- throw error
    js-waku->>Alice: else, return the subscription object 
    Alice->>js-waku: unsubscribe (shard1)
    js-waku->>connection manager: drop connection with peers that exclusively support that shard

[weboko]

My view here is that there are the following sub-tasks:

• enable createWakuNode({ shards });
• enable discover peers, cache, and mark in PeerStore shards relevant to it;
• update to IEncoder & IDecoder to facilitate the passing of shard in ISubscriber;
• update unsubscribe so that if no listeners to a shard are found - drop connection to a peer which was used for that shard exclusively;

[danisharora099]

Taking into account #1430 (comment), where we assume that shards will not change mid-run; we can update the sequence diagram as:

sequenceDiagram
    participant Alice
    participant js-waku
    participant connection manager

    Alice->>js-waku: start (pass [`shard1`, `shard2`])
    js-waku->>connection manager: discovers & attempts connections to relevant peers
    Alice->>js-waku: send a subscription request that includes a pubsubtopic based on IDecoder
    js-waku->>js-waku: if the new pubsubtopic is different from initialisation, throw error
    js-waku->>js-waku: if the new pubsubtopic is one of the ones from initialisation, proceed
    js-waku->>connection manager: open a stream with one of the connected peers from the relevant shard
    js-waku->>Alice: return the subscription object 

this definitely narrows our scope of work (v1) but we can design our system as above discussed where we think of it being potentially scaled up to being used for run-time changes as well (v2?)

---

Tasks (v1)

Create

• take in pubsubTopics during the creation of node: feat(static-sharding)!: allow multiple pubSubTopics #1586
  • eg: const node = createRelayNode({ cluster: M, shards: [N, O] });

Discovery

• when ENR is decoded within discovery classes, find its rs/rsv and add that information in PeerStore for easier access

ConnectionManager

• before peers are attempted to dial within the Manager, check if the peer corresponds to a relevant shard by reading its rs/rsv value & then dial them accordingly

Protocols

• within individual protocols, when a request is sent, ensure that the pubsubTopic for that request is for a shard that the js-waku node was setup with during initialisation: feat(static-sharding)!: allow multiple pubSubTopics #1586
  • else, we throw an error

---

let me know what you think @weboko @fryorcraken

[fryorcraken]

Looks good. Would be good to have a review from @waku-org/go-waku-developers and @waku-org/nwaku-developers if they see any issue from their experience/implementaiton.

Agreed that for static sharding, we can assume that the static shard does not change after a start to support the Community Preview portal and Community Channel web pages for Status. Cc @felicio @prichodko

Do note that with auto-sharding, shards can change mid run (but out of scope for this issue).

[danisharora099]

Weekly Update

• achieved: allowing for multiple pubsub topics to be configured & refactoring protocols to support
• next: enabling peer management to only dial relevant shards

[danisharora099]

Weekly Update

• achieved: PR open for allowing peer management for multiple pubsub topics/shard
• next: getting reviews & releasing

[danisharora099]

Weekly Update

• achieved: peer management has been merged to master and the Static Sharding milestone is now complete

[danisharora099]

reopening this in reference to #1684 to be done

[danisharora099]

Closing considering #1684 is done