feat: Deprecate Named Sharding and Update Lightpush Client API

[chair28980]

In the context of discussions around named sharding and peer management for the Waku network, it is proposed to make the following changes to the Waku Network:

1. Deprecate Support for Named Sharding:

• It is recommended to remove support for named sharding.
• Peer management should ensure that only peers supporting sharded topics are connected.

2. Lightpush Protocol Updates:

• The lightpush protocol itself should never be aware of the sharded format of pubsub topics. It is an extension of libp2p's gossipsub.
• On a protocol level, the lightpush service node needs to handle requests on unsupported pubsub topics by returning a clear error code.

3. Lightpush Client API Changes:

• To maintain adherence to the sharded format, the lightpush client API should be updated.
• New proposed format: lightPublish(WakuMessage, ClusterID, ShardNumber)
• For legacy requests on named topics, the API could use a trial and error process to send the request to random service nodes. This can be done using the format: lightPublish(WakuMessage, PubsubTopic)
• A request format that bypasses our peer management/selection: lightPublish(WakuMessage, PubsubTopic, ServiceNodeMultiaddr)

---

• Users of custom pubsub topics should select a cluster id.
• Use static sharding until the launch of The Waku Network.
• Default Pubsub topic support remains for now, to be phased out as a later step

Acceptance Criteria:

• Remove support for named sharding (except for default pubsub topic)
• Update lightpush protocol to handle unsupported pubsub topics with clear error codes.
• Introduce the new lightpush client API as mentioned above.
• Update documentation to reflect these changes and guide users accordingly.

[danisharora099]

To maintain adherence to the sharded format, the lightpush client API should be updated.
New proposed format: lightPublish(WakuMessage, ClusterID, ShardNumber)
For legacy requests on named topics, the API could use a trial and error process to send the request to random service nodes. This can be done using the format: lightPublish(WakuMessage, PubsubTopic)
A request format that bypasses our peer management/selection: lightPublish(WakuMessage, PubsubTopic, ServiceNodeMultiaddr)

For js-waku, we use Encoders and Decoders to encapsulate the pubsub topic -- IMO: this change should be added on the createEncoder and createDecoder, and converted to a string for the protocol.
All the error handling should be done on the createEncoder/Decoder function.
An example of this looks like: a75a66b

cc @fryorcraken

[danisharora099]

Weekly Update

• achieved: PR ready and currently in review-iteration phase ( feat!: add support for sharded pubsub topics & remove support for named pubsub topics #1697 )
• next: to be merged

[fryorcraken]

FYI #1749 (comment)

[chair28980]

Planning to move this issue to Priority column in week 3 of January 2024.

[fryorcraken]

Note that Status is contemplating using named shard so I don't think we can fully deprecate.

[danisharora099]

Note that Status is contemplating using named shard so I don't think we can fully deprecate.

Moving this back to triage. Once we have confirmation in either direction, we can handle this issue as required.

[chair28980]

Moving to icebox, standing by for update from Status to confirm this update is required or not.

@fryorcraken do you know who the appropriate stakeholder on Status team is to verify the necessity of deprecating named sharding?

[fryorcraken]

Moving to icebox, standing by for update from Status to confirm this update is required or not.

@fryorcraken do you know who the appropriate stakeholder on Status team is to verify the necessity of deprecating named sharding?

We are stakeholders now.

Status software uses static sharding all across. Even if not custom shards just yet.

Named sharding can be deprecated across the clients.

cc @jm-clius @Ivansete-status @weboko

[jm-clius]

Indeed. There should be no more use of named sharding in any clients.

[danisharora099]

Thanks for the clarification! Sounds great.

Named sharding can be deprecated across the clients.

This does remain blocked on js-waku until we allow js-waku to connect/work with The Waku Network.

cc @weboko keeping this in blocked