bug: Lightpush.send() failure and node disconnection beyond 32 content topics on nwaku v0.25.0

[danisharora099]

Problem

When testing with js-waku against nwaku version 0.25.0, a discrepancy in handling lightpush requests beyond 30 content topics was observed, despite recent support extension to 100 content topics. Specifically, lightpush.send() fails starting from the 33rd execution, resulting in a disconnection from the nwaku node. This behavior is unexpected and persists across different nwaku images tested.

Impact

Significant

To reproduce

1. Set up a nwaku node using the harbor.status.im/wakuorg/nwaku:latest image or version 0.25.0.
2. Using js-waku, subscribe to more than 30 content topics.
3. Attempt to send lightpush messages beyond the 32nd message.
4. Observe the failure in lightpush.send() and eventual disconnection from the nwaku node.

PR can be used to repro: waku-org/js-waku#1857: npm i && npm run build && cd packages/tests && npm run test:node

Expected behavior

The expected behavior was for js-waku to successfully send lightpush messages for up to 100 content topics without encountering disconnections or failures, as per the recent extension of support in nwaku.

Screenshots/logs

Logs indicate that after the 32nd lightpush message, lightpush.send() fails with "Remote peer fault: No response in PushRPC", followed by disconnection errors and warnings related to peer availability.

nwaku version/commit hash

The issue was observed in nwaku version 0.25.0.

Additional context

The problem was initially discussed in a chat thread with nwaku engineers, where it was suggested that the issue could be related to ws stream reset after 32 messages or a potential issue with the mplex configuration. No RLN is used, and the setup was over the default pubsub topic.
Original Discord thread: https://discord.com/channels/1110799176264056863/1201820015284723753

[NagyZoltanPeter]

@danisharora099 Can you attach logs in greater context of the issue? I mean from the last few successful lightpush send.

[weboko]

@NagyZoltanPeter can you check logs here, they should be sufficient enough

[gabrielmer]

The root cause seems to be in js-libp2p, in the DEFAULT_MAX_INBOUND_STREAMS const defined here

When running the test, got the following log

  libp2p:mplex:stream:receiver:36 abort with error CodeError: Too many inbound protocol streams for protocol "/vac/waku/filter-push/2.0.0-beta1" - limit 32
    at file:///Users/gabrielmer/status/workspace2/js-waku/node_modules/libp2p/src/upgrader.ts:388:29
    at processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 'ERR_TOO_MANY_INBOUND_PROTOCOL_STREAMS',
  props: {}
} +0ms

If we configure a higher limit of input streams in js-libp2p, the bug is fixed

Opened the following PR in js-waku

[danisharora099]

The root cause seems to be in js-libp2p, in the DEFAULT_MAX_INBOUND_STREAMS const defined here

When running the test, got the following log

  libp2p:mplex:stream:receiver:36 abort with error CodeError: Too many inbound protocol streams for protocol "/vac/waku/filter-push/2.0.0-beta1" - limit 32
    at file:///Users/gabrielmer/status/workspace2/js-waku/node_modules/libp2p/src/upgrader.ts:388:29
    at processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 'ERR_TOO_MANY_INBOUND_PROTOCOL_STREAMS',
  props: {}
} +0ms

If we configure a higher limit of input streams in js-libp2p, the bug is fixed

Opened the following PR in js-waku

Sounds right!
Each request opens a new stream, so seems only right for the limit to be increased.

Thank you @gabrielmer !

[gabrielmer]

Fixed by waku-org/js-waku#2021