bug/regression: some lightnodes are not receiving filter push in certain conditions

[fbarbu15]

Problem

When having a setup with multiple nodes, one relay and others filter(without relay), some go-waku nodes are not receiving the relayed message via filter push.
This doesn't happens with wakuorg/nwaku:v0.25.0
It was reproduced with both harbor.status.im/wakuorg/nwaku:latest and quay.io/wakuorg/nwaku-pr:2493 so it may be unrelated with #2491
Ex setup

node1 - nwaku latest - relay + filter
node2 - nwaku latest - filter
node3 - go-waku latest - filter
node4 - go-waku latest - filter
node5 - go-waku latest - filter

It seems to depend also on node2. If instead of nwaku latest I set node2 as go-waku latest the issue doesn't reproduce anymore

To reproduce

1. Start nodes like in the above example and connect them via discv5
2. Filter subscribe all nodes to the same topics
3. Publish one message from node1
4. Check using filter GET each of the filter nodes for the pushed message

Expected behavior

All nodes should receive the message

Actual behavior

Some nodes do not get the message and I see no filter push in the logs
node4 and 5 don't receive the pushed message_nwaku_latest.zip
node4 doesn't receive the pushed message_pr2493.zip

[richard-ramos]

@fbarbu15 I tried the following scenario succesfully using latest master in both nwaku and go-waku:

export YOUR_IP_GOES_HERE=192.168.0.100

# NODE 1 - NWAKU
clear && ./build/wakunode2 --rest --filter --relay --pubsub-topic=/waku/2/default-waku/proto --tcp-port=61000 --nodekey=1122334455667788990011223344556677889900112233445566778899001122 --discv5-discovery --nat=extip:$YOUR_IP_GOES_HERE
 
export NODE1_ENR_GOES_HERE=............................

# NODE 2 - NWAKU
clear && ./build/wakunode2 --rest --rest-port=8646 --filter --tcp-port=61100 --discv5-udp-port=22343 --nodekey=1122334455667788990011223344556677889900112233445566778899001133 --nat=extip:$YOUR_IP_GOES_HERE --discv5-discovery --discv5-bootstrap-node=$NODE1_ENR_GOES_HERE

# NODE 3 - GOWAKU
clear && ./build/waku --rest --rest-port=8647 --filter --tcp-port=61200 --discv5-udp-port=22344 --nodekey=1122334455667788990011223344556677889900112233445566778899001144 --discv5-discovery --discv5-bootstrap-node=$NODE1_ENR_GOES_HERE

# NODE 4 - GOWAKU
clear && ./build/waku --rest --rest-port=8648 --filter --tcp-port=61300 --discv5-udp-port=22346 --nodekey=1122334455667788990011223344556677889900112233445566778899001155 --discv5-discovery --discv5-bootstrap-node=$NODE1_ENR_GOES_HERE
 
# NODE 5 - GOWAKU
clear && ./build/waku --rest --rest-port=8649 --filter --tcp-port=61400 --discv5-udp-port=22347 --nodekey=1122334455667788990011223344556677889900112233445566778899001166 --discv5-discovery --discv5-bootstrap-node=$NODE1_ENR_GOES_HERE

Sent the following REST calls to subscribe the nodes to NODE 1:

curl -vv -X POST "http://localhost:8646/filter/v2/subscriptions" -H "accept: text/plain" -H "content-type: application/json" -d '{"requestId":"146546546","contentFilters":["/my-app/2/chatroom-1/proto"], "pubsubTopic": "/waku/2/default-waku/proto"}'

curl -vv -X POST "http://localhost:8647/filter/v2/subscriptions" -H "accept: text/plain" -H "content-type: application/json" -d '{"requestId":"146546546","contentFilters":["/my-app/2/chatroom-1/proto"], "pubsubTopic": "/waku/2/default-waku/proto"}'

curl -vv -X POST "http://localhost:8648/filter/v2/subscriptions" -H "accept: text/plain" -H "content-type: application/json" -d '{"requestId":"146546546","contentFilters":["/my-app/2/chatroom-1/proto"], "pubsubTopic": "/waku/2/default-waku/proto"}'

curl -vv -X POST "http://localhost:8649/filter/v2/subscriptions" -H "accept: text/plain" -H "content-type: application/json" -d '{"requestId":"146546546","contentFilters":["/my-app/2/chatroom-1/proto"], "pubsubTopic": "/waku/2/default-waku/proto"}'

Then, I sent a message from node1:

curl -X POST "http://127.0.0.1:8645/relay/v1/messages/%2Fwaku%2F2%2Fdefault-waku%2Fproto"  -H "content-type: application/json"  -d '{"payload":"aGVsbG8gd29ybGQ","contentTopic":"/my-app/2/chatroom-1/proto", "timestamp":'$(date +%s%N)'}'

And retrieved the messages:

curl -vv -X GET "http://localhost:8646/filter/v2/messages/%2Fmy-app%2F2%2Fchatroom-1%2Fproto"

curl -vv -X GET "http://localhost:8647/filter/v2/messages/%2Fwaku%2F2%2Fdefault-waku%2Fproto/%2Fmy-app%2F2%2Fchatroom-1%2Fproto"

curl -vv -X GET "http://localhost:8648/filter/v2/messages/%2Fwaku%2F2%2Fdefault-waku%2Fproto/%2Fmy-app%2F2%2Fchatroom-1%2Fproto"

curl -vv -X GET "http://localhost:8649/filter/v2/messages/%2Fwaku%2F2%2Fdefault-waku%2Fproto/%2Fmy-app%2F2%2Fchatroom-1%2Fproto"

Do notice that in nwaku, it is not possible to specify the pubsub topic when retrieving the messages. I think this setup is similar to what you have. However I wasn't able to reproduce the problem. I'm probably having a difference in the setup, so please indicate if i'm missing some flag.

[richard-ramos]

Ah, something else I had to do was to use --nat=extip:$YOUR_IP_GOES_HERE, otherwise, the enr would contain the ip 0.0.0.0 and I would not be able to connect to NODE 1

[fbarbu15]

@richard-ramos thanks for script but was unfortunately unable to build go-waku to try it...
However I've provided a similar raw script that reproduces the bug but uses docker containers instead.
For me it usually fails 70% of the times, and when it fails it can be either NODE 3, NODE 4 or NODE 5 not receiving the messages.
Let me know if anything is missing. Thanks
bug_2512.zip

[richard-ramos]

I found out what's the problem!

In the script that uses Docker, the nodes use the flag --filter, meaning that it will mount the filter-subscribe protocol.
Go-waku for the peer selection relies on its peerstore which is populated both by the --filternode flags and also by discovery

In this scenario what's happening is that the go-waku nodes are discovering the peer via discv5. This peer's ENR contains in the waku field the flag for Filter enabled, and then, when you use the filter/v2/subscriptions REST endpoint, the automatic peer selection happens and it will choose randomly from any of the discovered nodes that have the --filter flag enabled.

It's not really necessary to use the --filter flag, for Nodes 2..5, as these will act as clients, and they don't have relay enabled. Unless it's expected for a filter client to be able to act as a filter node for other nodes? (i.e. all messages they receive via filter will be forwarded to their subscribers).

[fbarbu15]

Thanks, this indeed fixed the tests.
However I feel that this is something that can happen to other node operators IRL. It's logical to use the filter flag when you are using the filter protocol

[richard-ramos]

Depends on the point of view. Using a flag like --filter, --store, --peer-exchange means that you're supporting the 'server' side of the protocol (i.e. providing support for other nodes to consume the protocol), however, acting as a client is always possible as these features do not require mounting a protocol to consume from other nodes.