-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bug: can't discover peers when using DNS Discovery URL + shards #2162
Comments
I tried the following in nwaku:
Do notice that the following log line gets printed when using this configuration
Can confirm that nodes are not getting discovered, however if I remove the |
I created the following tool with go-waku to discover peers via discv5: https://github.com/waku-org/test-discv5/tree/master You can use any of these commands to execute it:
or
This will continuously try to discover peers, and print the enr, ip, port, multiaddresses, rs and rsv if available. If something is not available it wont be printed. In the results you can see that neither of the nodes have an
|
Doing more experiments with nwaku, if I remove the following lines of code Lines 90 to 91 in 3be6163
|
Looks like a Status problem not really Waku. Bootnodes without common shards should be filtered out. Same with nodes found through discv5.
|
I don't see this as priority for nwaku, as this is mostly relevant to Community clients relying on bootstrap nodes with ever-changing shard subscriptions - the go-waku fix will suffice for Status Communites. @chair28980 should we remove the epic label? |
While @chaitanyaprem and I were dogfooding the shards.test fleet, we noticed that only see a single store node being discovered, which was a strange behavior because this fleet has 6 store nodes according to https://fleets.status.im/ . After doing some investigation on why this is happening, I found out that the store nodes are actually not discovered at all. In status-go we have something called the mailserver cycle, that automatically chooses a store node based on ping reply time, to connect to it and retrieve message history. The reason why the store nodes are not being discovered via discV5 is because the dns discovery URL: https://github.com/status-im/infra-shards/blob/710444384b18f78e94eef62d8ac91b1322f6d333/ansible/group_vars/store.yml#L45C107-L46C25 After viewing the information of the nodes returned by this DNS discovery URL using https://github.com/waku-org/test-discv5, I saw that they enrs do not contain the shards we're interested into (for reasons explained in this issue), and while we fixed this in go-waku: https://github.com/waku-org/go-waku/blob/d7249fc123d3a27e3eb60b85d58d4d7a51df64a1/waku/v2/discv5/discover.go#L403-L405 and we can discover other go-waku peers, in nwaku the fix is not present ( Lines 67 to 69 in b31c182
I think this means that the store nodes and boot nodes are not connected to each other. I'm thinking we should add this to nwaku: https://github.com/waku-org/go-waku/blob/d7249fc123d3a27e3eb60b85d58d4d7a51df64a1/waku/v2/discv5/discover.go#L403-L405 to avoid this situation. |
The nodes returned by DNS DIscovery in
shards.test
don't have the information about the shards:These nodes are currently subscribed to shards:
/waku/2/rs/16/32
/waku/2/rs/16/64
/waku/2/rs/16/128
/waku/2/rs/16/256
However if you check the ENRs in https://enr-viewer.com/ these nodes lack the
rs
orrsv
attributes. Also, if you compare the current ENR of the nodes (using the nodes' RPC server), against those from the DNS Discovery URL, you'd see that the latter are outdated compared to the former, which makes sense since the DNS Discovery URL was manually created when the fleet was setup, while the subscription to shards is something that happens 'dynamically'.This is problematic because If i have a node subscribed to shards 32,64,128 and/or 256, if I use the dns discovery URL, I'll not be able to find new peers, because they'll get filtered out. DiscV5 will populate the routing tables with those shardless ENRs retrieved from the URL, and then they get filtered out, since Discv5 (at least in go-waku implementation, and looking at nwaku implementation seems to be similar) will not ask those nodes for their current ENR, and the filtering logic defined in this predicate:
nwaku/waku/waku_discv5.nim
Lines 53 to 75 in 3be6163
This is particularly problematic in the case of Status, since the DNS discovery URL is hardcoded in the node configuration.
The text was updated successfully, but these errors were encountered: