Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: store autosharding tests are not working with a cluster != 1 #1845

Closed
fbarbu15 opened this issue Feb 13, 2024 · 3 comments
Closed

bug: store autosharding tests are not working with a cluster != 1 #1845

fbarbu15 opened this issue Feb 13, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@fbarbu15
Copy link
Collaborator

This is a bug report

Problem

Because cluster-id 1 is used from now for WakuNetwork and only of on-chain connection we needed to update the tests.
After updating the cluster-id to 2 for the Waku Store (Autosharding), custom pubsub topic tests, they started to fail with no message retrieved by go-waku store query

Logs

Waku Store (Autosharding), custom pubsub topic
  waku:info:test:docker Confirming that image wakuorg/nwaku:v0.24.0 exists +0ms
  waku:info:test:docker Image wakuorg/nwaku:v0.24.0 successfully found +0ms
  waku:info:test:docker Running node with args: --listen-address=0.0.0.0 --rpc=true --relay=true --rpc-admin=true --websocket-support=true --log-level=TRACE --rpc-port=23128 --tcp-port=23129 --websocket-port=23130 --rpc-address=0.0.0.0 --store=true --pubsub-topic=/waku/2/rs/2/1 --pubsub-topic=/waku/2/rs/2/0 --cluster-id=2 +0ms
  waku:info:test:docker a9708ef7891a800f009c17eb665a8a3e7b3b67fb444a28b56154447df6065e57 started at 3:29:27 PM +461ms
  waku:info:test:node Waiting to see 'Node setup complete' in nwaku logs +0ms
  waku:info:test:node nwaku node has been started +10ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_subscriptions [ [ '/waku/2/content/utf8', '/myapp/1/latest/proto' ] ] +0ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +31ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'AA==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967401000000n
  }
] +0ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +7ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'AQ==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967409000000n
  }
] +1ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +5ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'Ag==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967416000000n
  }
] +2ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +5ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'Aw==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967422000000n
  }
] +1ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +6ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'BA==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967429000000n
  }
] +1ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +6ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'BQ==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967437000000n
  }
] +2ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +6ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'Bg==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967445000000n
  }
] +2ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +4ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'Bw==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967451000000n
  }
] +2ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +8ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'CA==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967460000000n
  }
] +1ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +6ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'CQ==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967468000000n
  }
] +2ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +5ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'Cg==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967475000000n
  }
] +2ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +5ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'Cw==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967481000000n
  }
] +1ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +5ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'DA==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967487000000n
  }
] +1ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +6ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'DQ==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967494000000n
  }
] +2ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +4ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'Dg==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967501000000n
  }
] +2ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +5ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'Dw==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967507000000n
  }
] +2ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +5ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'EA==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967515000000n
  }
] +2ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +5ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'EQ==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967522000000n
  }
] +2ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +5ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'Eg==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967529000000n
  }
] +2ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +5ms
  waku:info:test:node Making an RPC Query:  post_waku_v2_relay_v1_auto_message [
  {
    payload: 'Ew==',
    contentTopic: '/waku/2/content/utf8',
    timestamp: 1707830967535000000n
  }
] +1ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":true} +5ms
  waku:info:waku Waku node created 12D3KooWKZdeYhgXmPQjr1XEFYE9Dz6BEbZhDtQTPTGhLsqp6npo relay: false, store: true, light push: true, filter: true +0ms
  waku:info:connection-manager Connection Manager is now running +0ms
  waku:info:test:node Making an RPC Query:  get_waku_v2_debug_v1_info [] +16ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":{"listenAddresses":["/ip4/172.18.198.82/tcp/23129/p2p/16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W","/ip4/172.18.198.82/tcp/23130/ws/p2p/16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W"],"enrUri":"enr:-K64QLDcXsuBCkI3iGnGT8DtEM3zNinkHWjIn9uRflyYmfo2Ovo6dx9npuMu5BlGuqwKNEqvNFyvCBfayu9AlBPVrBYBgmlkgnY0gmlwhKwSxlKKbXVsdGlhZGRyc4wACgSsEsZSBlpa3QOCcnOHAAICAAEAAIlzZWNwMjU2azGhApZx-SX56MF-vgI5zfDf89NrODddp2-7qXc-n5w1D9SFg3RjcIJaWYV3YWt1MgM"}} +6ms
  waku:info:waku Dialing to /ip4/127.0.0.1/tcp/23130/ws/p2p/16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W with protocols store,filter,lightpush +8ms
  waku:info:connection-manager Connected to peer 16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W +137ms
  waku:error:connection-manager Failed to get peer 16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W, error: Error: Not Found +0ms
  waku:info:stream-manager:/vac/waku/metadata/1.0.0 Preemptively opening a stream to 16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W +0ms
  waku:warn:connection-manager Already connected to peer 16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W. Not dialing. +0ms
  waku:info:stream-manager:/vac/waku/metadata/1.0.0 Preemptively opening a stream to 16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W +4ms
  waku:info:stream-manager:/vac/waku/metadata/1.0.0 Preemptively opening a stream to 16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W +6ms
  waku:info:stream-manager:/vac/waku/metadata/1.0.0 Preemptively opening a stream to 16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W +7ms
  waku:info:stream-manager:/vac/waku/store/2.0.0-beta4 Preemptively opening a stream to 16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W +0ms
  waku:info:stream-manager:/vac/waku/metadata/1.0.0 Preemptively opening a stream to 16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W +1ms
  waku:info:stream-manager:/vac/waku/store/2.0.0-beta4 Preemptively opening a stream to 16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W +1ms
  waku:info:stream-manager:/vac/waku/metadata/1.0.0 Preemptively opening a stream to 16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W +5ms
  waku:info:stream-manager:/vac/waku/store/2.0.0-beta4 Preemptively opening a stream to 16Uiu2HAm5Yww5SsNWz6rRoqQjpN61BXtpqJ6PM1vUj2PEyc5Rs1W +5ms
  waku:info:test:node Making an RPC Query:  get_waku_v2_admin_v1_peers [] +220ms
  waku:info:test:node Received RPC Response:  {"jsonrpc":"2.0","id":1,"result":[{"multiaddr":"/ip4/127.0.0.1/tcp/33293/ws/p2p/12D3KooWKZdeYhgXmPQjr1XEFYE9Dz6BEbZhDtQTPTGhLsqp6npo","protocol":"/vac/waku/relay/2.0.0","connected":true}]} +3ms
  waku:info:test:store Waku node created +0ms
  waku:info:store Querying store peer for (/waku/2/rs/2/1) [ '/waku/2/content/utf8' ] +0ms
  waku:warn:store Stopping pagination due to store `response.messages` field missing or empty +0ms
    1) Generator, custom pubsub topic
@fbarbu15 fbarbu15 added the bug Something isn't working label Feb 13, 2024
@adklempner
Copy link
Member

I guess this is technically expected behavior since the cluster IDs do not match?

I see two options two resolve this:

  1. Don't do interop tests with cluster IDs besides 1 until other nodes add option to change cluster ID.
  2. Always use cluster ID 1 in js-waku. Remove option for library consumer to set cluster ID.

@fbarbu15
Copy link
Collaborator Author

Cluster ID 1 is reserved for Waku Network.
AFAIK we can't use this cluster from now on (recent versions of nwaku) on test envs because we need to have RLN enabled in order to publish messages.
Also other protocols are working fine with other cluster IDs, ex autosharded relay

@fbarbu15
Copy link
Collaborator Author

fbarbu15 commented Mar 4, 2024

After more digging, found out that if we are using cluster ID 1, the test works if we use the correct /relay/v1/auto/messages endpoint. There is a typo here (notice the missing s) and not sure how the test worked in the past.
For cluster IDs != 1 the issue (if any) is on nwaku side and will open a new issue on that repo.

@fbarbu15 fbarbu15 closed this as completed Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Waku
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@adklempner @fbarbu15