bug(share/p2p/shrexnd): Request from light node to blockspacerace for GetSharesByNamespace hangs forever

[renaynay]

Problem

Performing a GetSharesByNamespace request will hang forever.

2023-04-06T16:52:26.460+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWHzEtY3x7tCZDKvBpvWZjEkJtsEKCZegbEa9Rngtg3cXe", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:52:26.564+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWBct27GArLQoZZLwnShT7DnaXmNZpEiK8oQPvNuZrptp8", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:52:28.301+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWRmzCosNSVeeZnwXwm9JSmcq8CkhMYAG6J2PUscpuK7t2", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:52:29.232+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWBbc9ixMAH4SzF3EH6SwSfo8gRW1bVrDUX19oxC4DaRbz", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:52:29.590+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWBct27GArLQoZZLwnShT7DnaXmNZpEiK8oQPvNuZrptp8", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:52:31.416+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": 

also at a certain point, it will begin to request the same peer over and over

2023-04-06T16:55:41.256+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:55:44.289+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:55:46.364+0200    INFO    header/store    store/store.go:353      new head        {"height": 184565, "hash": "28C9563444EE06B692CC846DE131E9AB85BB400AEB309350278EBFF0469C1BEA"}
2023-04-06T16:55:46.367+0200    INFO    das     das/subscriber.go:35    new header received via subscription    {"height": 184565}
2023-04-06T16:55:46.480+0200    INFO    das     das/worker.go:80        finished sampling headers       {"from": 184565, "to": 184565, "errors": 0, "finished (s)": 0.113163417}
2023-04-06T16:55:47.319+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:55:50.352+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:55:53.387+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:55:56.417+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:55:57.060+0200    INFO    header/store    store/store.go:353      new head        {"height": 184566, "hash": "FF864BC83161C5A85C89BCD067C16EBD18AADF5DD359EB5778563F30D3795FED"}
2023-04-06T16:55:57.060+0200    INFO    das     das/subscriber.go:35    new header received via subscription    {"height": 184566}
2023-04-06T16:55:57.126+0200    INFO    das     das/worker.go:80        finished sampling headers       {"from": 184566, "to": 184566, "errors": 0, "finished (s)": 0.066025417}
2023-04-06T16:55:59.446+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:56:02.481+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:56:05.515+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}

Steps to reproduce

1. Run light node with --log.level.module shrex/nd:debug against blockspacerace
2. Submit a PFB to blockspacerace
3. Wait til light node syncs up until height at which PFB was included, and do a shares.GetSharesByNamespace for the nID under which PFB was submitted with the DAH for that given height
4. request will hang, check out node logs

[renaynay]

#2044 related

[renaynay]

#2066 related

[renaynay]

Steps to reproduce

1. Run a light node against blockspacerace or a robusta network (something that reliably has non-empty blocks)
2. Either submit a PFB or if you already know of a PFB at a certain height, you can query for that
3. GetSharesByNamespace request (either via RPC or gateway)