Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

serving request {"endpoint": "/namespaced_data", "err": "getter/shrex: client-nd: verifying response: row verification failed: row 0 doesn't match original root: ... #2066

Open
Tracked by #2106
jcstein opened this issue Apr 13, 2023 · 3 comments
Open
Tracked by #2106

serving request {"endpoint": "/namespaced_data", "err": "getter/shrex: client-nd: verifying response: row verification failed: row 0 doesn't match original root: ... #2066

jcstein opened this issue Apr 13, 2023 · 3 comments
Assignees
@walldiss
Labels
bug Something isn't working external Issues created by non node team members

Comments

@jcstein
Copy link
Member

jcstein commented Apr 13, 2023

Celestia Node version

v0.8.2

OS

Ubuntu

Install tools

No response

Others

No response

Steps to reproduce it

Running https://github.com/rollkit/gm on top of this, occasionally see this error

Expected result

No error

Actual result

2023-04-13T02:44:32.375Z        INFO    header/store    store/store.go:353      new head        {"height": 236317, "hash": "2043EB2C8A48DC2B4E84EF608B5259A03242924AB2FF8379CCE67AE6A600811A"}
2023-04-13T02:44:32.376Z        INFO    das     das/subscriber.go:35    new header received via subscription      {"height": 236317}
2023-04-13T02:44:32.472Z        INFO    das     das/worker.go:80        finished sampling headers{"from": 236317, "to": 236317, "errors": 0, "finished (s)": 0.096353672}
2023-04-13T02:44:34.594Z        ERROR   shrex/nd        shrexnd/client.go:195   client-nd: request status INTERNAL returned
2023-04-13T02:44:34.616Z        ERROR   gateway gateway/util.go:9       serving request {"endpoint": "/namespaced_data", "err": "getter/shrex: client-nd: verifying response: row verification failed: ro
w 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row veri
fication failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying
 response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; c
lient-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\u
fffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\u
fffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\uff
fdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\u
fffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufff
d\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u00
12\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original roo
t: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: response code is not OK: server returned an invalid respon
se or caused an internal error; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\u
fffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,
.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\uff
fdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\uf
ffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \u
fffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't matc
h original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed:
 row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; failed to find peers: routing: not f
ound; context deadline exceeded; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\
ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; context deadline exceeded; getter/ipld: failed to retrieve shares by namespace: retrieving nID 05fafd7808fd77a7 for row 626167616f34616d6235796174616
16161616161616161616161716d79327569636f7a366266673662677672736575356f616b6f64367133727a616a35687a76616f36763670677474727471783276746f37647a62376467667979: context canceled", "errCauses": [{"error": "ge
tter/shrex: client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u
0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\uf
ffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.
ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; context deadline exceeded; getter/ipld: failed to retrieve shares by namespace: retrieving nID 05fafd7808fd77a7 for row 626167616f34616d6235796174616
16161616161616161616161716d79327569636f7a366266673662677672736575356f616b6f64367133727a616a35687a76616f36763670677474727471783276746f37647a62376467667979: context canceled", "errCauses": [{"error": "ge
tter/shrex: client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u
0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\uf
ffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.
\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufff
dB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\uff
fd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \uf
ffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match
 original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed:
row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row ve
rification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifyi
ng response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd;
 client-nd: response code is not OK: server returned an invalid response or caused an internal error; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\u
fffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match orig
inal root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0
 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verific
ation failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying re
sponse: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; clie
nt-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufff
d\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufff
d\u0011\u00107\ufffd\ufffd\ufffd; failed to find peers: routing: not found; context deadline exceeded; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\
ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; context deadline exceeded"}, {"error": "getter/ipld: failed to retrieve shares
by namespace: retrieving nID 05fafd7808fd77a7 for row 626167616f34616d623579617461616161616161616161616161716d79327569636f7a366266673662677672736575356f616b6f64367133727a616a35687a76616f367636706774747
27471783276746f37647a62376467667979: context canceled"}]}
2023-04-13T02:44:43.315Z        INFO    net/identify    identify/id.go:396      failed negotiate identify protocol with peer      {"peer": "12D3KooWARVnGZsAUCUZ9UhU94tsvW3caK2WpCnwpqUqkKeVNJG8", "error
": "Application error 0x0 (local)"}
2023-04-13T02:44:43.315Z        WARN    net/identify    identify/id.go:371      failed to identify 12D3KooWARVnGZsAUCUZ9UhU94tsvW3caK2WpCnwpqUqkKeVNJG8: Application error 0x0 (local)

Relevant log output

No response

Notes

No response
@jcstein jcstein added the bug Something isn't working label Apr 13, 2023
@github-actions github-actions bot added the external Issues created by non node team members label Apr 13, 2023
@jcstein jcstein changed the title serving request {"endpoint": "/namespaced_data", "err": "getter/shrex: client-nd: verifying response: row verification failed: ro w 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row veri fication failed: row 0 doesn't match original serving request {"endpoint": "/namespaced_data", "err": "getter/shrex: client-nd: verifying response: row verification failed: row 0 doesn't match original root: ... Apr 13, 2023
@Wondertan
Copy link
Member

Oooops, this is very concerning

@Bidon15 Bidon15 mentioned this issue Apr 13, 2023
Closed
@renaynay renaynay mentioned this issue Apr 13, 2023
Closed
@jcstein
Copy link
Member Author

jcstein commented Apr 13, 2023 •
edited
Loading

Link to issue: #2051

This is also related from a user in discord:
image
Thread: https://discord.com/channels/638338779505229824/1093868258802814986/1094027383713046568

@jcstein
Copy link
Member Author

jcstein commented Apr 13, 2023

And here rollkit/rollkit#861

@walldiss walldiss mentioned this issue Apr 20, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@walldiss walldiss

Labels
bug Something isn't working external Issues created by non node team members
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@walldiss @Wondertan @jcstein