debug_traceBlockByHash returns "method handler crashed" for certain blocks

[aaronbuchwald]

Moving this issue from AvalancheGo, since this is caused by a bug in Coreth: ava-labs/avalanchego#1689

Hello,

We have been running multiple Avalanche Archive nodes (both mainnet and testnet) for quite some time but lately we've had some issues showing up.

For Mainnet:

Example RPC Call:

curl http://127.0.0.1:9650/ext/bc/C/rpc -X POST -H "Content-Type: application/json" --data '{"method":"debug_traceBlockByHash","params":["0xef836031411b2c15ef330ce28911aaf6de1995ccbdf45419fbfd4c43aa992a3e", {"tracer": "callTracer", "tracerConfig": {"onlyTopCall":false,"withLog":true}}],"id":1,"jsonrpc":"2.0"}'

Returns:

{"jsonrpc":"2.0","id":1,"error":{"code":-32603,"message":"method handler crashed"}}

For Testnet:

Example RPC Call:

curl https://127.0.0.1:9650/ext/bc/C/rpc -X POST -H "Content-Type: application/json" --data '{"method":"debug_traceBlockByHash","params":["0x260238a8b034c48d2b48f40c3a34587077aefc012a96abd026e116c594c0a688", {"tracer": "prestateTracer", "tracerConfig": {"onlyTopCall": false, "diffMode": true}}],"id":1,"jsonrpc":"2.0"}'

Returns:

{"jsonrpc":"2.0","id":1,"error":{"code":-32603,"message":"method handler crashed"}}

To give some context about the node setup environment, we are running the following binary version: avalanche/1.10.3 [database=v1.4.5, rpcchainvm=26, commit=eb6e7973a900dc647d29a2da8e7a233ac0a5fe77] and we have synced the nodes from scratch, given the fact that there were no official snapshots available at that time.

We were wondering whether anyone else has seen this problem before or not. Any ideas would be highly appreciated. Thanks in advance!

[tibineacsu95]

Hi there @aaronbuchwald!

I see that the issue has been closed, the response of the nodes has changed to the below:

curl http://127.0.0.1:9650/ext/bc/C/rpc -X POST -H "Content-Type: application/json" --data '{"method":"debug_traceBlockByHash","params":["0xef836031411b2c15ef330ce28911aaf6de1995ccbdf45419fbfd4c43aa992a3e", {"tracer": "callTracer", "tracerConfig": {"onlyTopCall":false,"withLog":true}}],"id":1,"jsonrpc":"2.0"}'
{"jsonrpc":"2.0","id":1,"result":[{"txHash":"0xc8e59a9b6ca253e9e28f4ea3dde5a675cb6bdb93f593c4f2c441d38029e30607","result":{"from":"0xc873c70deff6f2e05e9709da70c910a127d5f107","gas":"0x2e21e","gasUsed":"0x2e21e","input":"0xa2a1623d000000000000000000000000000000000000000000000000016345785d8a00000000000000000000000000000000000000000000000000000000000000000080000000000000000000000000c873c70deff6f2e05e9709da70c910a127d5f107000000000000000000000000000000000000000000000000000000006093973c0000000000000000000000000000000000000000000000000000000000000002000000000000000000000000b31f66aa3c1e785363f0875a1b74e27b85fd66c70000000000000000000000006e7f5c0b9f4432716bdd0a77a3601291b9d9e985","error":"stack underflow (0 \u003c=\u003e 4)","value":"0x16345785d8a0000","type":"CREATE"}}]}

Avalanchego Version: avalanche/1.10.7 [database=v1.4.5, rpcchainvm=27, commit=aa481ddce393fbb3ab957e4c8f87d5cd50b5eb4e]

I am not quite sure that's what we should expect to get here, can you please take a look whenever you get the time? Any feedback is highly appreciated. Thanks a lot!

[anusha-ctrl]

Hey @tibineacsu95,

This is to be expected! If we look at this transaction on snow trace we can see that the transaction failed due to stack underflow.

In general, this JSON RPC call is successful since it returned a result instead of an error. The stack underflow error you see was the error from the transaction itself.

Thanks and let me know if that addresses the issue.

[tibineacsu95]

Hey @anusha-ctrl !
Thank you for your explanation! In that case we will proceed with upgrading all of our nodes. It really does make sense to be expected, I just thought I should double check. Really appreciate it!