Unsigned transaction execution failure when interacting with blockchain node #2019

RedCuckoo · 2023-04-15T00:36:31Z

System information

zkEVM Node version: v0.0.3-RC36
OS & Version: Linux
Commit hash : latest released tag
Network: Mainnet

Expected behaviour

When executing unsigned transactions, I am successfully getting a response by calling read functions of the contract deployed on the mainnet.

Actual behaviour

Some of the unsigned transactions are not executing successfully.

Steps to reproduce the behaviour

Option 1:

Execute eth_call, calling tokenURI function on the contract Top ERC721 transfers on Polygonscan. Worth noting, that I have tried multiple popular rpc endpoints and directly various running prod nodes.

curl --location --request POST 'https://rpc.ankr.com/polygon_zkevm' --header 'Content-Type: application/json' --data-raw '[{ "jsonrpc":"2.0", "method":"eth_call", "params":[{ "from": "0x0000000000000000000000000000000000000000", "to": "0xd8e1e7009802c914b0d39b31fc1759a865b727b1", "data": "0xc87b56dd0000000000000000000000000000000000000000000000000000000000000001" }, "latest"], "id":1 }]'

Then receive error:

[{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"failed to execute the unsigned transaction"}}]

Option 2:

Go to zkEVM PolygonScan: https://zkevm.polygonscan.com/address/0xd8e1e7009802c914b0d39b31fc1759a865b727b1#readContract

Go to tokenURI(tokenId uint256) func and enter any tokenID, for example 1.

Receive string Error: Returned error: failed to execute the unsigned transaction

Option 3:

Run prod node locally following this guide.

Send response from the Option 1, but to the local node (http://localhost:8545).

Receive the same error from Option 1.

Attempts and Debugging Details

Attempt 1

Checking if forking the blockchain locally will produce the same error. It was done to eliminate contract code error.

ganache-cli --fork https://rpc.ankr.com/polygon_zkevm

And then executing the same request, finally got me the wanted result, but not in the wanted way.

[{"id":1,"jsonrpc":"2.0","result":"0x00..00"}]

The resulting returned string is 27642 characters long, not including it here.

So the problem is with zkEVM infrastructure itself.

Attempt 2

Run prod node locally following this guide.

When executing

curl --location --request POST 'http://localhost:8545' --header 'Content-Type: application/json' --data-raw '[{ "jsonrpc":"2.0", "method":"eth_call", "params":[{ "from": "0x0000000000000000000000000000000000000000", "to": "0xd8e1e7009802c914b0d39b31fc1759a865b727b1", "data": "0xc87b56dd0000000000000000000000000000000000000000000000000000000000000001" }, "latest"], "id":1 }]'

I have received [{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"failed to execute the unsigned transaction"}}] error. NOTING, it's the same errors got from mainnet blockchain.

In the zkevm-node container, I have such error message for this request:

2023-04-14T23:14:34.420Z	ERROR	jsonrpc/server.go:401	failed to execute the unsigned transaction:not enough keccak counters to continue the execution %!s(MISSING)	{"pid": 1, "version": ""}
github.com/0xPolygonHermez/zkevm-node/jsonrpc.rpcErrorResponseWithData
   /src/jsonrpc/server.go:401
github.com/0xPolygonHermez/zkevm-node/jsonrpc.rpcErrorResponse
   /src/jsonrpc/server.go:396
github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*EthEndpoints).Call.func1
   /src/jsonrpc/endpoints_eth.go:109
github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*dbTxManager).NewDbTxScope
   /src/jsonrpc/dbtxmanager.go:21
github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*EthEndpoints).Call
   /src/jsonrpc/endpoints_eth.go:57
reflect.Value.call
   /usr/local/go/src/reflect/value.go:556
reflect.Value.Call
   /usr/local/go/src/reflect/value.go:339
github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*Handler).Handle
   /src/jsonrpc/handler.go:142
github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*Server).handleBatchRequest
   /src/jsonrpc/server.go:301
github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*Server).handle
   /src/jsonrpc/server.go:251
net/http.HandlerFunc.ServeHTTP
   /usr/local/go/src/net/http/server.go:2084
github.com/didip/tollbooth/v6.LimitHandler.func1
   /go/pkg/mod/github.com/didip/tollbooth/v6@v6.1.2/tollbooth.go:311
net/http.HandlerFunc.ServeHTTP
   /usr/local/go/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
   /usr/local/go/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
   /usr/local/go/src/net/http/server.go:2916
net/http.(*conn).serve
   /usr/local/go/src/net/http/server.go:1966
2023-04-14T23:14:34.421Z	INFO	jsonrpc/handler.go:144	failed call: [-32000]failed to execute the unsigned transaction. Params: [{
       "from": "0x0000000000000000000000000000000000000000",
       "to": "0xd8e1e7009802c914b0d39b31fc1759a865b727b1",
       "data": "0xc87b56dd0000000000000000000000000000000000000000000000000000000000000001"
   }, "latest"]	{"pid": 1, "version": "", "method": "eth_call", "requestId": 1}
]

As we see, we get a bit bigger error message failed to execute the unsigned transaction:not enough keccak counters to continue the execution. To investigate which service returns this error, I have put down zkevm-node docker container and have run zkevm-node locally. It lead me to the executor service on line https://github.com/0xPolygonHermez/zkevm-node/blob/v0.0.3-RC36/state/state.go#L1313

It's getting error ROM_ERROR_OUT_OF_COUNTERS_STEP, which returns keccak error, which is declared here

Looking for the code of this error in the logs. Found it here.

Next step is looking for logs of zkevm-prover, where executor service is running. The error code for this error is "OOCS" (got from utils.zkasm). Getting next error logs message in zkevm-prover container:

FullTracer::onError() error=OOCS zkPC=4663 rom= jmpAddr=4694 JMP=1 useJmpAddr=1 cmdBefore[0]={ op=functionCall funcName=eventLog params[0]={ op=getVar varName=onError } params[1]={ op=getVar varName=OOCS }} fileName=utils.zkasm line=875 lineStr=    $${eventLog(onError, OOCS)} :JMP(handleBatchError)
ExecutorServiceImpl::ProcessBatch() returns error=1 new_state_root=0x23c3523866b639b6d8feab83d6f3b9621cf9f965d43b87e41ed741e592bb82a0 new_acc_input_hash=0x5c0bc9fa55c9e1b7b579283c48b17a28e433d549d37f856540195ca7aa219809 new_local_exit_root=0x3092fd78d1e470bf2a0dfe0f1700e19a22469819de6df719d887ccfa23881a2b steps=7968531 gasUsed=2135579 counters.keccakF=12 counters.poseidonG=4701 counters.paddingPG=2892 counters.memAlign=28750 counters.arith=21079 counters.binary=412396 nTxs=1 tx[0].hash=0xc033d1a8983dd52f79d1d1a918d9288399f0a9426dc6254e1be447fcdedf176e gasUsed=2135579 gasLeft=27864421 gasUsed+gasLeft=30000000 gasRefunded=0 error=OOCS

What was tried, is increasing

"maxExecutorThreads":` 20 -> 900,
"dbNumberOfPoolConnections": 28 -> 950

as offered in issue #1930

It didn't help much, the problem persisted.

So I decided to bring up zkevm-prover locally, but unfortunately I didn't success as building project was using up all available RAM and swap on my laptop. I couldn't figure out how to do it correctly.

Another thing I did was looking for steps amount of zkevm-prover to increase it, but it is locked in the constants file, so it is not possible to config it for docker image.

Attempt 3

The steps executed in Attempt 2 was also run again latest available version v0.0.5-RC4, the behavior was the same.

Attempt 4

Checking if such behavior doesn't repeat for all such contracts.

Running the exact same request over another contract results in the expected behavior.

curl --location --request POST 'https://rpc.ankr.com/polygon_zkevm' --header 'Content-Type: application/json' --data-raw '[{
    "jsonrpc":"2.0",
    "method":"eth_call",
    "params":[{
        "from": "0x0000000000000000000000000000000000000000",
        "to": "0x43b6fe6c1aeaad671a7afd9787797d94988bf6da",
        "data": "0xc87b56dd0000000000000000000000000000000000000000000000000000000000000001"
    }, "latest"],
    "id":1
}]'

Returns

[{"jsonrpc":"2.0","id":1,"result":"0x0000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000005f68747470733a2f2f6261667962656967726664356f32626b616e367a7679647a356532643566613264756871746336706575657a6667343437366765716a32626770792e697066732e6e667473746f726167652e6c696e6b2f312e6a736f6e00"}]

Possible solutions

Taking into account the research above, it seems that max counter amount or some other counter amounts needs to be modified to comply with some heavy (?) transaction requirements.

Reason for Duplicating Issue

This issue duplicates #1930

It was done to provide a more in-depth investigation of the error and to gather all the relevant information in one place, as the request in the referenced issue is not reproducible anymore and its offered solution (workaround) is not working. This issue have to be fixed for proper utilization of the blockchain.

The text was updated successfully, but these errors were encountered:

ToniRamirezM · 2023-04-20T07:32:08Z

Hi @RedCuckoo,

Thanks for such a detailed analysis. As you say the counter limits are the source of the problem here. We have done a modification to the way we use our executor not to take into account these limits for unsigned transactions (limits will be applied anyway, but they will be much bigger).

GloWE3 · 2023-04-20T10:07:43Z

I'm having some connections issues while doing testing

RedCuckoo · 2023-07-21T11:36:28Z

Hello, @ToniRamirezM
I am sorry to inform about it but the issue has not been resolved with this patch.

Version Request

curl https://rpc.ankr.com/polygon_zkevm \
  -X POST \                                                                                                                                                                                             
  -H "Content-Type: application/json" \
  --data '{"method":"web3_clientVersion","params":[],"id":1,"jsonrpc":"2.0"}'

Version Reply

{"jsonrpc":"2.0","id":1,"result":"v0.1.2"}

Not working request

curl --location --request POST 'https://rpc.ankr.com/polygon_zkevm' --header 'Content-Type: application/json' --data-raw '[{ "jsonrpc":"2.0", "method":"eth_call", "params":[{ "from": "0x0000000000000000000000000000000000000000", "to": "0xd8e1e7009802c914b0d39b31fc1759a865b727b1", "data": "0xc87b56dd0000000000000000000000000000000000000000000000000000000000000001" }, "latest"], "id":1 }]'

Reply

[{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"failed to execute the unsigned transaction"}}]

The versions of our nodes have been updated to latest and the issue is reproducible still with other endpoints.

There has already been created duplicate issue for the same problem by other users #2238

istankovic · 2023-07-31T21:55:08Z

Can confirm, we from the Beamer bridge team are seeing this as well. Can something be done to prioritise this?

ToniRamirezM self-assigned this Apr 17, 2023

ToniRamirezM mentioned this issue Apr 17, 2023

Do not use zkcounters on eth_Call #2021

Merged

ToniRamirezM added this to the Release 0.1.0 milestone May 2, 2023

ToniRamirezM closed this as completed in #2021 May 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unsigned transaction execution failure when interacting with blockchain node #2019

Unsigned transaction execution failure when interacting with blockchain node #2019

RedCuckoo commented Apr 15, 2023 •

edited

Loading

ToniRamirezM commented Apr 20, 2023

GloWE3 commented Apr 20, 2023

RedCuckoo commented Jul 21, 2023

istankovic commented Jul 31, 2023

Unsigned transaction execution failure when interacting with blockchain node #2019

Unsigned transaction execution failure when interacting with blockchain node #2019

Comments

RedCuckoo commented Apr 15, 2023 • edited Loading

System information

Expected behaviour

Actual behaviour

Steps to reproduce the behaviour

Option 1:

Option 2:

Option 3:

Attempts and Debugging Details

Attempt 1

Attempt 2

Attempt 3

Attempt 4

Possible solutions

Reason for Duplicating Issue

ToniRamirezM commented Apr 20, 2023

GloWE3 commented Apr 20, 2023

RedCuckoo commented Jul 21, 2023

istankovic commented Jul 31, 2023

RedCuckoo commented Apr 15, 2023 •

edited

Loading