[bug/portal_loop] `runtime error: index out of range [2] with length 0` while deploying certain realm on poral-loop #2283

r3v4s · 2024-06-05T03:20:15Z

Description

We're trying to deploy our realms against portal-loop, but pool realm is failing while gnokey maketx addpkg with below error.

$ gnokey maketx addpkg -pkgdir ./pool -pkgpath gno.land/r/demo/p2 -broadcast=true -gas-fee 1ugnot -gas-wanted 100000000 -insecure-password-stdin=true test1 -chainid portal-loop -remote https://rpc.gno.land:443
Enter password.

--= Error =--
Data: internal error
Msg Traces:
    0  /Users/n3wbie/gno_orig/tm2/pkg/crypto/keys/client/maketx.go:213 - deliver transaction failed: log:recovered: runtime error: index out of range [2] with length 0
stack:
goroutine 103295 [running]:
runtime/debug.Stack()
        /opt/hostedtoolcache/go/1.22.3/x64/src/runtime/debug/stack.go:24 +0x5e
github.com/gnolang/gno/tm2/pkg/sdk.(*BaseApp).runTx.func1()
        /home/runner/work/gno/gno/tm2/pkg/sdk/baseapp.go:749 +0x205
panic({0xfb75c0?, 0xc01816b428?})
        /opt/hostedtoolcache/go/1.22.3/x64/src/runtime/panic.go:770 +0x132
go/types.(*Checker).handleBailout(0xc00c057800, 0xc0076e5d40)
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/check.go:367 +0x88
panic({0xfb75c0?, 0xc01816b428?})
        /opt/hostedtoolcache/go/1.22.3/x64/src/runtime/panic.go:770 +0x132
github.com/gnolang/gno/gnovm/pkg/gnolang.(*StructValue).GetPointerToInt(...)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/values.go:464
github.com/gnolang/gno/gnovm/pkg/gnolang.(*StructValue).GetPointerTo(0x12aaa60?, {0x12aaa60?, 0xc00aa30280?}, {0x28?, 0xac?, 0x128?, {0xc016960340?, 0x2?}})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/values.go:460 +0xc5
github.com/gnolang/gno/gnovm/pkg/gnolang.(*TypedValue).GetPointerTo(0xc006286028, 0x0, {0x12aaa60, 0xc00aa30280}, {0x0?, 0x0?, 0x0?, {0xc016960340, 0x7}})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/values.go:1716 +0x96d
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).PopAsPointer(0xc00c7fb780?, {0x129b2f0?, 0xc010f48ba0?})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:1953 +0x270
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).doOpAssign(0xc00ea366c8)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/op_assign.go:35 +0x254
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).Run(0xc00ea366c8)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:1380 +0xd73
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).runDeclaration(0xc00ea366c8, {0x129c998?, 0xc019f7d570})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:841 +0x265
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).runFiles.func1(0xc01eb22608, {0x129c998, 0xc019f7d570})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:609 +0x334
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).runFiles(0xc00ea366c8, {0xc00697a988, 0x1, 0x1})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:623 +0x78e
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).RunFiles(...)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:496
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).runMemPackage(0xc00ea366c8, 0xc0147b49c0, 0x1, 0x0)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:287 +0x22d
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).RunMemPackage(...)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:255
github.com/gnolang/gno/gno.land/pkg/sdk/vm.(*VMKeeper).initBuiltinPackagesAndTypes.func1({0xc015226c79, 0xf}, {0x12aaa60, 0xc00aa30280})
        /home/runner/work/gno/gno/gno.land/pkg/sdk/vm/builtins.go:40 +0x245
github.com/gnolang/gno/gnovm/pkg/gnolang.(*defaultStore).GetPackage(0xc00aa30280, {0xc015226c79, 0xf}, 0x18?)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/store.go:170 +0x125
github.com/gnolang/gno/gnovm/pkg/gnolang.(*defaultStore).getMemPackage(0xc00aa30280, {0xc015226c79, 0xf}, 0x0)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/store.go:558 +0xbd
github.com/gnolang/gno/gnovm/pkg/gnolang.(*defaultStore).GetMemPackage(0xedfd00?, {0xc015226c79?, 0xc015226c79?})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/store.go:545 +0x1a
github.com/gnolang/gno/gnovm/pkg/gnolang.(*gnoImporter).ImportFrom(0xc01330b5c0, {0xc015226c79, 0xf}, {0xc0076e52a8?, 0x22acd8?}, 0xf?)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/go2gno.go:540 +0x64
go/types.(*Checker).importPackage(0xc00c057800, {0x1287700, 0xc017779240}, {0xc015226c79, 0xf}, {0x127abc0, 0x1})
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/resolver.go:158 +0x53a
go/types.(*Checker).collectObjects.func1({0x1287880?, 0xc01423b6e0})
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/resolver.go:267 +0x125
go/types.(*Checker).walkDecl(0xc00c057800, {0x128e270?, 0xc01d7b9780}, 0xc0076e5c38)
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/decl.go:412 +0x2e7
go/types.(*Checker).walkDecls(...)
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/decl.go:399
go/types.(*Checker).collectObjects(0xc00c057800)
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/resolver.go:254 +0xe2f
go/types.(*Checker).checkFiles(0xc00c057800, {0xc00aa30320, 0x12, 0x13})
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/check.go:405 +0x17c
go/types.(*Checker).Files(...)
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/check.go:372
go/types.(*Config).Check(0xc0091b37a0, {0xc0152260a8?, 0x12?}, 0xc01d7b8200, {0xc00aa30320, 0x12, 0x13}, 0x0)
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/api.go:458 +0x79
github.com/gnolang/gno/gnovm/pkg/gnolang.(*gnoImporter).parseCheckMemPackage(0xc01330b5c0, 0xc01d7b8100)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/go2gno.go:574 +0x366
github.com/gnolang/gno/gnovm/pkg/gnolang.TypeCheckMemPackage(0xc01d7b8100, {0x7fe1a29010f0, 0xc00aa30280})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/go2gno.go:505 +0x14c
github.com/gnolang/gno/gno.land/pkg/sdk/vm.(*VMKeeper).AddPackage(_, {{0x12900f0, 0xc00d19b5f0}, 0x1, {0x128e890, 0xc0161eb150}, {0x1290470, 0xc00d5d09a0}, {0xc006887720, 0xb}, ...}, ...)
        /home/runner/work/gno/gno/gno.land/pkg/sdk/vm/keeper.go:166 +0x545
github.com/gnolang/gno/gno.land/pkg/sdk/vm.vmHandler.handleMsgAddPackage({_}, {{0x12900f0, 0xc00d19b5f0}, 0x1, {0x128e890, 0xc0161eb150}, {0x1290470, 0xc00d5d09a0}, {0xc006887720, 0xb}, ...}, ...)
        /home/runner/work/gno/gno/gno.land/pkg/sdk/vm/handler.go:44 +0xb8
github.com/gnolang/gno/gno.land/pkg/sdk/vm.vmHandler.Process({_}, {{0x12900f0, 0xc00d19b5f0}, 0x1, {0x128e890, 0xc0161eb150}, {0x1290470, 0xc00d5d09a0}, {0xc006887720, 0xb}, ...}, ...)
        /home/runner/work/gno/gno/gno.land/pkg/sdk/vm/handler.go:31 +0x358
github.com/gnolang/gno/tm2/pkg/sdk.(*BaseApp).runMsgs(_, {{0x12900f0, 0xc00d19b5f0}, 0x1, {0x128e890, 0xc0161eb150}, {0x1290470, 0xc00d5d09a0}, {0xc006887720, 0xb}, ...}, ...)
        /home/runner/work/gno/gno/tm2/pkg/sdk/baseapp.go:650 +0x2ed
github.com/gnolang/gno/tm2/pkg/sdk.(*BaseApp).runTx(0xc000582200, 0x1, {0xc004ba2000, 0x129b1, _}, {{0xc00c2ab950, 0x1, 0x1}, {0x5f5e100, {{0xc013722db1, ...}, ...}}, ...})
        /home/runner/work/gno/gno/tm2/pkg/sdk/baseapp.go:829 +0xae5
github.com/gnolang/gno/tm2/pkg/sdk.(*BaseApp).Simulate(...)
        /home/runner/work/gno/gno/tm2/pkg/sdk/helpers.go:17
github.com/gnolang/gno/tm2/pkg/sdk.handleQueryApp(0xc000582200, {0xc01330aca0?, 0x0?, _}, {{}, {0xc004ba2000, 0x129b1, 0x129b1}, {0xc013722c30, 0xd}, ...})
        /home/runner/work/gno/gno/tm2/pkg/sdk/baseapp.go:418 +0x25d
github.com/gnolang/gno/tm2/pkg/sdk.(*BaseApp).Query(0xc000582200, {{}, {0xc004ba2000, 0x129b1, 0x129b1}, {0xc013722c30, 0xd}, 0x0, 0x0})
        /home/runner/work/gno/gno/tm2/pkg/sdk/baseapp.go:391 +0x14f
github.com/gnolang/gno/tm2/pkg/bft/abci/client.(*localClient).QuerySync(0xc00abf9220?, {{}, {0xc004ba2000, 0x129b1, 0x129b1}, {0xc013722c30, 0xd}, 0x0, 0x0})
        /home/runner/work/gno/gno/tm2/pkg/bft/abci/client/local_client.go:180 +0x154
github.com/gnolang/gno/tm2/pkg/bft/appconn.(*query).QuerySync(0x7fe1a1d56528?, {{}, {0xc004ba2000, 0x129b1, 0x129b1}, {0xc013722c30, 0xd}, 0x0, 0x0})
        /home/runner/work/gno/gno/tm2/pkg/bft/appconn/app_conn.go:144 +0xdc
github.com/gnolang/gno/tm2/pkg/bft/rpc/core.ABCIQuery(0xc009d8a9c0?, {0xc013722c30, 0xd}, {0xc004ba2000, 0x129b1, 0x129b1}, 0x0?, 0x33?)
        /home/runner/work/gno/gno/tm2/pkg/bft/rpc/core/abci.go:59 +0x132
reflect.Value.call({0xf0d780?, 0x114d600?, 0x1030d60?}, {0x1033772, 0x4}, {0xc017e95b80, 0x5, 0x5?})
        /opt/hostedtoolcache/go/1.22.3/x64/src/reflect/value.go:596 +0xca6
reflect.Value.Call({0xf0d780?, 0x114d600?, 0x18d29?}, {0xc017e95b80?, 0x1030d60?, 0xe973a0?})
        /opt/hostedtoolcache/go/1.22.3/x64/src/reflect/value.go:380 +0xb9
github.com/gnolang/gno/tm2/pkg/bft/rpc/lib/server.RegisterRPCFuncs.makeJSONRPCHandler.func9({0x128e800, 0xc00606ca68}, 0xc00d8ec360)
        /home/runner/work/gno/gno/tm2/pkg/bft/rpc/lib/server/handlers.go:185 +0x745
github.com/gnolang/gno/tm2/pkg/bft/rpc/lib/server.RegisterRPCFuncs.handleInvalidJSONRPCPaths.func10({0x128e800?, 0xc00606ca68?}, 0x0?)
        /home/runner/work/gno/gno/tm2/pkg/bft/rpc/lib/server/handlers.go:209 +0x42
net/http.HandlerFunc.ServeHTTP(0xc0008f5340?, {0x128e800?, 0xc00606ca68?}, 0x411dbb?)
        /opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:2166 +0x29
net/http.(*ServeMux).ServeHTTP(0xc008d4b0a0?, {0x128e800, 0xc00606ca68}, 0xc00d8ec360)
        /opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:2683 +0x1ad
github.com/gnolang/gno/tm2/pkg/bft/node.(*Node).startRPC.(*Cors).Handler.func5({0x128e800, 0xc00606ca68}, 0xc00d8ec360)
        /home/runner/go/pkg/mod/github.com/rs/cors@v1.10.1/cors.go:281 +0x184
net/http.HandlerFunc.ServeHTTP(0x0?, {0x128e800?, 0xc00606ca68?}, 0xc00140ba10?)
        /opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:2166 +0x29
github.com/gnolang/gno/tm2/pkg/bft/rpc/lib/server.maxBytesHandler.ServeHTTP({{0x1285280?, 0xc01b602de0?}, 0xd?}, {0x128e800, 0xc00606ca68}, 0xc00d8ec360)
        /home/runner/work/gno/gno/tm2/pkg/bft/rpc/lib/server/http_server.go:216 +0xfb
github.com/gnolang/gno/tm2/pkg/bft/rpc/lib/server.StartHTTPServer.RecoverAndLogHandler.func1({0x128e740, 0xc01977f420}, 0xc00d8ec360)
        /home/runner/work/gno/gno/tm2/pkg/bft/rpc/lib/server/http_server.go:189 +0x1f8
net/http.HandlerFunc.ServeHTTP(0x412165?, {0x128e740?, 0xc01977f420?}, 0xc01977f401?)
        /opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:2166 +0x29
net/http.serverHandler.ServeHTTP({0x128b898?}, {0x128e740?, 0xc01977f420?}, 0x6?)
        /opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:3137 +0x8e
net/http.(*conn).serve(0xc014b2b440, {0x12900f0, 0xc021ace120})
        /opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:2039 +0x5e8
created by net/http.(*Server).Serve in goroutine 74
        /opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:3285 +0x4b4

--= /Error =--

Steps to reproduce on LOCAL [couldn't really reproduce on local. it doesn't panic, but succeeds]

unable to reproduce this issue on local chain, seems to be happening only in portal-loop.

# clone fresh version of gno
$ git clone https://github.com/gnolang/gno gno_orig
$ cd ~/gno_orig

# gnokey install
$ make install


# gnoland install and start
$ cd gno.land
$ make install
$ gnoland start -lazy=true

## cloned gno repo every single commit from 0d299c37d0027edf16de1823b0be824030a2823a ~ 93299802139012ba8e80a99ee406319430ef7277 to re-produce on local but succeeded every commit.


# clone fresh version of gnoswap
$ git clone https://github.com/gnoswap-labs/gnoswap
$ cd gnoswap
$ git switch tmp/portal-loop-deploy-fail

# local deploy test
$ make -f __local/test/test_data.mk init

Tried below ways on portal-loop, but none of them succeeded

Change pkg_path to deploy
Reduce pool contract size (by removing comments, some common functions and etc...)
Set gas-wanted to max
Change client's env ( darwin-arm64, linux-amd64 )

Expected behaviour

pool should be deployed

Actual behaviour

panic

FYI

There is realm that imports pool realm, such as position

Since deploying pool has been failed on portal-loop, deploying position needs to be fail with message like unknown import found gno.land/r/demo/pool. However deploying position also panic with same error(index ouf of range)

The text was updated successfully, but these errors were encountered:

thehowl · 2024-06-07T17:23:49Z

Okay, leaving an update on my current investigation.

Replication set-up

I'm running a gno.land node with this genesis.json. This is a trimmed-down version of the portal loop genesis.json. With this genesis.json, publishing succeeds.

To run using this genesis file, you'll need to change the validators key so that your local instance is a validator. You can get the fields you need from gnoland-data/secrets/priv_validator_key.json.

Then, I'm running the node with this command:

rm -rf gnoland-data/{db,wal} && gnoland secrets init -force ValidatorState && (go run ./cmd/gnoland start --lazy --skip-failing-genesis-txs | tee gnoland-node.txt)

This allows me to reset the node and start from this genesis. I get the logs redirected to gnoland-node.txt for inspection. Using the two commands at the beginning allows me not to have to update the validator address in genesis each time.

Findings

Running the replication step pointed above by @r3v4s:

make -f __local/test/test_data.mk init CHAINID=portal-loop

As I said in the previous section, publishing succeeds with the provided genesis.json. However, it won't if you first broadcast this transaction: feeds-add.json.

(This should work by simply doing gnokey broadcast of the file; otherwise, try to adjust the signatures)

After that, you get the panic pointed above. (This error happens also if the transaction is in the genesis).

The surprising, and most revealing, fact about this is that if you re-start the node (without clearing the database/state); then publishing works as normal.

Suspicions

Looking through some logs, my suspicions are:

That this has to do with encoding/base64, and what happens when handling a failing transaction.
That this is because of defaultStore.cacheNodes, which unlike the name would suggest is not a cache, but it is instead re-initialised on every node restart.
That the cause of this, is that having a failing transaction makes cacheNodes "dirty", leading to this panic, while there is nothing stored permanently on-chain which can cause the panic; thus why the problem doesn't show up when the node restart.

That's it for this week, I may continue this another time.

thehowl · 2024-06-07T17:26:09Z

Note: I'm trying to get a txtar working before doing an actual fix. If my suspicion is correct, then we'll probably need to discuss either a) removing the caches from the Store (and just using the tm2 store cache), which does support transaction rollback properly, or b) creating a cache system in the Store that can properly handle rollbacks/commits.

thehowl · 2024-06-10T14:03:11Z

#2319 I found a way to replicate the issue; don't know if I'll have time to fix it though

I'm thinking that attempting to fix it would mean trying to make Store properly transactional; so that'll take a while to get working

thehowl · 2024-06-28T15:26:34Z

I'm prioritizing a fix for this one in the next week, to be ready by Gophercon

r3v4s added help wanted Extra attention is needed 🐞 bug Something isn't working 📦 ⛰️ gno.land Issues or PRs gno.land package related labels Jun 5, 2024

r3v4s self-assigned this Jun 5, 2024

zivkovicmilos added the 📦 🤖 gnovm Issues or PRs gnovm related label Jun 5, 2024

adr-sk mentioned this issue Jun 5, 2024

Onbloc's Builder Journey gnolang/hackerspace#29

Open

thehowl mentioned this issue Jun 10, 2024

fix(gno.land): make gno store cache respect rollbacks #2319

Draft

Kouteki mentioned this issue Jun 12, 2024

Minutes: Core Staff Weekly Syncs [every Monday] gnolang/meetings#36

Open

michelleellen mentioned this issue Jun 12, 2024

Minutes: Public Dev Calls [every Wednesday] gnolang/meetings#37

Open

thehowl assigned thehowl and unassigned r3v4s Jun 28, 2024

thehowl mentioned this issue Jul 4, 2024

fix(gno.land): pre-load all standard libraries in vm.Initialize #2504

Merged

7 tasks

moul closed this as completed in #2504 Jul 6, 2024

moul closed this as completed in f28444a Jul 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bug/portal_loop] `runtime error: index out of range [2] with length 0` while deploying certain realm on poral-loop #2283

[bug/portal_loop] `runtime error: index out of range [2] with length 0` while deploying certain realm on poral-loop #2283

r3v4s commented Jun 5, 2024

thehowl commented Jun 7, 2024

thehowl commented Jun 7, 2024

thehowl commented Jun 10, 2024

thehowl commented Jun 28, 2024

[bug/portal_loop] runtime error: index out of range [2] with length 0 while deploying certain realm on poral-loop #2283

[bug/portal_loop] runtime error: index out of range [2] with length 0 while deploying certain realm on poral-loop #2283

Comments

r3v4s commented Jun 5, 2024

Description

Steps to reproduce on LOCAL [couldn't really reproduce on local. it doesn't panic, but succeeds]

Tried below ways on portal-loop, but none of them succeeded

Expected behaviour

Actual behaviour

FYI

thehowl commented Jun 7, 2024

Replication set-up

Findings

Suspicions

thehowl commented Jun 7, 2024

thehowl commented Jun 10, 2024

thehowl commented Jun 28, 2024

[bug/portal_loop] `runtime error: index out of range [2] with length 0` while deploying certain realm on poral-loop #2283

[bug/portal_loop] `runtime error: index out of range [2] with length 0` while deploying certain realm on poral-loop #2283