ChainIndexer panic

[rvagg]

Reported in here. Some combination of turning it on and off again I suspect. This looks to me like it doesn't have a reference to the reconciliation function and it's panicking on that. Looking at the code it seems that this function is provided on init by the DI system. I'm not sure how this would error but it is erroring so we should figure it out and make it more robust.

2025-06-03T22:44:59.030Z        INFO    modules modules/eth.go:123      Start prefilling GetTipsetByHeight cache
2025-06-03T22:44:59.030Z        INFO    chainindex      index/reconcile.go:66   starting chain reconciliation from head height 2722140; reconciliation epoch is 2719937
2025-06-03T22:44:59.031Z        INFO    chainindex      index/reconcile.go:91   found matching tipset at height 2722129, setting reconciliation epoch to 2722130
2025-06-03T22:44:59.031Z        INFO    chainindex      index/reconcile.go:112  Marking tipsets as reverted from height 2722130
2025-06-03T22:44:59.031Z        INFO    chainindex      index/reconcile.go:127  marked 0 tipsets as reverted from height 2722130
2025-06-03T22:44:59.031Z        INFO    chainindex      index/reconcile.go:136  indexing missing tipsets backwards from head height 2722140 to reconciliation epoch 2722130
2025-06-03T22:44:59.031Z        INFO    chainindex      index/reconcile.go:199  backfilling chain index backwards starting from head height 2722140
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1bd59e1]

goroutine 630 [running]:
github.com/filecoin-project/lotus/chain/index.withTx.func1()
        /home/ubuntu/lotus/chain/index/helpers.go:117 +0x3f
panic({0x474be40?, 0x91b8170?})
        /snap/go/10888/src/runtime/panic.go:792 +0x132
github.com/filecoin-project/lotus/chain/index.loadExecutedMessages.func1()
        /home/ubuntu/lotus/chain/index/events.go:170 +0x121
github.com/filecoin-project/lotus/chain/index.loadExecutedMessages({0x6d67f88, 0xc011a52230}, {0x6d7b158, 0xc000527320}, 0xc011efdbb0, 0xc011bc9b00, 0xc008d81100)
        /home/ubuntu/lotus/chain/index/events.go:222 +0x858
github.com/filecoin-project/lotus/chain/index.(*SqliteIndexer).SetRecomputeTipSetStateFunc.(*SqliteIndexer).buildExecutedMessagesLoader.func1({0x6d67f88?, 0xc011a52230?}, {0x6d7b158?, 0xc000527320?}, 0xc0174261b0?, 0x1?)
        /home/ubuntu/lotus/chain/index/indexer.go:152 +0x34
github.com/filecoin-project/lotus/chain/index.(*SqliteIndexer).indexEvents(0xc01191ea00, {0x6d67f88, 0xc011a52230}, 0xc000472c00, 0xc011bc9b00, 0xc008d81100)
        /home/ubuntu/lotus/chain/index/events.go:85 +0x3b1
github.com/filecoin-project/lotus/chain/index.(*SqliteIndexer).indexTipsetWithParentEvents(0xc01191ea00, {0x6d67f88, 0xc011a52230}, 0xc000472c00, 0xc011bc9b00, 0xc008d81100)
        /home/ubuntu/lotus/chain/index/indexer.go:325 +0xf4
github.com/filecoin-project/lotus/chain/index.(*SqliteIndexer).applyMissingTipset(0xc01191ea00, {0x6d67f88, 0xc011a52230}, 0xc000472c00, 0xc008d81100)
        /home/ubuntu/lotus/chain/index/reconcile.go:271 +0x10c
github.com/filecoin-project/lotus/chain/index.(*SqliteIndexer).backfillIndex(0xc01191ea00, {0x6d67f88, 0xc011a52230}, 0xc000472c00, 0xc008d80400, 0x298952)
        /home/ubuntu/lotus/chain/index/reconcile.go:211 +0x1b9
github.com/filecoin-project/lotus/chain/index.(*SqliteIndexer).ReconcileWithChain.func1(0xc000472c00)
        /home/ubuntu/lotus/chain/index/reconcile.go:139 +0xbb4
github.com/filecoin-project/lotus/chain/index.withTx({0x6d67f88, 0xc011a52230}, 0xc001376dd0, 0xc0117a1b08)
        /home/ubuntu/lotus/chain/index/helpers.go:121 +0x130
github.com/filecoin-project/lotus/chain/index.(*SqliteIndexer).ReconcileWithChain(0xc01191ea00, {0x6d67f88, 0xc011a52230}, 0xc008d80400)
        /home/ubuntu/lotus/chain/index/reconcile.go:42 +0x107
github.com/filecoin-project/lotus/node.ConfigFullNode.InitChainIndexer.func122.1({0x4601de0?, 0x6d4d0a0?})
        /home/ubuntu/lotus/node/modules/chainindex.go:99 +0x27a
go.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook(0xc0000002a0, {0x6d67f50, 0xc0004cb5f0}, {0xc011971340, 0x0, {0x0, 0x0}, {0x0, 0x0}, {{0xc0119ef0e0, ...}, ...}})
        /home/ubuntu/go/pkg/mod/go.uber.org/fx@v1.23.0/internal/lifecycle/lifecycle.go:256 +0x1f2
go.uber.org/fx/internal/lifecycle.(*Lifecycle).Start(0xc0000002a0, {0x6d67f50, 0xc0004cb5f0})
        /home/ubuntu/go/pkg/mod/go.uber.org/fx@v1.23.0/internal/lifecycle/lifecycle.go:216 +0x468
go.uber.org/fx.(*App).start-fm.(*App).start.func1({0x6d67f50?, 0xc0004cb5f0?})
        /home/ubuntu/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:704 +0x31
go.uber.org/fx.(*App).withRollback(0xc0001955c0, {0x6d67f50, 0xc0004cb5f0}, 0x0?)
        /home/ubuntu/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:686 +0x32
go.uber.org/fx.(*App).start(...)
        /home/ubuntu/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:703
go.uber.org/fx.withTimeout.func1()
        /home/ubuntu/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:803 +0x6b
created by go.uber.org/fx.withTimeout in goroutine 1
        /home/ubuntu/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:791 +0xc5

[rvagg]

Another report of the same thing: #13183

[rvagg]

The repro steps in #13183 are likely helpful in getting this solved.