Open
Description
Checklist
- This is not a security-related bug/issue. If it is, please follow please follow the security policy.
- This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
- This is not a new feature request. If it is, please file a feature request instead.
- This is not an enhancement request. If it is, please file a improvement suggestion instead.
- I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
- I am running the
Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these. - I did not make any code changes to lotus.
Lotus component
- lotus daemon - chain sync
- lotus miner - mining and block production
- lotus miner/worker - sealing
- lotus miner - proving(WindowPoSt)
- lotus miner/market - storage deal
- lotus miner/market - retrieval deal
- lotus miner/market - data transfer
- lotus client
- lotus JSON-RPC API
- lotus message management (mpool)
- Other
Lotus Version
boostd version 1.4.0+git.810afec
Based on the boostd version, the SP is running either Lotus v1.17.0 or v1.17.1
Describe the Bug
This is was an error reported initially by @stuberman in the Boost-repo, but the crash is coming from the reader in the lotus storage. Issue-report here.
When running boostd dagstore initialize-all --concurrency=3 or even boostd dagstore initialize-all --concurrency=1 then boostd crashes after a few minutes. See log below:
Logging Information
2022-09-15T18:28:09.163Z INFO boost-storage-deal logs/log.go:40 current sealing state {"id": "dd917e35-d1ac-4f30-9b96-e55d00bf4799", "state": "Packing"}
panic: runtime error: slice bounds out of range [:16777216] with capacity 8388608
goroutine 599691 [running]:
github.com/filecoin-project/lotus/storage/sealer/fr32.(*unpadReader).Read(0x0, {0xc02eba5bfc, 0x140b154, 0x140b154})
/home/stuart/go/pkg/mod/github.com/filecoin-project/lotus@v1.17.0/storage/sealer/fr32/readers.go:62 +0x378
bufio.(*Reader).Read(0xc026924060, {0xc02eba5bfc, 0x140b154, 0x1})
/usr/local/go/src/bufio/bufio.go:213 +0x106
bufio.(*Reader).Read(0xc0269240c0, {0xc02eba5bfc, 0x140b154, 0xa0a640})
/usr/local/go/src/bufio/bufio.go:213 +0x106
io.ReadAtLeast({0x42ef300, 0xc0269240c0}, {0xc02eb86000, 0x142ad50, 0x142ad50}, 0x142ad50)
/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
/usr/local/go/src/io/io.go:347
github.com/filecoin-project/lotus/storage/sealer.(*pieceReader).readAtUnlocked(0xc021ceafc0, {0xc02eb86000, 0x64c349, 0x142ad50}, 0x4)
/home/stuart/go/pkg/mod/github.com/filecoin-project/lotus@v1.17.0/storage/sealer/piece_reader.go:187 +0xb17
github.com/filecoin-project/lotus/storage/sealer.(*pieceReader).Read(0xc021ceafc0, {0xc02eb86000, 0x142ad50, 0x142ad50})
/home/stuart/go/pkg/mod/github.com/filecoin-project/lotus@v1.17.0/storage/sealer/piece_reader.go:100 +0x155
io.ReadAtLeast({0x7f4dae370820, 0xc021ceafc0}, {0xc02eb86000, 0x142ad50, 0x142ad50}, 0x142ad50)
/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
/usr/local/go/src/io/io.go:347
github.com/ipld/go-car/v2/internal/carv1/util.LdRead({0x7f4dae370820, 0xc021ceafc0}, 0x0, 0x2000000)
/home/stuart/go/pkg/mod/github.com/ipld/go-car/v2@v2.4.2-0.20220707083113-89de8134e58e/internal/carv1/util/util.go:85 +0x19e
github.com/ipld/go-car/v2/internal/carv1.ReadHeader({0x7f4dae370820, 0xc021ceafc0}, 0x401)
/home/stuart/go/pkg/mod/github.com/ipld/go-car/v2@v2.4.2-0.20220707083113-89de8134e58e/internal/carv1/car.go:63 +0x32
github.com/ipld/go-car/v2.ReadVersion({0x7f4dae370820, 0xc021ceafc0}, {0xc00f2d7c98, 0x9749a5, 0xc011b88300})
/home/stuart/go/pkg/mod/github.com/ipld/go-car/v2@v2.4.2-0.20220707083113-89de8134e58e/reader.go:364 +0x90
github.com/ipld/go-car/v2.ReadOrGenerateIndex({0x7f4dae3707f8, 0xc021ceafc0}, {0xc00f2d7c98, 0x2, 0x2})
/home/stuart/go/pkg/mod/github.com/ipld/go-car/v2@v2.4.2-0.20220707083113-89de8134e58e/index_gen.go:191 +0x7a
github.com/filecoin-project/dagstore.(*DAGStore).initializeShard.func1({0xc00f2d7d80, 0xc00f2d7d50})
/home/stuart/go/pkg/mod/github.com/filecoin-project/dagstore@v0.5.3/dagstore_async.go:123 +0xcb
github.com/filecoin-project/dagstore/throttle.(*throttler).Do(0xc000b6c280, {0x433abd8, 0xc0000520b8}, 0xc025a847b0)
/home/stuart/go/pkg/mod/github.com/filecoin-project/dagstore@v0.5.3/throttle/throttler.go:38 +0x118
github.com/filecoin-project/dagstore.(*DAGStore).initializeShard(0xc000a99b80, {0x433abd8, 0xc0000520b8}, 0xc016d8a3f0, {0x43489f0, 0xc0129bdef0})
/home/stuart/go/pkg/mod/github.com/filecoin-project/dagstore@v0.5.3/dagstore_async.go:121 +0x422
created by github.com/filecoin-project/dagstore.(*DAGStore).control
/home/stuart/go/pkg/mod/github.com/filecoin-project/dagstore@v0.5.3/dagstore_control.go:101 +0x705
^C
[1]+ Exit 2 nohup boostd run --pprof > /market/logs/boost.log 2>&1
Repo Steps
- Run
boostd dagstore initialize-all --concurrency=3on a Boost-node - See error coming from the reader in the lotus storage
...
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Boost <> Lotus collab