Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: Return soft err when sector alloc fails in acquire #11338

Merged
merged 1 commit into from
Oct 18, 2023

Conversation

magik6k
Copy link
Contributor

@magik6k magik6k commented Oct 17, 2023

Related Issues

#11087 is missing one case making it possible that some sectors will get removed when SP pipelines are under heavy load

Proposed Changes

Additional Info

Example logs before

3.	2023-10-17 03:04:43 +0200 CEST:	[event;sealing.SectorStartPacking]	{"User":{}}
4.	2023-10-17 03:04:43 +0200 CEST:	[event;sealing.SectorPacked]	{"User":{"FillerPieces":null}}
5.	2023-10-17 03:04:43 +0200 CEST:	[event;sealing.SectorTicket]	{"User":{"TicketValue":"ai7dVU55W2rBSUfblw+o8f81YfGyIWKjQu8eSV1KibA=","TicketEpoch":3305709}}
6.	2023-10-17 03:16:38 +0200 CEST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: acquiring sector paths: local acquire error: couldn't find a suitable path for a sector [name: okonomiyaki]: acquiring sector paths: local acquire error: couldn't find a suitable path for a sector
7.	2023-10-17 03:17:38 +0200 CEST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
8.	2023-10-17 04:30:32 +0200 CEST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: acquiring sector paths: local acquire error: couldn't find a suitable path for a sector [name: okonomiyaki]: acquiring sector paths: local acquire error: couldn't find a suitable path for a sector
9.	2023-10-17 04:31:32 +0200 CEST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
10.	2023-10-17 04:44:06 +0200 CEST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: acquiring sector paths: local acquire error: couldn't find a suitable path for a sector [name: okonomiyaki]: acquiring sector paths: local acquire error: couldn't find a suitable path for a sector
11.	2023-10-17 04:45:06 +0200 CEST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
12.	2023-10-17 05:05:54 +0200 CEST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: acquiring sector paths: local acquire error: couldn't find a suitable path for a sector [name: okonomiyaki]: acquiring sector paths: local acquire error: couldn't find a suitable path for a sector
13.	2023-10-17 05:05:54 +0200 CEST:	[event;sealing.SectorRemove]	{"User":{}}
14.	2023-10-17 05:05:55 +0200 CEST:	[event;sealing.SectorRemoved]	{"User":{}}

TODO:

  • Try to repro on a real setup, look for the new retrying soft error

Checklist

Before you mark the PR ready for review, please make sure that:

  • Commits have a clear commit message.
  • PR title is in the form of of <PR type>: <area>: <change being made>
    • example: fix: mempool: Introduce a cache for valid signatures
    • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
    • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
  • If the PR affects users (e.g., new feature, bug fix, system requirements change), update the CHANGELOG.md and add details to the UNRELEASED section.
  • New features have usage guidelines and / or documentation updates in
  • Tests exist for new functionality or change in behavior
  • CI is green

@magik6k
Copy link
Contributor Author

magik6k commented Oct 18, 2023

Definitely works on my setup, seeing logs like 2023-10-18T15:37:56.879+0200 ERROR sectors pipeline/states_sealing.go:235 retrying soft error {"err": "storage call error 102: couldn't find a suitable path for a sector", "code": 102}, and no removed sectors (and no error logs in sector info!)

@magik6k magik6k marked this pull request as ready for review October 18, 2023 14:26
@magik6k magik6k requested a review from a team as a code owner October 18, 2023 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants