keep retrying the proof until we run out of sectors to skip

[Stebalien]

If we have a bunch of corrupted but not missing sectors on disk, we may need to
retry many times before we get a proof to pass. Simply giving up doesn't help anyone.

[magik6k]

1 Comment

[Stebalien]

This will explicitly check the context. We should cancel in

lotus/storage/wdpost_changehandler.go

Lines 412 to 426 in 077bc83

	// Replace the aborted postWindow with a new one so that we can
	// submit again at any time without the state getting clobbered
	// when the abort completes
	abort := pw.abort
	if abort != nil {
	pw = &postWindow{
	di: pw.di,
	ts: advance,
	submitState: SubmitStateStart,
	}
	s.postWindows[pw.di.Open] = pw
	// Abort the current submit
	abort()
	}

.

[Stebalien]

I could also stop retrying once we get, e.g., 2/3rds of the way through the proof time but I'm not sure if that really makes sense. I guess sectors assigned to a single partition are somewhat correlated in time so their failure may be correlated? But I don't wan to:

1. Spend a lot of time trying to prove one partition.
2. Give up on that partition because we're running out of time.
3. Spend a little time trying to prove all other partitions in the deadline and fail because we have a lot of faulty sectors.

When we could have eventually submitted a valid proof for the first partition, if we had simply stuck with it.

[Stebalien]

Note: I agree this isn't the optimal solution, it's just strictly better than what we're doing now. Once we merge this, I plan on:

• performing a more thorough check before recovering.
• running the recovery check on a partition batch if we fail PoSt 2 times in a row.