fix(datadir): wait out schelk state-lock contention instead of failing the job

[qu0b]

What happened

benchmark-ci-6 destroyed five consecutive Besu jobs today, each in ~12 seconds, all in the Build state-actor data directory step:

Build failed error=ensuring schelk mount: `schelk mount` failed: exit status 1
  0: Another schelk process is already running (flock on /var/lib/schelk/schelk.lock): EAGAIN: Try again

Every Besu job that landed on a different runner succeeded. ci-6 has no successful job at all in its history today.

Root cause

Runners serialise schelk work through a single flock at /var/lib/schelk/schelk.lock. While another schelk process holds it, schelk mount exits non-zero with EAGAIN — and EnsureSchelkMounted returned that straight to the caller:

output, runErr := RunSchelk(ctx, log, "mount", "-y")
if runErr != nil {
    return fmt.Errorf("`%s mount` failed: %w (output: %s)", bin, runErr, ...)
}

So a transient, self-clearing condition discarded a multi-hour benchmark job before it ran a single block.

It also compounds: the job fails in ~12s, the runner is immediately free, so it wins the next queued job of that client and burns that one too. One host with a held lock acts as a black hole, and the failure rate scales with how much work is queued.

Note the lock holder is necessarily a live process — flock is released on process death, so this is never a stale lock file. Waiting is the correct response.

Change

Retry schelk mount while the failure is lock contention, up to SchelkLockWait (10m), polling every 15s. Any other failure still returns on the first attempt, so genuine mount problems — including the is_mounted inconsistency that needs full-recover — stay fail-fast.

This covers both call sites, since state_actor.go and eest_payloads.go both go through EnsureSchelkMounted.

Tests

• TestSchelkLockHeld — classification, including that "Volume is already mounted" is not treated as lock contention
• TestMountWaitingForLock_RetriesUntilLockClears — fake schelk fails with the flock error twice then succeeds; asserts exactly 3 attempts
• TestMountWaitingForLock_NonLockErrorFailsImmediately — asserts exactly 1 attempt, no retry

go test ./pkg/datadir/ and go vet pass.

Not a substitute for unblocking ci-6

This stops one stuck host from shredding the queue, but something is still holding that lock on benchmark-ci-6. That needs lsof /var/lib/schelk/schelk.lock on the host to identify the holder, and the runner pulled from the pool until it is cleared.

[qu0b]

Both points were valid — fixed in 8d22bbe.

🟡 nil-logger panic — confirmed and fixed. Verified the call chain rather than taking it on trust: pkg/config/config.go:3153 does pass nil, RunSchelk documents "log may be nil" (schelk.go:57), and the file already guards at :92 and :108. My warn dereferenced it unconditionally, so it would have panicked on precisely the contention path this PR exists to survive — the original code never logged there, which is why nil was safe before. Guarded in the existing if log != nil style.

Added TestMountWaitingForLock_NilLoggerDoesNotPanic, and checked it actually catches the bug by removing the guard:

--- FAIL: TestMountWaitingForLock_NilLoggerDoesNotPanic
    Error: func (assert.PanicTestFunc) should not panic
    Panic value: runtime error: invalid memory address or nil pointer dereference

Suggestion on schelk.lock matching — agreed, narrowed. You're right that it inverts the intent: I added it as a hedge against wording changes, but the failure it creates is worse than the one it prevents. A missing or unwritable lock path names the file too, and would then be retried for the full 10 minutes and hang the host instead of surfacing. If the wording ever changes, the narrow match just falls back to today's fail-fast behaviour — strictly better than masking a real error.

Now matches the explicit contention wording or the EAGAIN marker, with a test asserting "failed to open /var/lib/schelk/schelk.lock: permission denied" is not classified as contention.

go vet and the full pkg/datadir suite pass (11 tests).