Further unblocking of recovery

[DaveCTurner]

Follow-on from #95115 using the new async runUnderPrimaryPermit to remove almost all the remaining blocking in RecoverySourceHandler.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

I hope that this is easy enough to review as a sequence of individual commits (ignoring whitespace changes). If not I can make a sequence of PRs.

Note that after these changes the only blocking during recovery happens at relocation handoff when waiting for all ongoing operations to finish. I hope to fix that soon too, but it's a more involved change because the blocking is deep within IndexShardOperationPermits. Once that's done, we should be able to write deterministic tests that execute all kinds of recovery, including primary relocations.

[DaveCTurner]

Marking this as WIP because this simple approach doesn't quite work, see #95309.

[DaveCTurner]

The retention lease bits are a little tricky; we've fixed the create-lease step in #95309 and #95324 will fix the delete-lease step. Once that's done I'll update this PR because the remaining steps are much simpler.

[DaveCTurner]

Ok this is good to review now

[henningandersen]

LGTM.

Thanks for splitting it up into individual commits, made it an easy review.