Refactor RecoveryTarget state management

[bleskes]

The PR rewrites the state controls in the RecoveryTarget family classes to make it easier to guarantee that:

• recovery resources are only cleared once there are no ongoing requests
• recovery is automatically canceled when the target shard is closed/removed
• canceled recoveries do not leave temp files behind when canceled.

Highlights of the change:

1. All temporary files are cleared upon failure/cancel (see Recovery files left behind when replica building fails #7315 )
2. All newly created files are always temporary
3. Doesn't list local files on the cluster state update thread (which throw unwanted exception)
4. Recoveries are canceled by a listener to IndicesLifecycle.beforeIndexShardClosed, so we don't need to explicitly call it.
5. Simplifies RecoveryListener to only notify when a recovery is done or failed. Removed subtleties like ignore and retry (they are dealt with internally)

Relates to #7893

[s1monw]

can we get a javadoc string what this class does?

[s1monw]

I think you can just do:

IOUtils.closeWhileHandlingException(openIndexOutputs.values());
openIndexOutputs.clear();

[bleskes]

That's strictly speaking thread unsafe. I think we can do this here in practice, but part of the refactoring was not to have to worry about these kind of things. I think we should keep it as is?

[s1monw]

I left a bunch of comments - I really like that change btw :) 👍

[bleskes]

@s1monw I pushed some commits to address your comments. Can we do another round?

[s1monw]

LGTM

[s1monw]

I actually think we should use AbstractRunnable here

     threadPool.generic().execute(new AbstractRunnable() {
            @Override
            public void onFailure(Throwable t) {
              // call listener
            }

            @Override
            protected void doRun() throws Exception {
 RecoveriesCollection.StatusRef statusRef = onGoingRecoveries.getStatus(recoveryId);
                if (statusRef == null) {
                    return;
                }
                try {
                    doRecovery(statusRef.status());
                } finally {
                    // make sure we never interrupt the thread after we have released it back to the pool
                    statusRef.status().clearWaitingRecoveryThread(Thread.currentThread());
                    statusRef.close();
                }
            }

WDYT?

[bleskes]

++

[bleskes]

@s1monw I pushed another commit based on the latest feedback. I'm giving CI an hour or two to chew this and will then merge.

[bleskes]

merged. Thx @s1monw !