Improve peer recovery performance in active indexing

[dnhatn]

Phase2 of a peer recovery can take a lot of time if the index is large when the recovery starts, and we are actively indexing to it. The reason is that we have to replay a huge number of operations that occur while we are transferring the segment files. I see two things that we can improve:

• Send/replay operations concurrently like we are doing for file chunks

• Before starting phase2, we check the total operations that we will replay. If the number of the pending operations crosses the file-based recovery threshold, then it would be better to start over the recovery.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Recovery)

[dnhatn]

We discussed this two weeks ago. The root cause of the related issue is that the setting cluster.routing.allocation.node_concurrent_recoveries was set too high. We agreed to introduce a limit for that setting. We also agreed to support concurrency in phase 2 of peer recoveries but disable it by default. I will close this issue and continue working on the PR.