Node shut down during the last phase of recovery needlessly fails shard

[bleskes]

During the final stage of recovery, the target shard is being moved to POST_RECOVERY and the master is sent a request to activate the shard. At the point the master reports the cluster as green, i.e., it is safe to shut down a node without loosing data (potentially going to yellow). However, if the source node is shut down quickly enough before the recovery code cleanly finishes, we may fail the new copy resulting in a red index.

This is an issue with a non-released refactoring done on 1.x and master.

See:
http://build-us-00.elasticsearch.org/job/es_g1gc_1x_metal/3366/testReport/junit/org.elasticsearch.recovery/FullRollingRestartTests/testFullRollingRestart/

[brwe]

I think this failure of RelocationTests.testMoveShardsWhileRelocation could be caused by the same effect: http://build-us-00.elasticsearch.org/job/es_core_1x_small/1474/

[s1monw]

@bleskes I moved this out to 1.6

[bleskes]

This should be fixed with #9902