Fix waiting for recovery for red indices.

[dennisoelkers]

Description

Motivation and Context

When one or more ES nodes are in flood stage, cycling the deflector fails due to not being able to create a new index and toggling the deflector alias. During this step, the code waits for a new index to be created and healthy. Unfortunately, the waitForRecovery method used internally does not deal properly with indices being red. The response we are receiving when querying the health of a red index is set to unsuccessful, triggering our abstracted error handling mechanism which is throwing an exception instead of passing through the red health state.

This change is now using a different, less safe way of executing the health query request and returning the response as is. This allows waiting for recovery on red indices and timing out instead of throwing an exception.

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactoring (non-breaking change)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.

[bernd]

Hmm, I think this is an issue because most consumers of Indices#waitForRecovery(String) expect the method to throw an error and don't use the return value. We would need to adjust all consumers to check the return value and abort or retry their execution flow if an index is red or the timeout kicks in.