Propagate index errors in field_caps

[cbuescher]

Currently we don't report any exceptions occuring during field_caps requestrs back to the user.
This PR adds a new failure section to the response which contains exceptions per index, keyed by
the index name the error occured in. In addition the response contains another field failed_indices
which reports the number of indices that threw an exception. If all of the requested indices failed,
the whole request fails with a 500 error code, otherwise the request succeeds and it is up to the caller
to check for potential errors in the response body.

Closes #68994

[cbuescher]

This is an example of the response with failures with the changes in this PR:

{
	"indices": [...],
	"fields": {
		...
	},
	"failed_indices": 1,
	"failures": {
		"errorindex": {
			"error": {
				"root_cause": [{
					"type": "illegal_argument_exception",
					"reason": "test"
				}],
				"type": "illegal_argument_exception",
				"reason": "test"
			}
		}
	}
}

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[jimczi]

The approach looks good to me. I left one comment regarding the reporting of the exception.

[cbuescher]

@jimczi thanks for the review, I will rework the part around how to transport back the exceptions.

[cbuescher]

@jimczi I pushed and update reworking how the exceptions get transported back.

[jimczi]

This case is not handled automatically. We need to add all the indices from that cluster in the index failures map. Since all indices on that cluster failed, maybe that the key in the map could be the cluster name instead ?

[cbuescher]

Individual index failures on a remote should be handled (there is also a test for it now that should fail otherwise). So you mean this should cover general remote failures? How could this happen? Also asking since I cannot thing around a way of testing this atm...

[jimczi]

Looks good to me. Could you add a test a yml test for ccs as well ?

[jimczi]

Do we need this logic now that we throw an exception if all indices fail ? We could even assert in the ctr that failures must be empty if indices is empty.

[cbuescher]

@jimczi thanks, I pushed changes that remove the rest status logic you mentioned, reworked the failure collection a bit to be all on a central place and added the test you asked for.

[jimczi]

LGTM, thanks @cbuescher !