The only other place that 503 is used in ES is when rejecting requests due to thread poll constraints, and I think 503 is correct for it?

Yes agreed - that is an appropriate use case.

If those are the only two places, then I'll close this issue.

but what will we do in the clients then? thread pool rejection is different compared to a node returning 503 because its no longer part of the cluster... . I do think we need to somehow tell the different from the clients perspective?

OK - reopened. 503 for me means: retry on another node. I guess if a request on another node gets rerouted back to the same node you could end up with the same 503 response?

What should the clients do for thread pool rejection then?

yea, exactly, a 503 because of thread pool rejection is different, and internally, the language client should not retry, but expose it as a proper failure to the user code, and then they can decide what to do (back off, ...)

maybe we can use the Retry-After header, which will indicate high load (and try to come up with a default value for it :))

@uboness we can't really put a value there..., I think its more geared towards single "usage" type, we can't really sensibly give any value there

yeah.. I know.. hence the :).. that's the closes "official" way I thought of to distinguish between the two scenarios

Maybe use a 502 for when the node is no longer part of the cluster? Not exact but if one considers the elected master the "upstream" server simply because it delivers cluster state needed in order to route queries it kinda works.

Fixed by #6627