Fixing overlord issued too many redirects

[cryptoe]

Description

While debugging a cluster, I found that during an overlord switch, the tasks were failing.

The task log showed

Caused by: org.apache.druid.rpc.RpcException: Service [overlord] issued too many redirects
	at org.apache.druid.rpc.ServiceClientImpl$1.onSuccess(ServiceClientImpl.java:203)
	at org.apache.druid.rpc.ServiceClientImpl$1.onSuccess(ServiceClientImpl.java:168)
	at com.google.common.util.concurrent.Futures$4.run(Futures.java:1181)

Which lead me to debug the overlord redirect code.

The RedirectFilter on the overlord checks if the current node is the elected leader and is initialized using TaskMaster.isLeader() and if no, redirect it to the elected leader using overlordLeaderSelector.getCurrentLeader().

Now the redirectFilter was redirecting requests to itself resulting in a never-ending loop until the initialized flag is set hence causing issued too many redirects for the clients.

Note that the new leader may take some time to set the initialized flag as it has to fetch tasks/locks from the metadatastore.

Adjusted the code in such a way that the redirectFilter will not redirect the requests to itself resulting in a SC_SERVICE_UNAVAILABLE = 503 which will cause the overlord clients to back off exponentially which IMHO is the correct behaviour.

---

Key changed/added classes in this PR

• TaskMaster

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[gianm]

The error message here isn't right: the url isn't known to be invalid at this point. It may very well be valid. It just exceeded the limit on number of redirects.