don't allow proxying requests to self

[ryane]

Even with the change in #463, it still appears to be possible to end up in a state where a supposed non-leader instance ends up getting itself as the leader path. When this happens, in RedirectFilter, jobScheduler.isLeader == false but jobScheduler.getLeader returns the path of the current instance. This results in chronos redirecting all requests to itself, and, ultimately an unresponsive REST api. It never appears to be able to recover when in this state. If the other chronos instances think that the stuck node is the leader, they will also end up being unresponsive resulting in an unusable chronos cluster.

Unfortunately, it is a bit difficult to reproduce the problem. One way I have found where I can duplicate it with some consistency is to have 3 servers, each running zookeeper, chronos, and mesos. Then, reboot each server serially. Often (but not always), one of the Chronos instances will have proxied all requests to itself and will be unresponsive.

I am not very familiar with the chronos code (or scala for that matter) so there may be a better way to handle this. And, it does not address the root cause of why and how JobScheduler ends up thinking it is not a leader but yet still returns the current instance's path from getLeader - I have not been able to figure that out. But, this commit does seem to prevent chronos from proxying requests to itself and ending up in the unresponsive state.

[kolloch]

Hi @ryane, thanks for your pull request.

If we get a request and we do not have consistent leadership information, we should probably wait for consistent leadership information or reject the request.

That's what we have done for Marathon. See here:

https://github.com/mesosphere/marathon/blob/master/src/main/scala/mesosphere/marathon/api/LeaderProxyFilter.scala#L121-L152

[pdericson]

👍

[brndnmtthws]

Thanks!