Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Forbid using canBeReadonly when connecting zookeeper #463

Merged
merged 1 commit into from
Jun 5, 2015
Merged

Forbid using canBeReadonly when connecting zookeeper #463

merged 1 commit into from
Jun 5, 2015

Conversation

depay
Copy link
Contributor

@depay depay commented Jun 4, 2015

Chronos may get a wrong leader path, which may cause a fatal error that:
once the non-leader instance gets an out-of-date leader path and the wrong
path happens to be itself, it will redirects a request to itself and
then blocks other any requests in less than 1 second. In this circumstance,
scheduler is still running but no rest api responds, even it becomes leader.

So, setting connecting parameter 'canBeReadOnly' to false(default) to provide
Chronos from reading wrong leader info is necessary.

Chronos may get a wrong leader path, which may cause a fatal error that:
once the non-leader instance gets an out-of-date leader path and the wrong
path happens to be itself, it will redirects a request to itself  and
then blocks other any requests in less than 1 second. In this circumstance,
scheduler is still running but no rest api responds, even it becomes leader.

So, setting connecting parameter 'canBeReadOnly' to false(default) to provide
Chronos from reading wrong leader info is necessary.
@depay
Copy link
Contributor Author

depay commented Jun 4, 2015

#285

@elingg
Copy link
Member

elingg commented Jun 4, 2015

Thanks @depay! This change makes sense, and we need to fix this issue so I will merge. Question though, why does the leader path become correct? Shouldn't the HA be properly managed by Curator?

@depay
Copy link
Contributor Author

depay commented Jun 5, 2015

@elingg it seems that something wrong with zookeeper at the very moment, chronos gets
the leader-changing event but cannot connect to the zookeeper leader, so it reads a leader path from another zk node. This weak-consistency strategy is supplied by zookeeper directly. Nothing wrong with curator.

canBeReadOnly - (added in 3.4) whether the created client is allowed to go
to read-only mode in case of partitioning. Read-only mode basically means
that if the client can't find any majority servers but there's partitioned server
it could reach, it connects to one in read-only mode, i.e. read requests are
allowed while write requests are not. It continues seeking for majority in the
background. 

@modeyang
Copy link

modeyang commented Jun 5, 2015

LGTM, thanks @depay, the read-only flag need be false when leader switch.

@elingg
Copy link
Member

elingg commented Jun 5, 2015

Thanks for the explanation, @depay. 🚢

elingg added a commit that referenced this pull request Jun 5, 2015
Forbid using canBeReadonly when connecting zookeeper
@elingg elingg merged commit 2f9b7d3 into mesos:master Jun 5, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants