Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Jepsen] During a partition, tservers can log ~250,000 "Leader Master has changed" messages per minute #823

Closed
aphyr opened this Issue Jan 31, 2019 · 0 comments

Comments

4 participants
@aphyr
Copy link

commented Jan 31, 2019

In Jepsen tests, during partitions, version 1.1.10.0 (CE) can get into a mode where tservers will log ~250,000 lines/sec repeating:

W0131 17:50:07.581857 48755 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.582269 48756 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.582762 48757 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.583237 48761 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.583580 48754 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.584204 48755 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.584537 48758 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.584877 48760 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.585235 48761 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.585634 48757 meta_cache.cc:562] Leader Master has changed, re-trying...

It might be worth reducing the frequency of this log message to cut down on disk use and IO contention?

@kmuthukk kmuthukk added this to To Do in YBase features via automation Jan 31, 2019

@ravimurthy ravimurthy assigned bbaddepudi and unassigned ravimurthy Feb 1, 2019

yugabyte-ci pushed a commit that referenced this issue Feb 1, 2019

ENG-4619: During a partition, tservers can log ~250,000 "Leader Maste…
…r has changed" messages per minute #823

Summary:
Reduce the spew of messages from the client (embedded in tserver) when master leader is being re-detected, as it could be in the process of being elected/changed.

Test Plan: Jenkins

Reviewers: ravi, amitanand

Reviewed By: amitanand

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D6094

@bbaddepudi bbaddepudi closed this Feb 1, 2019

YBase features automation moved this from To Do to Done Feb 1, 2019

@mbautin mbautin changed the title During a partition, tservers can log ~250,000 "Leader Master has changed" messages per minute [Jepsen] During a partition, tservers can log ~250,000 "Leader Master has changed" messages per minute Feb 7, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.