Skip to content

[Jepsen] During a partition, tservers can log ~250,000 "Leader Master has changed" messages per minute #823

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
aphyr opened this issue Jan 31, 2019 · 0 comments
Closed

[Jepsen] During a partition, tservers can log ~250,000 "Leader Master has changed" messages per minute #823

aphyr opened this issue Jan 31, 2019 · 0 comments
Assignees
@bbaddepudi
Labels
community/request Issues created by external users kind/enhancement This is an enhancement of an existing feature

Comments

@aphyr
Copy link

aphyr commented Jan 31, 2019

In Jepsen tests, during partitions, version 1.1.10.0 (CE) can get into a mode where tservers will log ~250,000 lines/sec repeating:

W0131 17:50:07.581857 48755 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.582269 48756 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.582762 48757 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.583237 48761 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.583580 48754 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.584204 48755 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.584537 48758 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.584877 48760 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.585235 48761 meta_cache.cc:562] Leader Master has changed, re-trying...
W0131 17:50:07.585634 48757 meta_cache.cc:562] Leader Master has changed, re-trying...

It might be worth reducing the frequency of this log message to cut down on disk use and IO contention?
@kmuthukk kmuthukk added the kind/enhancement This is an enhancement of an existing feature label Jan 31, 2019
@ravimurthy ravimurthy assigned bbaddepudi and unassigned ravimurthy Feb 1, 2019
yugabyte-ci pushed a commit that referenced this issue Feb 1, 2019
@bbaddepudi
ENG-4619: During a partition, tservers can log ~250,000 "Leader Maste…
0274a1c 
…r has changed" messages per minute #823

Summary:
Reduce the spew of messages from the client (embedded in tserver) when master leader is being re-detected, as it could be in the process of being elected/changed.

Test Plan: Jenkins

Reviewers: ravi, amitanand

Reviewed By: amitanand

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D6094
@mbautin mbautin changed the title During a partition, tservers can log ~250,000 "Leader Master has changed" messages per minute [Jepsen] During a partition, tservers can log ~250,000 "Leader Master has changed" messages per minute Feb 7, 2019
@yugabyte-ci yugabyte-ci added the community/request Issues created by external users label Jul 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bbaddepudi bbaddepudi

Labels
community/request Issues created by external users kind/enhancement This is an enhancement of an existing feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@aphyr @ravimurthy @bbaddepudi @kmuthukk @yugabyte-ci