Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mon: handle monitor lag when killing mgrs #18268

Merged
merged 1 commit into from
Nov 1, 2017
Merged

mon: handle monitor lag when killing mgrs #18268

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
25 changes: 24 additions & 1 deletion src/mon/MgrMonitor.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -490,7 +490,28 @@ void MgrMonitor::tick()
return;

const auto now = ceph::coarse_mono_clock::now();
const auto cutoff = now - std::chrono::seconds(g_conf->get_val<int64_t>("mon_mgr_beacon_grace"));

const auto mgr_beacon_grace = std::chrono::seconds(
g_conf->get_val<int64_t>("mon_mgr_beacon_grace"));

// Note that this is the mgr daemon's tick period, not ours (the
// beacon is sent with this period).
const auto mgr_tick_period = std::chrono::seconds(
g_conf->get_val<int64_t>("mgr_tick_period"));

if (last_tick != ceph::coarse_mono_clock::time_point::min()
&& (now - last_tick > (mgr_beacon_grace - mgr_tick_period))) {
// This case handles either local slowness (calls being delayed
// for whatever reason) or cluster election slowness (a long gap
// between calls while an election happened)
dout(4) << __func__ << ": resetting beacon timeouts due to mon delay "
"(slow election?) of " << now - last_tick << " seconds" << dendl;
for (auto &i : last_beacon) {
i.second = now;
Copy link
Contributor

@tchaikov tchaikov Oct 13, 2017 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't think we should be so generous to update all mgr's beacon timestamps. instead, i think we might want to use last_tick + mgr_tick_period - mgr_beacon_grace as the value of cutoff to offset some of the time because of our own fault (slowness).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, the idea is that if we have gone slow, then there are probably MgrBeacon messages queued up on peons waiting to be sent through -- the idea of resetting the last_beacons to now is that we will then be giving them mgr_beacon_grace to forward those beacons (assuming they should come soon, now that the mons are back online).

If we used an offset of (now - last_tick) - mgr_tick_period to account for the slowness with a specific number of seconds, then we would only be accounting for that this time through tick(), and we would then be strict again next time through tick -- the effect would be to only allow mgr_tick_period to receive the next beacons, instead of mgr_beacon_grace.

Because we see this behaviour on systems that are slow/laggy, I think it's better to use the more generous approach to allow for peons that are also being slow to forward messages after the cluster recovers (they're swapping a lot or something).

Copy link
Contributor

@tchaikov tchaikov Nov 1, 2017 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh, right! thanks for your explanation. makes sense to me. the next time when the laggy mon ticks, the innocent mgr(s) won't survive when the fix proposed by me. but they will with your fix.

}
}

last_tick = now;

// Populate any missing beacons (i.e. no beacon since MgrMonitor
// instantiation) with the current time, so that they will
Expand All @@ -508,6 +529,7 @@ void MgrMonitor::tick()
// Cull standbys first so that any remaining standbys
// will be eligible to take over from the active if we cull him.
std::list<uint64_t> dead_standbys;
const auto cutoff = now - mgr_beacon_grace;
for (const auto &i : pending_map.standbys) {
auto last_beacon_time = last_beacon.at(i.first);
if (last_beacon_time < cutoff) {
Expand Down Expand Up @@ -566,6 +588,7 @@ void MgrMonitor::on_restart()
{
// Clear out the leader-specific state.
last_beacon.clear();
last_tick = ceph::coarse_mono_clock::now();
}


Expand Down
5 changes: 5 additions & 0 deletions src/mon/MgrMonitor.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,11 @@ class MgrMonitor: public PaxosService
void count_metadata(const string& field, std::map<string,int> *out);

friend class C_Updated;

// When did the mon last call into our tick() method? Used for detecting
// when the mon was not updating us for some period (e.g. during slow
// election) to reset last_beacon timeouts
ceph::coarse_mono_clock::time_point last_tick;
};

#endif