[DNM][Downstream][hotfix]mon/Elector: Change how we handle removed_ranks and notify_rank_removed() #48698

kamoltat · 2022-11-02T02:15:03Z

Problem:

Currently, there is an issue when performing a DR test with 2 sites stretch cluster
where removing monitors and adding new ones to the cluster
causes incorrect rank in ConnectionTracker class.

This causes the monitor to think that they are someone else
in the peer_tracker copy and will never update the
correct field of itself, causing a deadlock in the election process (Ceph becoming unresponsive)
when using election strategy: 3 (Connectivity mode).

Solution:

It was really hard to debug the issue so the first thing we did was to add additional loggings to ConnectionTracker,
Elector and ElectionLogic Classes.
In ConnectionTracker::receive_peer_report we loop through ranks which is bad when there is notify_rank_removed before this and the ranks are not adjusted yet. When we rely on the rank in certain scenarios, we end up with extra peer_report copy which we don't want. Therefore, instead of passing report.rank in the function
ConnectionTracker::reports, we pass i.first instead so that we trim old ranks properly. We also added an assert in notify_rank_removed(), comparing the expected rank provided by the monmap against the rank that we adjust ourselves to as a sanity check. We edited test/mon/test_election.cc to reflect the changes made in notify_rank_removed().
MonMap::removed_rank does not get cleared every update of the epoch, this was the root cause of the problem. Therefore, we fix this by making MonMap clear removed_ranks every update. Moreover, When there is discontinuity between monmaps for more than 1 epoch or the new monitor never joined the quorum before,
we always reset peer_tracker.
Added a way for us to manually reset peer_tracker.rank when executing the command: ceph connection scores reset for each monitor. The peer_tracker.rank will match the current rank of the Monitor.
When upgrading the monitors (including booting up), we check if peer_tracker is dirty or not. If so, we clear it. Added some functions in the Elector and ConnectionTracker classes to check for clean peer_tracker. Moreover, there could be some cases where due to startup weirdness or abnormal circumstances, we might get a report from our own rank. Therefore, it doesn't hurt to add a sanity check in ConnectionTracker::report_live_connection and ConnectionTracker::report_dead_connection.

Fixes: https://tracker.ceph.com/issues/58049

Signed-off-by: Kamoltat ksirivad@redhat.com

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

kamoltat · 2022-11-02T02:21:17Z

Hi @gregsfortytwo

I have tested this by running my own reproducer local using vstart.sh, removing and adding monitors in stretch_cluster.
So far I was able to come up with this fix to prevent peer_tracker.rank from being adjusted incorrectly and for removed_ranks to store old state so that we don't incorrectly remove ranks and adjust them.

Please only look at the commits that have actual code changes and not just logging:

9c2af4d
ebe20d9
b223a71

Also, I was thinking of separating elector.notify_rank_removed() from notify_new_monmap().
So that we don't have to pass extra flags in notify_new_monmap() when we don't want to call notify_rank_removed().
However, I don't want to overcomplicate things and I have more peace of mind when I don't change too many things.

gregsfortytwo

Good find! There's at least one edge case still not covered, and a few others I'm worried about.

Plus of course figuring out what we want to do with all the new logging — much of it probably has value but at 20 or 30 instead of 10?

src/mon/ConnectionTracker.cc

src/mon/MonmapMonitor.cc

gregsfortytwo · 2022-11-02T18:33:50Z

src/mon/Monitor.cc

-      notify_new_monmap(false);
+      if (!has_ever_joined) {
+        dout(20) << " !has_ever_joined " << *monmap << dendl;
+        notify_new_monmap(false, false);


Hmm. I forget when we start pinging and generating ConnectionTracker state. Is the following possible?

New monitor turns on
Start pinging existing 5 monitors
Start copying paxos state from mon.1
mon.4 is removed from monitor list
finish copying state
get new map with mon.4 in removed_ranks
fail to remove because we are !has_ever_joined

Hi Greg,

How about we only allow mon remove when it has joined quorum before and only is 1 epoch different from the new map, for other cases like
never joined or joined but we have a very outdated version, we don't adjust and also make
sure to nuke peer_report, incase it has already been populated with like a removed_rank:

if (newmap->get_epoch() - monmap->get_epoch() == 1 && has_ever_joined) { notify_new_monmap(false); } else { notify_new_monmap(false, false); elector.notify_clear_peer_state(); }

EDIT: hmm not sure I understood correctly.

That code snippet...probably works? As I noted below, we probably need to toss out our state whenever we have a discontinuity in witnessed monmaps.

by state, do you mean peer_tracker in the elector? if so I guess you mean even if we are 1 epoch apart we should always elector.notify_clear_peer_state();

Right, if we see a discontinuity we need to reset our scores because we don't know what might have happened to the monitor set and how the scores should align.

src/mon/Monitor.cc

kamoltat · 2022-11-03T01:56:42Z

Hi @gregsfortytwo

thank you for the comments, I will get to them shortly.

I just pushed a commit that rewrites ConnectionTracker::notify_rank_removed():

032629c

the motivations are in the commit message.

But in short, I believe by doing it this way will make the peer_tracker cleaner and less prone to
getting into a deadlock state during the election.

I have tested this in my local vstart, removing the lowest rank, highest rank, middle and etc.

kamoltat · 2022-11-04T02:09:26Z

@gregsfortytwo Just updated this PR with regards to your comments, I also added the mechanism where we can reset peer_tracker.rank to match mon. rank, this is embedded in the function: notify_clear_peer_state(). Which gets triggered when we do ceph connection scores reset.

kamoltat · 2022-11-07T20:32:35Z

jenkins test api

kamoltat · 2022-11-14T00:11:43Z

jenkins test api

gregsfortytwo · 2022-11-14T17:41:19Z

src/mon/ConnectionTracker.cc

-    peer_reports.erase(pi++);
+  // Go through everyone, remove & adjust current and history map.
+  for (auto pi = peer_reports.begin(); pi != peer_reports.end(); pi++) {
+    peer_reports[pi->first].current.erase(rank_removed);


pi->second.current.erase(rank_removed)

yep good catch!

gregsfortytwo · 2022-11-14T17:41:41Z

src/mon/ConnectionTracker.cc

+  // Go through everyone, remove & adjust current and history map.
+  for (auto pi = peer_reports.begin(); pi != peer_reports.end(); pi++) {
+    peer_reports[pi->first].current.erase(rank_removed);
+    peer_reports[pi->first].history.erase(rank_removed);


pi->second.history.erase(rank_removed)

etc etc in this loop. Is there a reason you're looking up the peer_report each time?

nope, should just use pi->second

gregsfortytwo · 2022-11-14T17:52:14Z

src/mon/ConnectionTracker.cc

+        peer_reports[pi->first].history[hi->first - 1] = hi->second;
+        peer_reports[pi->first].current.erase(ci++);
+        peer_reports[pi->first].history.erase(hi++);
+    }


So I really, really don't like this. The peer_reports mechanism is designed so only the monitor generating a report updates the numbers, and then we assimilate them. Directly manipulating them like this is icky, and actually can lead to differing election outcomes during an upgrade from the old-way monitors to this way.

From the commit message, I think this is the real problem:

However, there is a case where if we remove the highest rank of the quorum, then we will leave a the higest rank in report.current and report.history

If we just resolve that, which sounds like a fairly simple bug in ConnectionTracker::receive_peer_reports, I don't think we need to do this twiddling.

I think even with the old code, we are already directly manipulating peer_reports by adjusting ranks: https://github.com/ceph/ceph/blob/main/src/mon/ConnectionTracker.cc#L171-L176.
my thoughts are that since we are already adjusting the ranks of peer_reports, might as well adjust the ranks of peer_reports.current and peer_reports.history as well so that they are aligned with each other.

I understand your concerns about manipulating them, but if we go with that, then we should just only remove and not adjust the ranks in peer_reports at all correct?

I think even with the old code, we are already directly manipulating peer_reports by adjusting ranks: https://github.com/ceph/ceph/blob/main/src/mon/ConnectionTracker.cc#L171-L176. my thoughts are that since we are already adjusting the ranks of peer_reports, might as well adjust the ranks of peer_reports.current and peer_reports.history as well so that they are aligned with each other.

I understand your concerns about manipulating them, but if we go with that, then we should just only remove and not adjust the ranks in peer_reports at all correct?

That link is just removing scores, not updating the individual reports?

The old code moves the peer_report down so in a way it is adjusting the ranks implicitly since functions like get_total_connection_score rely on the position of the peer_report rather than the actual peer_report.rank.

If we use the old code we'll have this in this scenario:

we are in mon.0 point of view. notify_rank_removed: 2 peer_reports before remove 2: { 0=rank=0, current links: {1=1, 2=1, 3=1, 4=1}, history: {1=0.9, 2=0.9 3=0.9, 4=0.9}, 1=rank=1, current links: {0=1, 2=1, 3=1, 4=1}, history: {0=0.9, 2=0.9, 3=0.9, 4=0.9}, 2=rank=2, current links: {0=1, 1=1, 3=1, 4=1}, history: {0=0.9, 2=0.9, 3=0.9, 4=0.9},  3=rank=3, current links: {0=1, 1=1, 2=1, 4=1}, history: {0=0.8, 1=0.7, 2=0.6, 4=0.9}, 4=rank=4, current links: {0=1, 1=1, 2=1, 3=1}, history: {0=0.9, 1=0.6, 2=0.5, 3=0.7}, } peer_reports after remove 2: { 0=rank=0, current links: {1=1, 2=1, 3=1, 4=1}, history: {1=0.9, 2=0.9 3=0.9, 4=0.9}, 1=rank=1, current links: {0=1, 2=1, 3=1, 4=1}, history: {0=0.9, 2=0.9, 3=0.9, 4=0.9},  2=rank=3, current links: {0=1, 1=1, 2=1, 4=1}, history: {0=0.8, 1=0.7, 2=0.6, 4=0.9}, 3=rank=4, current links: {0=1, 1=1, 2=1, 3=1}, history: {0=0.9, 1=0.6, 2=0.5, 3=0.7}, }

So now, rank 4 becomes rank 3, and rank 3 becomes rank 2.

let's say mon.0 call get_total_connection_score peer 3, peer_report[2] would give us nothing, but it should have given us 0.9.

So what I am trying to say is that since we are moving the ranks down by 1 already with the old code. I don't think it is fair to not move the ranks in current and history.

Moreover, the fact that the old code doesn't adjust the actual peer_report.rank leaves us a vulnerability as well since let's say after we remove ranks with the old code and we ping another monitor and they happened to assimilate_connection_reports of the peer_report provided by us, then with our unadjusted peer_report.rank, we will populate the incorrect information on their peer_reports and this also can lead to differing election outcomes.

@gregsfortytwo PTAL

src/mon/MonMap.cc

src/mon/ConnectionTracker.cc

src/mon/Monitor.cc

…ElectionLogic & Elector Problem: Currently there are not ConnectionTracker logs, therefore it is really hard to debug Solution: Enable loggings for most functions in ConnectionTracker.cc Most of the logs are in debug_mon = 30. Also Added some logs in Elector and ElectionLogic so that debugging will be easier in the future. Signed-off-by: Kamoltat <ksirivad@redhat.com>

gregsfortytwo · 2022-11-18T23:01:30Z

LGTM!

PROBLEM: In `ConnectionTracker::receive_peer_report` we loop through ranks which is bad when there is `notify_rank_removed` before this and the ranks are not adjusted yet. When we rely on the rank in certain scenarios, we end up with extra peer_report copy which we don't want. SOLUTION: In `ConnectionTracker::receive_peer_report` instead of passing `report.rank` in the function `ConnectionTracker::reports`, we pass `i.first` instead so that trim old ranks properly. We also added a assert in notify_rank_removed(), comparing expected rank provided by the monmap against the rank that we adjust ourself to as a sanity check. We edited test/mon/test_election.cc to reflect the changes made in notify_rank_removed(). Signed-off-by: Kamoltat <ksirivad@redhat.com>

when a new monitor joins, there is a chance that it will recive a monmap that recently removed a monitor and ``removed_rank`` will have some content in it. A new monitor that joins should never remove rank in peer_tracker but rather call ``notify_clear_peer_state()`` to reset the `peer_report`. In the case when it is a monitor that has joined quorum before and is only 1 epoch behind the newest monmap provided by the probe_replied monitor. We can actually remove and adjust ranks in `peer_report` since we are sure that if there is any content in removed_ranks, then it has to be because in the next epoch we are removing a rank, since every update of an epoch we always clear the removed_ranks. There is no point in keeping the content of ``removed_ranks`` after monmap gets updated to the epoch. Therefore, clear ``removed_ranks`` every update. When there is discontinuity between monmaps for more 1 epoch or the new monitor never joined quorum before, we always reset `peer_tracker`. Moreover, beneficial for monitor log to also log which rank has been removed at the current time of the monmap. So add removed_ranks to `print_summary` and `dump` in MonMap.cc. Signed-off-by: Kamoltat <ksirivad@redhat.com>

In `notify_clear_peer_state()` we another mechanism in reseting our `peer_tracker.rank` to match our own monitor.rank. This is added so there is a way for us to recover from a scenrio where `peer_tracker.rank` is messed up from adjusting the ranks or removing ranks. `notifiy_clear_peer_state()` can be triggered by using the command: `ceph connection scores reset` Also in `clear_peer_reports`, besides reassigning my_reports to an empty object, we also have to make `my_reports` = `rank` from `peer_tracker`, such that we don't get -1 as a rank in my_reports. Signed-off-by: Kamoltat <ksirivad@redhat.com>

…d connection report When upgrading the monitors (include booting up), we check if `peer_tracker` is dirty or not. If so, we clear it. Added some functions in `Elector` and `ConnectionTracker` class to check for clean `peer_tracker`. Moreover, there could be some cases where due to startup weirdness or abnormal circumstances, we might get a report from our own rank. Therefore, it doesn't hurt to add a sanity check in `ConnectionTracker::report_live_connection` and `ConnectionTracker::report_dead_connection`. Signed-off-by: Kamoltat <ksirivad@redhat.com>

kamoltat · 2022-11-20T21:32:38Z

jenkins test make check

kamoltat · 2022-11-20T21:35:17Z

jenkins retest this please

kamoltat · 2022-11-21T15:06:37Z

make check failure unrelated, rbd-unit-tests are known to be flaky

12 - run-rbd-unit-tests-1.sh (Failed)

kamoltat · 2022-11-21T15:06:47Z

jenkins test make check

kamoltat · 2022-11-21T15:07:05Z

jenkins test api

ljflores · 2022-11-21T17:21:31Z

@kamoltat let me know when this is ready for testing. You can tag it with needs-qa and ping me.

kamoltat · 2022-11-21T21:10:36Z

@kamoltat let me know when this is ready for testing. You can tag it with needs-qa and ping me.

@ljflores thanks Laura, it's ready.

amathuria · 2022-11-29T17:24:59Z

@kamoltat one of the failures in the run had the following assert failure:
/a/yuriw-2022-11-28_21:10:48-rados-wip-yuri10-testing-2022-11-28-1042-pacific-distro-default-smithi/7095280

Raised a tracker for it here: https://tracker.ceph.com/issues/58114

-1> 2022-11-29T04:58:26.511+0000 7fc090d88700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10-1303-g8622b733/rpm/el8/BUILD/ceph-16.2.10-1303-g8622b733/src/mon/ConnectionTracker.cc: In function 'void ConnectionTracker::notify_rank_removed(int, int)' thread 7fc090d88700 time 2022-11-29T04:58:26.510305+0000
/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10-1303-g8622b733/rpm/el8/BUILD/ceph-16.2.10-1303-g8622b733/src/mon/ConnectionTracker.cc: 252: FAILED ceph_assert(rank == new_rank)

ceph version 16.2.10-1303-g8622b733 (8622b7333736b9290df89221ec0bb0873f4194dd) pacific (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fc09e4b3df4]
2: /usr/lib64/ceph/libceph-common.so.2(+0x27900e) [0x7fc09e4b400e]
3: (ConnectionTracker::notify_rank_removed(int, int)+0xd68) [0x560d07e4d6c8]
4: (Elector::notify_rank_removed(int, int)+0x123) [0x560d07e3b053]
5: (Monitor::notify_new_monmap(bool, bool)+0x2a1) [0x560d07da3ee1]
6: (MonmapMonitor::update_from_paxos(bool*)+0x994) [0x560d07df1b24]
7: (PaxosService::refresh(bool*)+0x10e) [0x560d07ed01fe]
8: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x560d07d7a3ac]
9: (Paxos::do_refresh()+0x57) [0x560d07ec26a7]
10: (Paxos::handle_commit(boost::intrusive_ptr<MonOpRequest>)+0x309) [0x560d07ec2b39]
11: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x457) [0x560d07ecafe7]
12: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1324) [0x560d07db62d4]
13: (Monitor::_ms_dispatch(Message*)+0x670) [0x560d07db6d60]
14: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x560d07de5f0c]
15: (DispatchQueue::entry()+0x126a) [0x7fc09e6fa5ea]
16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc09e7ad921]
17: /lib64/libpthread.so.0(+0x814a) [0x7fc09c1e614a]
18: clone()

It looked related to this change. The coredump is available as well:
/a/yuriw-2022-11-28_21:10:48-rados-wip-yuri10-testing-2022-11-28-1042-pacific-distro-default-smithi/7095280/remote/smithi203/coredump/

kamoltat · 2022-11-29T18:28:52Z

@amathuria thanks for letting me know, looking at it right now

kamoltat · 2022-11-29T20:45:07Z

I have investigated this thoroughly and I have concluded that this failure is due to how we are not properly using
CEPH_ARGS+="--mon-initial-members=a,b,c " in the standalone test which results in prepopulated "removed_ranks" which causes the assert. In summary, it's the standalone test's fault, I'm writing up the details as we speak will share it ASAP.

kamoltat · 2022-11-30T14:31:36Z

Analysis on my investigation:
https://docs.google.com/document/d/1Rk-PUZ6cwBT8a9Z5RAFY5sb8-xFx-K02k7l4T2p2U5Q/edit

Problem: --mon-initial-members does nothing but causes monmap to populate ``removed_ranks`` because the way we start monitors in standalone tests uses ``run_mon $dir $id ..`` on each mon. Regardless of --mon-initial-members=a,b,c, if we set --mon-host=$MONA,$MONB,$MONC (which we do every single tests), everytime we run a monitor (e.g.,run mon.b) it will pre-build our monmap with ``` noname-a=mon.noname-a addrs v2:127.0.0.1:7127/0, b=mon.b addrs v2:127.0.0.1:7128/0, noname-c=mon.noname-c addrs v2:127.0.0.1:7129/0, ``` Now, with --mon-initial-members=a,b,c we are letting monmap know that we should have initial members name: a,b,c, which we only have `b` as a match. So what ``MonMap::set_initial_members`` do is that it will remove noname-a and noname-c which will populate `removed_ranks`. Solution: remove all instances of --mon-initial-members in the standalone test as it has no impact on the nature of the tests themselves. Fixes: https://tracker.ceph.com/issues/58132 Signed-off-by: Kamoltat <ksirivad@redhat.com>

kamoltat · 2022-11-30T23:20:22Z

@gregsfortytwo I have decided to get rid of the --mon-initial-members setting in the standalone test, as I have concluded that it does not really play a role in the tests at all except pre-populated removed_ranks with members that they think were originally there but it was a result of monmap initialization based on address/host provided by --mon-host setting. The details can be found in the commit: b143cc9

kamoltat · 2022-12-01T17:16:17Z

jenkins test api

kamoltat · 2022-12-07T20:15:19Z

Closing and moving this PR to #49312
For backport cleanliness of upstream PRs.

kamoltat requested a review from gregsfortytwo November 2, 2022 02:15

kamoltat requested a review from a team as a code owner November 2, 2022 02:15

kamoltat self-assigned this Nov 2, 2022

kamoltat added this to the pacific milestone Nov 2, 2022

kamoltat added the mon label Nov 2, 2022

github-actions bot added the core label Nov 2, 2022

kamoltat added bug-fix and removed core labels Nov 2, 2022

kamoltat requested a review from neha-ojha November 2, 2022 02:21

gregsfortytwo requested changes Nov 2, 2022

View reviewed changes

github-actions bot added the core label Nov 3, 2022

kamoltat force-pushed the wip-ksirivad-pacific-bz-2121452 branch 5 times, most recently from 69f9efa to a738e38 Compare November 4, 2022 01:46

kamoltat force-pushed the wip-ksirivad-pacific-bz-2121452 branch 2 times, most recently from fe01fbe to d56f185 Compare November 7, 2022 17:00

github-actions bot added the tests label Nov 7, 2022

kamoltat force-pushed the wip-ksirivad-pacific-bz-2121452 branch 3 times, most recently from 9758948 to c619de7 Compare November 7, 2022 19:43

kamoltat force-pushed the wip-ksirivad-pacific-bz-2121452 branch from ae8261a to c70929a Compare November 13, 2022 23:19

gregsfortytwo reviewed Nov 14, 2022

View reviewed changes

kamoltat force-pushed the wip-ksirivad-pacific-bz-2121452 branch from e3d7bda to 856fef5 Compare November 19, 2022 04:13

kamoltat added 4 commits November 20, 2022 00:05

kamoltat force-pushed the wip-ksirivad-pacific-bz-2121452 branch from 856fef5 to 3c49a06 Compare November 20, 2022 00:09

kamoltat removed the request for review from a team November 20, 2022 21:32

kamoltat added the needs-qa label Nov 21, 2022

yuriw added the wip-yuri10-testing label Nov 28, 2022

yuriw removed needs-qa wip-yuri10-testing labels Nov 29, 2022

kamoltat added the needs-qa label Dec 1, 2022

kamoltat mentioned this pull request Dec 7, 2022

mon/Elector: Change how we handle removed_ranks and notify_rank_removed() #48991

Merged

14 tasks

kamoltat closed this Dec 7, 2022

[DNM][Downstream][hotfix]mon/Elector: Change how we handle removed_ranks and notify_rank_removed() #48698

[DNM][Downstream][hotfix]mon/Elector: Change how we handle removed_ranks and notify_rank_removed() #48698

kamoltat commented Nov 2, 2022 •

edited

kamoltat commented Nov 2, 2022

gregsfortytwo left a comment

gregsfortytwo Nov 2, 2022

kamoltat Nov 3, 2022 •

edited

gregsfortytwo Nov 14, 2022 •

edited

kamoltat Nov 14, 2022

gregsfortytwo Nov 14, 2022

kamoltat commented Nov 3, 2022

kamoltat commented Nov 4, 2022 •

edited

kamoltat commented Nov 7, 2022

kamoltat commented Nov 14, 2022

gregsfortytwo Nov 14, 2022

kamoltat Nov 14, 2022

gregsfortytwo Nov 14, 2022

kamoltat Nov 14, 2022

gregsfortytwo Nov 14, 2022

kamoltat Nov 14, 2022

gregsfortytwo Nov 14, 2022

kamoltat Nov 14, 2022

kamoltat Nov 14, 2022 •

edited

kamoltat Nov 15, 2022

gregsfortytwo commented Nov 18, 2022

kamoltat commented Nov 20, 2022

kamoltat commented Nov 20, 2022

kamoltat commented Nov 21, 2022

kamoltat commented Nov 21, 2022

kamoltat commented Nov 21, 2022

ljflores commented Nov 21, 2022

kamoltat commented Nov 21, 2022

amathuria commented Nov 29, 2022

kamoltat commented Nov 29, 2022

kamoltat commented Nov 29, 2022

kamoltat commented Nov 30, 2022

kamoltat commented Nov 30, 2022

kamoltat commented Dec 1, 2022

kamoltat commented Dec 7, 2022

[DNM][Downstream][hotfix]mon/Elector: Change how we handle removed_ranks and notify_rank_removed() #48698

[DNM][Downstream][hotfix]mon/Elector: Change how we handle removed_ranks and notify_rank_removed() #48698

Conversation

kamoltat commented Nov 2, 2022 • edited

Contribution Guidelines

Checklist

kamoltat commented Nov 2, 2022

gregsfortytwo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kamoltat Nov 3, 2022 • edited

Choose a reason for hiding this comment

gregsfortytwo Nov 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kamoltat commented Nov 3, 2022

kamoltat commented Nov 4, 2022 • edited

kamoltat commented Nov 7, 2022

kamoltat commented Nov 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kamoltat Nov 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gregsfortytwo commented Nov 18, 2022

kamoltat commented Nov 20, 2022

kamoltat commented Nov 20, 2022

kamoltat commented Nov 21, 2022

kamoltat commented Nov 21, 2022

kamoltat commented Nov 21, 2022

ljflores commented Nov 21, 2022

kamoltat commented Nov 21, 2022

amathuria commented Nov 29, 2022

kamoltat commented Nov 29, 2022

kamoltat commented Nov 29, 2022

kamoltat commented Nov 30, 2022

kamoltat commented Nov 30, 2022

kamoltat commented Dec 1, 2022

kamoltat commented Dec 7, 2022

kamoltat commented Nov 2, 2022 •

edited

kamoltat Nov 3, 2022 •

edited

gregsfortytwo Nov 14, 2022 •

edited

kamoltat commented Nov 4, 2022 •

edited

kamoltat Nov 14, 2022 •

edited