mon: check is_shutdown() in timer callbacks #14919

tchaikov · 2017-05-03T02:42:21Z

Fixes: http://tracker.ceph.com/issues/19825
Signed-off-by: Kefu Chai kchai@redhat.com

tchaikov · 2017-05-04T13:36:59Z

retest this please.

tchaikov · 2017-05-04T13:40:22Z

@liewegas mind taking a look at this one also?

liewegas · 2017-05-04T13:57:30Z

We have lots of code like

void Elector::shutdown()
{
  if (expire_event)
    mon->timer.cancel_event(expire_event);
}

They won't break (cancel_event behaves if the event is already gone), but it makes me nervous having these dangling pointers to freed memory. I think this will be safer than maintaining all of the shutdown timer checks, though!

liewegas · 2017-05-04T13:57:48Z

src/mon/Monitor.cc

@@ -899,6 +899,10 @@ void Monitor::shutdown()

  wait_for_paxos_write();

+  // stop timer earlier so the event callbacks won't panic seeing an inactive


s/earlier/early/

tchaikov · 2017-05-04T14:16:16Z

@liewegas how about exposing the cancel_event() methods of all timer consumers in monitor and call them before calling timer.shutdown()?

tchaikov · 2017-05-04T14:17:29Z

changelog

s/earlier/early/

liewegas · 2017-05-04T14:24:54Z

Hrm my other worry is that we haven't stopped the finisher yet and it's possible one of those events will try to schedule something new. I'm leaning toward making sure every shutdown properly cancels all of its events... it seems like the bug here is just that Paxos::shutdown(), or whatever method is triggered when is_leader() switches to false, didn't clean up its events like it should have?

tchaikov · 2017-05-04T14:39:15Z

even if Paxos cancels all its events, there is still chance that the lease_timeout_event callback can be called after the is_leader() returns false. please note we change the mon's state before we shutdown the paxos.

maybe we can change mon's state to STATE_SHUTDOWN after paxos is shutdown?

tchaikov · 2017-05-04T14:54:31Z

changelog

change mon's state to STATE_SHUTDOWN after paxos is shutdown.

the latest change is different now. needs review.

tchaikov · 2017-05-04T14:56:14Z

adding [DNM], i haven't tested this yet.

liewegas · 2017-05-04T15:00:52Z

maybe the callback should just return and do nothing if is_leader is false then?

tchaikov · 2017-05-04T15:05:43Z

@liewegas there are a handful call backs checking the monitor state, i think we could check if (mon->is_shutdown()) in all of them, but if this approach works. i'd prefer go with postponing the "state = STATE_SHUTDOWN".

(i just skimmed through the code, seems it's safe to do so)

liewegas · 2017-05-04T17:59:36Z

src/mon/Monitor.cc

@@ -899,8 +899,6 @@ void Monitor::shutdown()

  wait_for_paxos_write();

-  state = STATE_SHUTDOWN;
-


I don't think we can do this either :( because we use is_shutdown() in places like ms_dispatch to drop new work on the floor. Without it, we will take in a start new work even after systems are shutting down..

liewegas · 2017-05-04T18:03:24Z

Looking a bit more it seems like checking for is_shutdown() in the callacks is the safest thing?

instead of doing it manually. Signed-off-by: Kefu Chai <kchai@redhat.com>

introduce a helper class: C_MonContext, and initialize all timer events using it, to ensure that they do check is_shutdown() before doing their work. Fixes: http://tracker.ceph.com/issues/19825 Signed-off-by: Kefu Chai <kchai@redhat.com>

tchaikov · 2017-05-05T05:57:15Z

changelog

check is_shutdown() in events hooked on monitor's timer

@liewegas fixed and repushed.

liewegas · 2017-05-05T13:27:58Z

src/mon/PaxosService.cc

-	    void finish(int r) override {
-	      ps->proposal_timer = 0;
+	  proposal_timer = new C_MonContext(mon, [this](int r) {
+	      proposal_timer = 0;


This won't run in is_shutdown(). I think it's okay in this case since timer tolerates a cancel_event on a non-event, though, and who cares during shutdown.

liewegas · 2017-05-05T13:28:13Z

src/mon/Elector.cc

-  expire_event = new C_ElectionExpire(this);
+  expire_event = new C_MonContext(mon, [this](int) {
+      expire();
+    });


omg so much better

liewegas · 2017-05-08T03:25:56Z

http://pulpito.ceph.com/sage-2017-05-06_05:54:44-rados-wip-sage-testing2---basic-smithi/

tchaikov added bug-fix mon labels May 3, 2017

tchaikov requested review from liewegas and jecluis May 3, 2017 02:48

tchaikov added the wip-kefu-testing label May 3, 2017

liewegas reviewed May 4, 2017

View reviewed changes

liewegas previously approved these changes May 4, 2017

View reviewed changes

tchaikov removed the wip-kefu-testing label May 4, 2017

tchaikov force-pushed the wip-19825 branch from f1cf3b4 to 7e209fb Compare May 4, 2017 14:17

tchaikov force-pushed the wip-19825 branch from 7e209fb to 5efd4bf Compare May 4, 2017 14:53

tchaikov changed the title ~~mon/Monitor: stop timer earlier in shutdown()~~ [DNM] mon/Monitor: stop timer earlier in shutdown() May 4, 2017

tchaikov force-pushed the wip-19825 branch from 5efd4bf to 5aa7582 Compare May 4, 2017 14:57

tchaikov changed the title ~~[DNM] mon/Monitor: stop timer earlier in shutdown()~~ mon/Monitor: stop timer earlier in shutdown() May 4, 2017

tchaikov requested a review from liewegas May 4, 2017 16:07

tchaikov changed the title ~~mon/Monitor: stop timer earlier in shutdown()~~ mon/Monitor: switch state to STATE_SHUTDOWN after stopping all services May 4, 2017

liewegas reviewed May 4, 2017

View reviewed changes

mon/Elector: call cancel_timer() in shutdown()

12139ae

instead of doing it manually. Signed-off-by: Kefu Chai <kchai@redhat.com>

tchaikov force-pushed the wip-19825 branch from 5aa7582 to 65bf37f Compare May 5, 2017 05:41

tchaikov changed the title ~~mon/Monitor: switch state to STATE_SHUTDOWN after stopping all services~~ mon: check is_shutdown() in timer callbacks May 5, 2017

mon: check is_shutdown() in timer callbacks

561cbde

introduce a helper class: C_MonContext, and initialize all timer events using it, to ensure that they do check is_shutdown() before doing their work. Fixes: http://tracker.ceph.com/issues/19825 Signed-off-by: Kefu Chai <kchai@redhat.com>

tchaikov force-pushed the wip-19825 branch from 65bf37f to 561cbde Compare May 5, 2017 05:55

tchaikov requested a review from liewegas May 5, 2017 05:57

liewegas reviewed May 5, 2017

View reviewed changes

liewegas approved these changes May 5, 2017

View reviewed changes

liewegas added needs-qa wip-sage2-testing labels May 5, 2017

liewegas merged commit 916c5e3 into ceph:master May 8, 2017

tchaikov deleted the wip-19825 branch May 8, 2017 04:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mon: check is_shutdown() in timer callbacks #14919

mon: check is_shutdown() in timer callbacks #14919

tchaikov commented May 3, 2017

tchaikov commented May 4, 2017

tchaikov commented May 4, 2017

liewegas commented May 4, 2017

liewegas May 4, 2017

tchaikov commented May 4, 2017 •

edited

tchaikov commented May 4, 2017

liewegas commented May 4, 2017 via email

tchaikov commented May 4, 2017 •

edited

tchaikov commented May 4, 2017

tchaikov commented May 4, 2017

liewegas commented May 4, 2017 via email

tchaikov commented May 4, 2017 •

edited

liewegas May 4, 2017

liewegas commented May 4, 2017

tchaikov commented May 5, 2017

liewegas May 5, 2017

liewegas May 5, 2017

liewegas commented May 8, 2017

		@@ -899,6 +899,10 @@ void Monitor::shutdown()

		wait_for_paxos_write();

		// stop timer earlier so the event callbacks won't panic seeing an inactive

		@@ -899,8 +899,6 @@ void Monitor::shutdown()

		wait_for_paxos_write();

		state = STATE_SHUTDOWN;

mon: check is_shutdown() in timer callbacks #14919

mon: check is_shutdown() in timer callbacks #14919

Conversation

tchaikov commented May 3, 2017

tchaikov commented May 4, 2017

tchaikov commented May 4, 2017

liewegas commented May 4, 2017

liewegas May 4, 2017

Choose a reason for hiding this comment

tchaikov commented May 4, 2017 • edited

tchaikov commented May 4, 2017

liewegas commented May 4, 2017 via email

tchaikov commented May 4, 2017 • edited

tchaikov commented May 4, 2017

tchaikov commented May 4, 2017

liewegas commented May 4, 2017 via email

tchaikov commented May 4, 2017 • edited

liewegas May 4, 2017

Choose a reason for hiding this comment

liewegas commented May 4, 2017

tchaikov commented May 5, 2017

liewegas May 5, 2017

Choose a reason for hiding this comment

liewegas May 5, 2017

Choose a reason for hiding this comment

liewegas commented May 8, 2017

tchaikov commented May 4, 2017 •

edited

tchaikov commented May 4, 2017 •

edited

tchaikov commented May 4, 2017 •

edited