rgw: use explicit flag to cancel RGWCoroutinesManager::run()

[cbodley]

RGWCoroutinesManager::run() was setting ret = -ECANCELED to break out of
the loop when it sees going_down. coroutines that failed with -ECANCELED
were confusing this logic and leading to coroutine deadlock assertions
below. when we hit the going_down case, set a 'canceled' flag, and check
that flag when deciding whether to break out of the loop

Fixes: http://tracker.ceph.com/issues/17465

[cbodley]

reproduced the deadlock with test_multi.py by injecting ECANCELED errors:

diff --git a/src/rgw/rgw_sync.cc b/src/rgw/rgw_sync.cc
index c44305e..c9761ac 100644
--- a/src/rgw/rgw_sync.cc
+++ b/src/rgw/rgw_sync.cc
@@ -1159,7 +1159,12 @@ int RGWMetaSyncSingleEntryCR::operate() {
     retcode = 0;
     for (tries = 0; tries < NUM_TRANSIENT_ERROR_RETRIES; tries++) {
       if (sync_status != -ENOENT) {
+        retcode = -ECANCELED;
-          yield call(new RGWMetaStoreEntryCR(sync_env, raw_key, md_bl));
       } else {
           yield call(new RGWMetaRemoveEntryCR(sync_env, raw_key));
       }

with this PR applied, the deadlock is avoided. will schedule for teuthology

[yehudasa]

lgtm

[cbodley]

http://pulpito.ceph.com/cbodley-2016-12-13_10:08:00-rgw-wip-rgw-fix-cr-deadlock---basic-mira/ completed with only valgrind failures and selinux denials