qa/workunits/rbd: wait for demote status is propagated

[trociny]

(before promoting the image on another cluster)

Signed-off-by: Mykola Golub to.my.trociny@gmail.com

[trociny]

It is believed to fix sporadic test failures like this [1]:

2017-11-21T04:12:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi051.stderr:+ demote_image cluster1 mirror split-brain
2017-11-21T04:12:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi051.stderr:+ local cluster=cluster1
2017-11-21T04:12:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi051.stderr:+ local pool=mirror
2017-11-21T04:12:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi051.stderr:+ local image=split-brain
2017-11-21T04:12:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi051.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/split-brain
2017-11-21T04:12:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi051.stdout:Image demoted to non-primary
2017-11-21T04:12:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi051.stderr:+ promote_image cluster2 mirror split-brain
2017-11-21T04:12:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi051.stderr:+ local cluster=cluster2
2017-11-21T04:12:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi051.stderr:+ local pool=mirror
2017-11-21T04:12:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi051.stderr:+ local image=split-brain
2017-11-21T04:12:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi051.stderr:+ local force=
2017-11-21T04:12:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi051.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/split-brain
2017-11-21T04:12:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi051.stderr:rbd: error promoting image to primary2017-11-21 04:12:41.514 7fd432ffd700 -1 librbd::mirror::PromoteRequest: 0x55f4673e6720 handle_get_info: image is still primary within a remote cluster

[1] http://qa-proxy.ceph.com/teuthology/trociny-2017-11-20_20:35:23-rbd-wip-mgolub-testing-distro-basic-smithi/1870723/teuthology.log

[trociny]

@dillaman I was hurry. The change is useful for teuthology, but breaks the test when running locally. On teuthology we do not stop mirror daemons, while when running locally, cluster2 rbd-mirror is stopped before split-brain test. I created a fixup #19082