librbd: managed lock refactoring #12922

trociny · 2017-01-13T14:41:35Z

No description provided.

trociny · 2017-01-13T14:44:22Z

@dillaman This is still WIP, I have some questions:

I had to add additional param to BreakRequest: blacklist_expire_seconds. In exclusive_lock implementation it was taken from ImageCtx, and now when BreakRequest does not use ImageCtx it should be provided separately.
Still there is issue with AcquireRequest: it has already used cct->_conf->rbd_break_on_blacklist and I have also introduces the usage of cct->_conf->rbd_blacklist_expire_seconds. This is wrong because if params are overriden in ImageCtx, they are ignored. So it looks we need to add these params to managed_lock::AcquireRequest too (and may be define exclusive_lock::AcquireRequest then)?
Althoug GetLockerRequest and BreakRequest do not use ImageCtx, they are still templates, so it is possible to instantiate mock classes. Does it look right for you?
When we implement shared lock mode, we will need to change GetLockerRequest to GetLockersRequest to return a list of lockers. May be do this right now?
I added helper methods to ManagedLock to get locker and break lock. Do you like this?

dillaman · 2017-01-13T14:49:51Z

@trociny

Makes sense
I would say they should be passed into ManagedLock as a constructor parameter and then passed to managed_lock::AcquireRequest as needed
Yes, we have other examples of things that don't use librbd::ImageCtx but are templated upon it just for mocking reasons.
I'd say wait for now -- cross that bridge under that shared locker PR
In general, yes -- however, I think in the case of librbd::lock_get_owners, it should work if I open the image as read-only. In that case, ImageCtx::exclusive_lock will be null. In librbd::lock_break, we should probably test for ImageCtx::read_only and return -EROFS before checking the exclusive lock state.

dillaman · 2017-01-13T15:19:08Z

src/librbd/managed_lock/GetLockerRequest.h

@@ -9,6 +9,10 @@

 class Context;

+namespace librados {
+  class IoCtx;


Nit: no indentation within namespaces. You could collapse to a single line, though.

dillaman · 2017-01-13T15:20:57Z

src/librbd/managed_lock/BreakRequest.h

-                              Context *on_finish) {
-    return new BreakRequest(image_ctx, locker, blacklist_locker,
-                            force_break_lock, on_finish);
+  static BreakRequest* create(librados::IoCtx& ioctx, ContextWQ *work_queue,


Nit: forward declare librados::IoCtx in this header as well?

dillaman · 2017-01-13T15:21:37Z

src/librbd/managed_lock/BreakRequest.h

@@ -13,6 +13,7 @@
 #include <boost/optional.hpp>

 class Context;
+class ContextWQ;

 namespace librbd {



Nit: can forward declare ImageCtx and move include to the CC file

dillaman · 2017-01-13T15:27:06Z

src/librbd/ManagedLock.h

@@ -8,6 +8,7 @@
 #include "include/Context.h"
 #include "include/rados/librados.hpp"
 #include "cls/lock/cls_lock_types.h"
+#include "librbd/managed_lock/Types.h"


Nit: forward declare managed_lock::Locker

dillaman · 2017-01-13T15:27:48Z

src/librbd/internal.cc

+    {
+      RWLock::RLocker l(ictx->owner_lock);
+
+      if (ictx->exclusive_lock == nullptr) {


We should support getting the lock owner when opened in read-only mode, which implies that the exclusive lock won't be initialized.

Then I suppose I can use ExclusiveLock(*ictx).get_locker(&locker, &get_owner_ctx) for this case, still looks a little better.

trociny · 2017-01-13T18:08:56Z

@dillaman

I would say they should be passed into ManagedLock as a constructor parameter and then passed to managed_lock::AcquireRequest as needed

Agree. There is a potential issue with this though. If a corresponding parameter in ImageCtx is updated by some reason, the exlusive_lock will still use the old value. Currently it looks like we don't update these params during ImageCtx life, this might be a surprise though when we want to do this in some future.

dillaman · 2017-01-13T19:37:54Z

@trociny There are a lot of parameters that, even if we eliminated the use of their ImageCtx replacement, would not function correctly w/ dynamic config updates. The blacklisting on lock break realistically should never be touched -- another example of way too many Ceph config options just to avoid hard-coding things.

trociny · 2017-01-15T14:49:04Z

@dillaman Updated.

Tested this locally and on subset of rbd suite (using --filter c_api_tests,python,fsx,cli,mirror and --filter-out bluestore,ec-data-pool) [1], there was only one failure for rbd_mirror_stress test [2], which does not look related.

[1] http://pulpito.ceph.com/trociny-2017-01-15_08:17:14-rbd-wip-mgolub-testing---basic-smithi/
http://pulpito.ceph.com/trociny-2017-01-15_07:25:31-rbd-wip-mgolub-testing---basic-smithi/
http://pulpito.ceph.com/trociny-2017-01-15_07:26:57-rbd-wip-mgolub-testing---basic-smithi/
http://pulpito.ceph.com/trociny-2017-01-15_07:28:14-rbd-wip-mgolub-testing---basic-smithi/
http://pulpito.ceph.com/trociny-2017-01-15_07:29:03-rbd-wip-mgolub-testing---basic-smithi/

[2] http://qa-proxy.ceph.com/teuthology/trociny-2017-01-15_08:17:14-rbd-wip-mgolub-testing---basic-smithi/719024/teuthology.log

trociny · 2017-01-15T21:48:08Z

updated: forgot to remove unused headers from internal.cc

trociny · 2017-01-16T15:58:52Z

src/librbd/managed_lock/BreakRequest.h

+  librados::IoCtx &m_ioctx;
+  CephContext *m_cct;
+  ContextWQ *m_work_queue;
+  std::string m_oid;
  const Locker &m_locker;


@dillaman Don't you think it would be safer to copy the locker in constructor? I just stepped on this in my LeaderWatcher, where I was calling break lock asynchronously and reset the locker not waiting for the request completed. No problem to change my code, but this was unexpected to me -- I had impression (may be wrong) we had a rule to copy input params like this.

I'd be fine if you changed it to support your use-case. W/ the original use-case, there was no race since the caller wouldn't be touching the memory while the call was in-progress.

Can I just update this PR to include this change?

trociny · 2017-01-16T16:25:31Z

@dillaman updated

dillaman

lgtm

dillaman · 2017-01-16T18:30:52Z

src/librbd/ManagedLock.h

-                             const std::string& oid, Watcher *watcher) {
-    return new ManagedLock(ioctx, work_queue, oid, watcher);
+                             const std::string& oid, Watcher *watcher,
+                             bool blacklist_on_break_lock = true,


Nit: can you remove the defaulted params here and below?

…kRequest Initially this was implemented for exclusive_lock (03533b9, 23f60fe) but was lost when merging managed_lock refactoring. Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

trociny · 2017-01-16T22:16:14Z

@dillaman updated

trociny added cleanup rbd labels Jan 13, 2017

trociny requested a review from dillaman January 13, 2017 14:41

dillaman reviewed Jan 13, 2017

View reviewed changes

dillaman mentioned this pull request Jan 13, 2017

librbd: Support for shared locking in ManagedLock #12886

Merged

trociny force-pushed the wip-managed_lock_refactoring branch from 2fba46f to 391e126 Compare January 14, 2017 21:39

trociny changed the title ~~[DNM] librbd: managed lock refactoring~~ librbd: managed lock refactoring Jan 15, 2017

trociny force-pushed the wip-managed_lock_refactoring branch from 391e126 to 8d53d23 Compare January 15, 2017 21:47

trociny commented Jan 16, 2017

View reviewed changes

trociny force-pushed the wip-managed_lock_refactoring branch from 8d53d23 to fd3e88e Compare January 16, 2017 16:24

dillaman added the wip-jason-testing label Jan 16, 2017

dillaman approved these changes Jan 16, 2017

View reviewed changes

dillaman reviewed Jan 16, 2017

View reviewed changes

Mykola Golub added 2 commits January 16, 2017 23:12

librbd: managed_lock: make AcquireRequest use GetLockRequest and Brea…

91b72f0

…kRequest Initially this was implemented for exclusive_lock (03533b9, 23f60fe) but was lost when merging managed_lock refactoring. Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: helper methods to query and break lock

f810839

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

trociny force-pushed the wip-managed_lock_refactoring branch from fd3e88e to f810839 Compare January 16, 2017 22:14

dillaman merged commit 06a4e14 into ceph:master Jan 17, 2017

trociny deleted the wip-managed_lock_refactoring branch February 11, 2017 15:41

tchaikov mentioned this pull request Sep 4, 2015

mon: fix the build with boost 1.59 #5804

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

librbd: managed lock refactoring #12922

librbd: managed lock refactoring #12922

trociny commented Jan 13, 2017

trociny commented Jan 13, 2017

dillaman commented Jan 13, 2017

dillaman Jan 13, 2017

dillaman Jan 13, 2017

dillaman Jan 13, 2017

dillaman Jan 13, 2017

dillaman Jan 13, 2017

trociny Jan 13, 2017

trociny commented Jan 13, 2017

dillaman commented Jan 13, 2017

trociny commented Jan 15, 2017

trociny commented Jan 15, 2017

trociny Jan 16, 2017

dillaman Jan 16, 2017

trociny Jan 16, 2017

dillaman Jan 16, 2017

trociny commented Jan 16, 2017

dillaman left a comment

dillaman Jan 16, 2017 •

edited

trociny commented Jan 16, 2017

librbd: managed lock refactoring #12922

librbd: managed lock refactoring #12922

Conversation

trociny commented Jan 13, 2017

trociny commented Jan 13, 2017

dillaman commented Jan 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trociny commented Jan 13, 2017

dillaman commented Jan 13, 2017

trociny commented Jan 15, 2017

trociny commented Jan 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trociny commented Jan 16, 2017

dillaman left a comment

Choose a reason for hiding this comment

dillaman Jan 16, 2017 • edited

Choose a reason for hiding this comment

trociny commented Jan 16, 2017

dillaman Jan 16, 2017 •

edited