rbd-nbd: fix kernel deadlock during teuthology testing #10985
Conversation
rbd-nbd: fix kernel deadlock during teuthology testing #10985
| @@ -614,6 +614,8 @@ static int do_map() | |||
|
|
|||
| server.start(); | |||
| ioctl(nbd, NBD_DO_IT); | |||
| ioctl(nbd, NBD_CLEAR_QUE); | |||
There was a problem hiding this comment.
@dillaman NBD_CLEAR_QUE looks like deprecated and nop.
There was a problem hiding this comment.
Yeah, I just put these in here to match qemu-nbd's implementation.
|
@dillaman Still observing hangs running this on teuthology: http://pulpito.ceph.com/trociny-2016-09-07_07:30:28-rbd-wip-mgolub-testing---basic-mira/ Also, there were failures: http://pulpito.ceph.com/trociny-2016-09-07_07:31:00-rbd-wip-mgolub-testing---basic-vps/ |
|
@trociny Definitely a different issue: [ 7241.278129] block nbd0: Other side returned error (22) This implies that rbd-nbd didn't receive an update notification:
The last recorded fsx resize should have made the image 118,824,448 bytes:
The backtrace from the fsx process shows its in the middle of a write-induced resize, so there must be a race condition between resizing the image and a synchronous read/write operation noticing the change. |
|
The resize is hanging because rbd-nbd (the exclusive lock owner) isn't responding -- and it isn't responding most likely because its librbd thread is deadlocked. It looks like you have gdb attached to it so I cannot see what its doing. |
|
Sorry, yes I tried attaching with gdb but it hanged and letft the terminal opened. I have kiled my session. |
|
@trociny Ok -- same thing is happening to me. I am thoroughly convinced that the nbd block driver has lots of edge conditions. Looking at the deadlock listed on mira023 (which is running a 4.8-rc5 kernel instead of stock), it is showing a deadlock in the nbd block driver shut down logic and looking at the code it is definitely a problem and I don't immediately see a way around it. |
|
@trociny It appears to be stable now -- I updated the tests to not colocate nbd with the OSDs and I fixed the resize failure. |
|
Kicked off rbd suite run: http://pulpito.ceph.com/jdillaman-2016-09-08_21:48:20-rbd-wip-16921---basic-smithi/ |
rbd-nbd: fix kernel deadlock during teuthology testing #10985
| @@ -121,6 +121,9 @@ class ResizeRequest : public Request<ImageCtxT> { | |||
| Context *send_append_op_event(); | |||
| Context *handle_append_op_event(int *result); | |||
|
|
|||
| void send_flush_cache(); | |||
| Context *handle_flush_cache(int *result); | |||
There was a problem hiding this comment.
@dillaman Update the diagram?
Fixes: http://tracker.ceph.com/issues/16921 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Any potential writeback outside the extents of a shrunk image would result in orphaned objects. Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
|
@trociny Pushed updates -- the unrelated blacklisting test failure is fixed by PR #11034 and the dynamic_features failure is covered by open ticket http://tracker.ceph.com/issues/17227 |
|
lgtm |
| @@ -727,6 +729,7 @@ static int rbd_nbd(int argc, const char *argv[]) | |||
| env_to_vec(args); | |||
| global_init(NULL, args, CEPH_ENTITY_TYPE_CLIENT, CODE_ENVIRONMENT_DAEMON, | |||
| CINIT_FLAG_UNPRIVILEGED_DAEMON_DEFAULTS); | |||
| g_ceph_context->_conf->set_val_or_die("pid_file", ""); | |||
There was a problem hiding this comment.
Hi Jason (and all reading this), I'm working on starting rbd-nbd with systemd. And it is best to let rbd-nbd write a pid file so that systemd can reliably determine the main process. But this config is disabled here. See this comment for more. Do you have any suggestions? thanks.
There was a problem hiding this comment.
Why not go directly and add notify/watchdog support? That would make more sense I think.
There was a problem hiding this comment.
I don't have a strong opinion here. I remember I was not very happy when it was added and we had some discussion. On the other hand for usual cases when the rbd-nbd processes are controlled by rbd map/unmap, which does not need the pid files, it is good to have the pid file disabled by default, to avoid pid file collisions.
So I think a solution could be to set the pidfile only if it is explicitly specified via command line arguments. Though I don't think it is a good place to discuss this issue. I suggest to open a tracker ticket, describing the issue, and the provide a PR with the proposed solution, and we may discuss it there.
Fixes: http://tracker.ceph.com/issues/16921
Signed-off-by: Jason Dillaman dillaman@redhat.com