os/bluestore/KernelDevice: use flock(2) for block device lock #26245
Conversation
The fcntl locks fail due to the classic posix lock gotcha: if you close *any* fd to the same inode from the process, the lock(s) go away. Use flock(2) instead. We have to be careful because we open the main bluestore device via two KernelDevice instances: one for bluestore and one for bluefs. Add a no-lock flag so that the bluefs instance does not try to lock and does not conflict with bluestore's. Fixes: http://tracker.ceph.com/issues/38150 Signed-off-by: Sage Weil <sage@redhat.com>
| discard_cb[id], static_cast<void*>(this)); | ||
| if (shared_with_bluestore) { | ||
| b->set_no_exclusive_lock(); | ||
| } |
There was a problem hiding this comment.
A little convoluted...if shared_with_bluestore is true then lock_exclusive becomes false, but this does look like it'll do the right thing.
That's correct -- you'll get a new inode in that case that just happens to be hooked up to the same device as the first one. I don't see a way to do meaningful locking in that situation. Maybe we should fix that in the kernel? |
I suspect the use of Perhaps there needs to be an ioctl that locks the backing bdev? |
|
Yeah, that could work. Open the bdev, use the ioctl to switch the fd over the bd_inode's locking context rather than the one for the device, and then set locks on it instead of the inode attached to the filp. We could also consider just doing this universally without the ioctl, but we'd need a pretty long testing cycle to see if anything would break, given that this is a subtle behavior change. |
Starting with Nautilus Ceph implements flock in the ceph-osd daemon directly. See: https://tracker.ceph.com/issues/38150, ceph/ceph#26245. There are backporting tickets for Mimic and Luminous, but they haven't been acted on since February 2019.
|
This may have introduced a regression, see https://tracker.ceph.com/issues/46124 Also, OFD locks might be even better, see https://tracker.ceph.com/issues/46124#note-3 |
The fcntl locks fail due to the classic posix lock gotcha: if you close
any fd to the same inode from the process, the lock(s) go away.
Use flock(2) instead. We have to be careful because we open the main
bluestore device via two KernelDevice instances: one for bluestore and
one for bluefs. Add a no-lock flag so that the bluefs instance does not
try to lock and does not conflict with bluestore's.
Fixes: http://tracker.ceph.com/issues/38150
Signed-off-by: Sage Weil sage@redhat.com