New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: in the future RBD images might become unaccessible in read-write mode #33013

Closed
rzarzynski opened this Issue Sep 19, 2016 · 7 comments

Comments

Projects
None yet
7 participants
@rzarzynski

The storage plugin for Ceph RBD employs advisory lock mechanism to provide basic fencing for RBD images. Lock management is performed with the CLI rbd command (rbd lock add, rbd lock remove and rbd lock list).

Unfortunately, advisory locks use the same infrastructure which is necessary to handle images with exclusive-lock flag set. That's the default for new RBD images in Jewel. Actually this isn't an issue as the kernel RBD driver (krbd) doesn't support the feature. However, when krbd finally get it (Ilya Dryomov is working on that), there might be a problem with mapping those images in the read-write mode.

The same, potential issue stopped similar fencing solution proposed to QEMU RBD driver.

@dillaman

This comment has been minimized.

Show comment
Hide comment
@dillaman

dillaman Sep 29, 2016

Contributor

What's the current use-case for the rbd advisory locking? Just wondering it would make sense to switch to the new Kraken+ managed lock API.

Contributor

dillaman commented Sep 29, 2016

What's the current use-case for the rbd advisory locking? Just wondering it would make sense to switch to the new Kraken+ managed lock API.

@smarterclayton

This comment has been minimized.

Show comment
Hide comment
@smarterclayton

smarterclayton Oct 6, 2016

Contributor

Somewhat covered under #34160

Contributor

smarterclayton commented Oct 6, 2016

Somewhat covered under #34160

@dillaman

This comment has been minimized.

Show comment
Hide comment
@dillaman

dillaman Oct 6, 2016

Contributor

Chatted with @rzarzynski on IRC the other day and was provided some background that this is relevant for krbd (which won't support exclusive locking until the 4.9+ kernel and a future RHEL 7.x kernel) and in the short/intermediate-term that feature shouldn't be enabled for kubernetes RBD devices.

Longer term, I have created Ceph feature request tickets for disabling automatic exclusive lock transitions for krbd [1] and rbd-nbd [2] to allow kubernetes to be in charge of which client should own the lock (while being able to re-use rbd exclusive lock's built-in blacklist lock recovery). I also created a ticket from our backlog item to add librbd support to tcmu-runner [3].

[1] http://tracker.ceph.com/issues/17524
[2] http://tracker.ceph.com/issues/17488
[3] http://tracker.ceph.com/issues/17489

Contributor

dillaman commented Oct 6, 2016

Chatted with @rzarzynski on IRC the other day and was provided some background that this is relevant for krbd (which won't support exclusive locking until the 4.9+ kernel and a future RHEL 7.x kernel) and in the short/intermediate-term that feature shouldn't be enabled for kubernetes RBD devices.

Longer term, I have created Ceph feature request tickets for disabling automatic exclusive lock transitions for krbd [1] and rbd-nbd [2] to allow kubernetes to be in charge of which client should own the lock (while being able to re-use rbd exclusive lock's built-in blacklist lock recovery). I also created a ticket from our backlog item to add librbd support to tcmu-runner [3].

[1] http://tracker.ceph.com/issues/17524
[2] http://tracker.ceph.com/issues/17488
[3] http://tracker.ceph.com/issues/17489

@xiangpengzhao

This comment has been minimized.

Show comment
Hide comment
@xiangpengzhao

xiangpengzhao Jun 23, 2017

Member

/sig storage

Member

xiangpengzhao commented Jun 23, 2017

/sig storage

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Dec 29, 2017

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Jan 28, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Feb 27, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment