[RBD] parallel PVC creation on a newly created block pool will hang #2521

Rakshith-R · 2021-09-20T10:03:27Z

Describe the bug

The following bug in librbd causes parallel pvc creation request on a newly created block
pool to hang.

Concurrent rbd_pool_init() or rbd_create() operations on an unvalidated
(uninitialized) pool trigger a lockup in ValidatePoolRequest state
machine caused by blocking selfmanaged_snap_{create,remove}() calls.

Ceph issue tracker : https://tracker.ceph.com/issues/52537

Ceph pacific backport pr with fix : ceph/ceph#43113

Environment details

Image/version of Ceph CSI driver : v3.4.0

Steps to reproduce

Create new rbd block pool(with no images) + StorageClass against the CSI provisioner.
Create Multiple PVCs in parallel
Creation request will stay in pending state indefinitely

Actual results

Creation request will stay in pending state indefinitely

Expected behavior

Creation request should succeed

Updated Work Around (does not leave stale imap entries, thanks @Madhu-1 )

execute rbd pool init <pool_name> directly on cluster or from csi pods.
Restart csi rbdplugin provisioner pod
and PVCs will go to bound state without leaving any stale resources.

After the above steps, parallel PVC creation requests should work fine.

Work Around (Not recommended, will leave stale omap entries)

Delete ongoing PVC creation requests.
Restart csi rbdplugin provisioner pod
Either
- Issue a single PVC create request which will succeed.
- or call rbd pool init <pool_name> on the ceph cluster.

After the above steps, parallel PVC creation requests should work fine.

The text was updated successfully, but these errors were encountered:

Rakshith-R · 2021-09-21T02:10:44Z

Hey @idryomov, a lot of users seem to be hitting this issue, would you recommend adding rbd_pool_validate = false (which does prevent this problem) in ceph.conf to workaround the issue in cephcsi ?

Will there be any side affects or reason not to do this ?

Thanks !

idryomov · 2021-09-21T12:15:56Z

How many reports of this exact issue did you get so far? I see you linked #2520 but it doesn't look related at first sight.

I could be wrong but it feels like the combination of a fresh uninitialized pool and concurrent creation requests should make this rather unlikely to hit for a general user. I'm inclined to just visibly document this as a known issue and wait for the 16.2.7 release.

Rakshith-R · 2021-09-21T12:24:25Z

How many reports of this exact issue did you get so far? I see you linked #2520 but it doesn't look related at first sight.

Quite a few are reported here.

I could be wrong but it feels like the combination of a fresh uninitialized pool and concurrent creation requests should make this rather unlikely to hit for a general user.

@idryomov
For a user using rook (>=v1.7.1), hitting this can be very easy.
Just create a BlockPool and issue more than one RBD PVC request.

I'm inclined to just visibly document this as a known issue and wait for the 16.2.7 release.

I suspect, we will get more such reports and if I am not wrong ceph: v16.2.7 will be out in two months?

Thanks!

idryomov · 2021-09-21T16:16:01Z

Quite a few are reported here.

I see two reports there.

For a user using rook (>=v1.7.1), hitting this can be very easy.
Just create a BlockPool and issue more than one RBD PVC request.

This is not the first rook release based on pacific, right? The bug is present in all pacific releases (16.2.0 - 16.2.6), if we only got two reports in this time frame I'm not sure it is worth rushing in a workaround instead of just waiting for 16.2.7.

We don't have a set date for 16.2.7 so yes it could take two months but we could also expedite it.

Rakshith-R · 2021-09-22T06:01:38Z

For a user using rook (>=v1.7.1), hitting this can be very easy.
Just create a BlockPool and issue more than one RBD PVC request.

This is not the first rook release based on pacific, right? The bug is present in all pacific releases (16.2.0 - 16.2.6), if we only got two reports in this time frame I'm not sure it is worth rushing in a workaround instead of just waiting for 16.2.7.

This is the first CephCSI release v3.4.0 based on pacific 8fafc42, picked up by rook in v1.7.1.

We don't have a set date for 16.2.7 so yes it could take two months but we could also expedite it.

okay 👍 .
Thanks ilya.

@ceph/ceph-csi-contributors I think we can go ahead with cephcsi v3.4.1 release and pick up the fix in v3.4.2 when its out.

Madhu-1 · 2021-09-22T06:13:16Z

@idryomov couple of questions

We will hit this one only on the new pools or on the existing pool also

Concurrent rbd_pool_init() or rbd_create() operations on an unvalidated (uninitialized) pool trigger a lockup in ValidatePoolRequest

Is there any command to check pool is uninitialized/unvalidated?
The issue exists on both the ceph cluster version (server) and also in the client library (librbd)?

@Rakshith-R below is the workaround you mentioned in the issue.

Delete ongoing PVC creation requests.
Restart csi rbdplugin provisioner pod
Either
Issue a single PVC create request which will succeed.
or call rbd pool init <pool_name> on the ceph cluster.

cant we just scale down the rbd provisioner pod and call rbd pool init on the ceph cluster and scale up the provisioner pod won't this workaround works?

Rakshith-R · 2021-09-22T06:20:51Z

We will hit this one only on the new pools or on the existing pool also

only new pools as per my tests.

@Rakshith-R below is the workaround you mentioned in the issue.

Delete ongoing PVC creation requests.
Restart csi rbdplugin provisioner pod
Either
Issue a single PVC create request which will succeed.
or call rbd pool init <pool_name> on the ceph cluster.

cant we just scale down the rbd provisioner pod and call rbd pool init on the ceph cluster and scale up the provisioner pod won't this workaround works?

yes, that works too but need direct access to ceph cluster to execute the commands and will still leave stale omap entries.

Madhu-1 · 2021-09-22T06:26:40Z

We will hit this one only on the new pools or on the existing pool also

only new pools as per my tests.

@Rakshith-R below is the workaround you mentioned in the issue.

Delete ongoing PVC creation requests.
Restart csi rbdplugin provisioner pod
Either
Issue a single PVC create request which will succeed.
or call rbd pool init <pool_name> on the ceph cluster.

cant we just scale down the rbd provisioner pod and call rbd pool init on the ceph cluster and scale up the provisioner pod won't this workaround works?

yes, that works too but need direct access to ceph cluster to execute the commands and will still leave stale omap entries.

executing a command on the ceph cluster is not an issue and it's better than leaving the stale resources. if the pending PVC is not deleted how this will leave a stale omap?

Rakshith-R · 2021-09-22T06:52:11Z

executing a command on the ceph cluster is not an issue and it's better than leaving the stale resources. if the pending PVC is not deleted how this will leave a stale omap?

okay, tested it. It will not leave any stale resources.

execute rbd pool init <pool_name> directly on cluster or from csi pods.
delete provisioner pods

and PVCs will go to bound state without leaving any stale resources.

idryomov · 2021-09-24T12:47:23Z

@idryomov couple of questions

* We will hit this one only on the new pools or on the existing pool also

Only on the new (empty) and uninitialized pools. It can't occur once even a single image is created or the pool is initialized.

Concurrent rbd_pool_init() or rbd_create() operations on an unvalidated (uninitialized) pool trigger a lockup in ValidatePoolRequest
* Is there any command to check pool is uninitialized/unvalidated?

I don't think so. One possibility is to check if the pool's application is "rbd" but that is a bit wonky -- as part of initialization we set it to "rbd" but it is possible to set the application separately.

OTOH initialization requests are more or less idempotent, so issuing them repeatedly should be safe.

* The issue exists on both the ceph cluster version (server) and also in the client library  (librbd)?

This is a librbd issue.

Madhu-1 · 2021-09-24T13:06:37Z

This is a librbd issue.

@idryomov Thanks for your reply. till pacific 16.2.7 is released is it good to switch back to octopus release as the issue is in librbd and doesn't exist in octopus? or as suggested earlier documenting it would be good enough?

idryomov · 2021-09-24T13:18:13Z

Technically the issue exists in octopus librbd too but it is much harder to hit there and rolling back to octopus should fix it.
Up to you -- I still feel that it shouldn't affect that many users and just documenting would be good enough but I could be underestimating it.

Madhu-1 · 2021-09-28T04:55:12Z

Technically the issue exists in octopus librbd too but it is much harder to hit there and rolling back to octopus should fix it.
Up to you -- I still feel that it shouldn't affect that many users and just documenting would be good enough but I could be underestimating it.

Thanks, @idryomov for your opinion. Much appreciated!. Looks like whoever is trying cephcsi with new pools might hit this issue.

Most of the things are automated like pool creation, PVC creation, etc. As its much harder to hit this in octopus, Instead of documenting am more inclined towards using octopus as the base image for 3.4.1 and switch to the new 16.2.7 in cephcsi 3.4.2

@humblec @nixpanic thoughts?

Rakshith-R · 2021-09-28T05:09:38Z

Most of the things are automated like pool creation, PVC creation, etc. As its much harder to hit this in octopus, Instead of documenting am more inclined towards using octopus as the base image for 3.4.1 and switch to the new 16.2.7 in cephcsi 3.4.2

We cannot use ceph octopus as base image since deep_copy() which is required for thick-provisioning does not work as expected in ceph octopus.
refer : #2202
#2187 (comment)

Madhu-1 · 2021-09-28T05:10:41Z

Most of the things are automated like pool creation, PVC creation, etc. As its much harder to hit this in octopus, Instead of documenting am more inclined towards using octopus as the base image for 3.4.1 and switch to the new 16.2.7 in cephcsi 3.4.2

We cannot use ceph octopus as base image since deep_copy() which is required for thick-provisioning does not work as expected in ceph octopus.
refer : #2202
#2187 (comment)

:D looks like we have another problem, Do we have any other option left except documentation to make the life easier?

github-actions · 2021-10-28T21:06:01Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

github-actions · 2021-11-04T21:06:08Z

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

Madhu-1 · 2022-01-19T13:42:03Z

@Rakshith-R this can be closed?

Rakshith-R · 2022-01-19T13:45:10Z

@Rakshith-R this can be closed?
Thanks madhu for reminding.

Yes, ceph 16.2.7 was released in December https://github.com/ceph/ceph/releases/tag/v16.2.7, it contains the fix and in turn cephcsi v3.5.0 will have the fix for this issue.

Closing the issue for the above reason.

Rakshith-R · 2022-01-21T06:05:17Z

@Rakshith-R this can be closed?
Thanks madhu for reminding.

Yes, ceph 16.2.7 was released in December https://github.com/ceph/ceph/releases/tag/v16.2.7, it contains the fix and in turn cephcsi v3.5.0 will have the fix for this issue.

Closing the issue for the above reason.

dockerhub did not have the latest version of ceph, base images are now being pulled from quay.io
And release v3.5.1 should have the fix for this issue.
refer #2796

Madhu-1 · 2022-01-21T06:08:48Z

@Rakshith-R this can be closed?
Thanks madhu for reminding.

Yes, ceph 16.2.7 was released in December https://github.com/ceph/ceph/releases/tag/v16.2.7, it contains the fix and in turn cephcsi v3.5.0 will have the fix for this issue.
Closing the issue for the above reason.

dockerhub did not have the latest version of ceph, base images are now being pulled from quay.io And release v3.5.1 should have the fix for this issue. refer #2796

Thanks for notifying on this one

Madhu-1 · 2022-01-21T14:23:44Z

closed by #2798

Seems like we're hitting parallel PVC creation bug in kubevirt/kubevirt: kubevirt/kubevirt#7783 ceph/ceph-csi#2521 Reproduced locally: ```bash [root@hera04 kubevirt]# cat ~/manifests/2pvcs.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ceph-block-pvc-1 spec: volumeMode: Block accessModes: - ReadWriteMany resources: requests: storage: 3Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ceph-block-pvc-2 spec: volumeMode: Block accessModes: - ReadWriteMany resources: requests: storage: 3Gi [root@hera04 kubevirt]# ./cluster-up/kubectl.sh create -f ~/manifests/2pvcs.yaml selecting docker as container runtime persistentvolumeclaim/ceph-block-pvc-1 created persistentvolumeclaim/ceph-block-pvc-2 created [root@hera04 kubevirt]# ./cluster-up/kubectl.sh get pvc selecting docker as container runtime NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ceph-block-pvc-1 Pending rook-ceph-block 2s ceph-block-pvc-2 Pending rook-ceph-block 2s [root@hera04 kubevirt]# ./cluster-up/kubectl.sh get pvc selecting docker as container runtime NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ceph-block-pvc-1 Pending rook-ceph-block 3s ceph-block-pvc-2 Pending rook-ceph-block 3s [root@hera04 kubevirt]# ./cluster-up/kubectl.sh create -f ~/manifests/pvc.yaml selecting docker as container runtime persistentvolumeclaim/ceph-block-pvc created [root@hera04 kubevirt]# ./cluster-up/kubectl.sh get pvc selecting docker as container runtime NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ceph-block-pvc Pending rook-ceph-block 2s ceph-block-pvc-1 Pending rook-ceph-block 66s ceph-block-pvc-2 Pending rook-ceph-block 66s ``` Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

Rakshith-R added component/rbd Issues related to RBD Priority-0 highest priority issue bug Something isn't working dependency/ceph depends on core Ceph functionality labels Sep 20, 2021

Rakshith-R mentioned this issue Sep 21, 2021

When I use the CSI RBD plug-in, the newly created PVC is always in the pending state; It also indicates that PVC already exists, which is very strange #2520

Closed

This was referenced Sep 27, 2021

ceph: add workaround for Parallel RBD PVC Creation hangs on new pools rook/rook#8770

Closed

New BlockPool / SC + Parallel RBD Volume Creation hangs and fails rook/rook#8696

Closed

github-actions bot added the wontfix This will not be worked on label Oct 28, 2021

github-actions bot closed this as completed Nov 4, 2021

Rakshith-R reopened this Nov 5, 2021

Rakshith-R added keepalive This label can be used to disable stale bot activiity in the repo and removed wontfix This will not be worked on labels Nov 5, 2021

Rakshith-R closed this as completed Jan 19, 2022

Madhu-1 reopened this Jan 21, 2022

Madhu-1 mentioned this issue Jan 21, 2022

Release 3.5.1 plan #2803

Closed

3 tasks

Madhu-1 closed this as completed Jan 21, 2022

akalenyu mentioned this issue May 31, 2022

Bump rook-ceph 1.7.1 -> 1.8.9 kubevirt/kubevirtci#803

Merged

akalenyu mentioned this issue Jun 29, 2022

Sync ceph bumps to 1.24 as well kubevirt/kubevirtci#823

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RBD] parallel PVC creation on a newly created block pool will hang #2521

[RBD] parallel PVC creation on a newly created block pool will hang #2521

Rakshith-R commented Sep 20, 2021 •

edited

Loading

Rakshith-R commented Sep 21, 2021

idryomov commented Sep 21, 2021 •

edited

Loading

Rakshith-R commented Sep 21, 2021

idryomov commented Sep 21, 2021

Rakshith-R commented Sep 22, 2021

Madhu-1 commented Sep 22, 2021 •

edited

Loading

Rakshith-R commented Sep 22, 2021

Madhu-1 commented Sep 22, 2021

Rakshith-R commented Sep 22, 2021 •

edited

Loading

idryomov commented Sep 24, 2021

Madhu-1 commented Sep 24, 2021

idryomov commented Sep 24, 2021

Madhu-1 commented Sep 28, 2021

Rakshith-R commented Sep 28, 2021

Madhu-1 commented Sep 28, 2021

github-actions bot commented Oct 28, 2021

github-actions bot commented Nov 4, 2021

Madhu-1 commented Jan 19, 2022

Rakshith-R commented Jan 19, 2022

Rakshith-R commented Jan 21, 2022

Madhu-1 commented Jan 21, 2022

Madhu-1 commented Jan 21, 2022

[RBD] parallel PVC creation on a newly created block pool will hang #2521

[RBD] parallel PVC creation on a newly created block pool will hang #2521

Comments

Rakshith-R commented Sep 20, 2021 • edited Loading

Describe the bug

Environment details

Steps to reproduce

Actual results

Expected behavior

Updated Work Around (does not leave stale imap entries, thanks @Madhu-1 )

Work Around (Not recommended, will leave stale omap entries)

Rakshith-R commented Sep 21, 2021

idryomov commented Sep 21, 2021 • edited Loading

Rakshith-R commented Sep 21, 2021

idryomov commented Sep 21, 2021

Rakshith-R commented Sep 22, 2021

Madhu-1 commented Sep 22, 2021 • edited Loading

Rakshith-R commented Sep 22, 2021

Madhu-1 commented Sep 22, 2021

Rakshith-R commented Sep 22, 2021 • edited Loading

idryomov commented Sep 24, 2021

Madhu-1 commented Sep 24, 2021

idryomov commented Sep 24, 2021

Madhu-1 commented Sep 28, 2021

Rakshith-R commented Sep 28, 2021

Madhu-1 commented Sep 28, 2021

github-actions bot commented Oct 28, 2021

github-actions bot commented Nov 4, 2021

Madhu-1 commented Jan 19, 2022

Rakshith-R commented Jan 19, 2022

Rakshith-R commented Jan 21, 2022

Madhu-1 commented Jan 21, 2022

Madhu-1 commented Jan 21, 2022

Rakshith-R commented Sep 20, 2021 •

edited

Loading

idryomov commented Sep 21, 2021 •

edited

Loading

Madhu-1 commented Sep 22, 2021 •

edited

Loading

Rakshith-R commented Sep 22, 2021 •

edited

Loading