Problems mounting XFS volume clones / restored snapshots

[jsafrane]

XFS does not allow to mount two volumes that have the same UUID on the same machine. The second mount fails with:

[44557.612032] XFS (vde): Filesystem has duplicate UUID fadf19ab-bbcc-4f40-8d4f-44550e822db1 - can't mount

This is problematic when using a cloned volume or a restored snapshot - the original volume and the new volume cannot be mounted on the same compute node.

On what level should be the issue solved?

• CSI spec could mention that it's CSI plugin problem to make sure cloned / restored volumes are usable on the same node as the original vol (e.g. by using -o nouuid mount opt. for xfs volumes or using xfs_admin -U genereate to re-generate UUID on the first mount after volume restore / clone).

• CSI spec could mention that it's CO problem to pass e.g. -o nouuid mount option to all XFS NodeStage/NodePublish calls.

In both cases, someone must check that XFS is used and know that it needs a special handling.

[jdef]

This sounds like a plug-in issue, not a CO issue.

…

On Wed, May 26, 2021, 6:24 AM Jan Šafránek ***@***.***> wrote: XFS does not allow to mount two volumes that have the same UUID on the same machine. The second mount fails with: [44557.612032] XFS (vde): Filesystem has duplicate UUID fadf19ab-bbcc-4f40-8d4f-44550e822db1 - can't mount This is problematic when using a cloned volume or a restored snapshot - the original volume and the new volume cannot be mounted on the same compute node. On what level should be the issue solved? - CSI spec could mention that it's CSI plugin problem to make sure cloned / restored volumes are usable on the same node as the original vol (e.g. by using -o nouuid mount opt. for xfs volumes or using xfs_admin -U genereate to re-generate UUID on the first mount after volume restore / clone). - CSI spec could mention that it's CO problem to pass e.g. -o nouuid mount option to all XFS NodeStage/NodePublish calls. In both cases, someone must check that XFS is used and know that it needs a special handling. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#482>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR5KLGBUMHJBN4LGK4USBTTPTD4FANCNFSM45RWFRPQ> .

[saad-ali]

Agreed with @jdef. We discussed this at the community meeting today, see Notes

Conclusion:

• For this bug:
  1. Pick a random CSI driver to fix this issue in the driver
  2. Write an E2E test to catch this issues
  3. Make the E2E test opt-in and send messages to CSI announce to encourage other CSI drivers to fix this issue.
  4. At some point in the future make the test required.
• For this class of bug:
  • Must do the same as above for each similar bug we discover.

[humblec]

Ceph CSI was hit on this issue at clone time , ceph/ceph-csi#966 (comment) , nouuid or regenerating UUID (- depends on xfsprogs version..etc) could be the possible fixes.

[jsafrane]

i. Pick a random CSI driver to fix this issue in the driver
ii. Write an E2E test to catch this issues

I'm experimenting with a Kubernetes e2e test and AWS EBS CSI driver fix in kubernetes-sigs/aws-ebs-csi-driver#913

[jsafrane]

Turned into real e2e test in Kubernetes: kubernetes/kubernetes#102538