clone creation is failing on CentOS when fstype is xfs

[w3aman]

zfspv-clone getting failed on centos based cluster when fstype is used as xfs
getting such error:

time="2020-07-15T09:10:51Z" level=error msg="zfspv: failed to mount volume /dev/zvol/zfs-test-pool/pvc-05e22f2a-9901-40ba-817d-82a43a922713 [xfs] to /var/lib/kubelet/pods/341b2fca-8430-45d1-9009-375a81588210/volumes/kubernetes.io~csi/pvc-05e22f2a-9901-40ba-817d-82a43a922713/mount, error mount failed: exit status 32\nMounting command: mount\nMounting arguments: -t xfs -o defaults /dev/zvol/zfs-test-pool/pvc-05e22f2a-9901-40ba-817d-82a43a922713 /var/lib/kubelet/pods/341b2fca-8430-45d1-9009-375a81588210/volumes/kubernetes.io~csi/pvc-05e22f2a-9901-40ba-817d-82a43a922713/mount\nOutput: mount: /var/lib/kubelet/pods/341b2fca-8430-45d1-9009-375a81588210/volumes/kubernetes.io~csi/pvc-05e22f2a-9901-40ba-817d-82a43a922713/mount: wrong fs type, bad option, bad superblock on /dev/zd16, missing codepage or helper program, or other error.\n\n"

and from the dmesg log it says :

[ 6421.004650] XFS (zd16): Filesystem has duplicate UUID 3af6518b-8f14-4585-b513-949053e63a4c - can't mount
[ 6421.603229] XFS (zd16): Filesystem has duplicate UUID 3af6518b-8f14-4585-b513-949053e63a4c - can't mount
[ 6422.705286] XFS (zd16): Filesystem has duplicate UUID 3af6518b-8f14-4585-b513-949053e63a4c - can't mount
[ 6424.812780] XFS (zd16): Filesystem has duplicate UUID 3af6518b-8f14-4585-b513-949053e63a4c - can't mount
[ 6428.937560] XFS (zd16): Filesystem has duplicate UUID 3af6518b-8f14-4585-b513-949053e63a4c - can't mount
[ 6437.072385] XFS (zd16): Filesystem has duplicate UUID 3af6518b-8f14-4585-b513-949053e63a4c - can't mount
[ 6453.143135] XFS (zd16): Filesystem has duplicate UUID 3af6518b-8f14-4585-b513-949053e63a4c - can't mount
[ 6485.287202] XFS (zd16): Filesystem has duplicate UUID 3af6518b-8f14-4585-b513-949053e63a4c - can't mount

zfs version: 0.8.0
konvoy cluster on centos 7

[pawanpraka1]

This issue is specific to xfs only, when we create a clone volume and system is taking time in creating the device.

When we create a clone volume from a xfs filesystem, ZFS-LocalPV will go ahead and generate a new UUID for the clone volumes as we need a new UUID to mount the new clone filesystem. To generate a new UUID for the clone volume, ZFS-LocalPV first replays the xfs log by mounting the device to a tmp localtion.

Here, what is happening is since device creation is slow, so we went ahead and created the tmp location to mount the clone volume but since device has not created yet, the mount failed. In the next try since the tmp location is present, it will keep failing there only at every reconciliation time.

[pawanpraka1]

Below log shows the mount failure because the device is not present :

time="2020-07-15T10:02:25Z" level=info msg="created clone zfs-test-pool/pvc-cc253e5f-dd7c-4bd8-abc7-26ff73848323"
time="2020-07-15T10:02:25Z" level=error msg="xfs: failed to mount volume /dev/zvol/zfs-test-pool/pvc-cc253e5f-dd7c-4bd8-abc7-26ff73848323=>/tmp/pvc-cc253e5f-dd7c-4bd8-abc7-26ff73848323 error: mount: /tmp/pvc-cc253e5f-dd7c-4bd8-abc7-26ff73848323: special device /dev/zvol/zfs-test-pool/pvc-cc253e5f-dd7c-4bd8-abc7-26ff73848323 does not exist.\n"

In retry it is failing to create the tmp directory

time="2020-07-15T10:02:25Z" level=error msg="xfs: failed to create tmpdir /tmp/pvc-cc253e5f-dd7c-4bd8-abc7-26ff73848323 error: mkdir /tmp/pvc-cc253e5f-dd7c-4bd8-abc7-26ff73848323: file exists"