devmapper + xfs - "Filesystem has duplicate UUID - can't mount"

[lobeck]

Description

I've tried to run 1.6 with devmapper and xfs on EKS, but it seems, that no containers can start up. Unfortunately, the node is terminated before I can login, so maybe the first container is starting but no subsequent ones.

XFS (dm-3): Filesystem has duplicate UUID 0686e31b-def5-45a1-a163-f22b1da6ec48 - can't mount

Steps to reproduce the issue

1. Starting a Kubernetes node with the latest AWS EKS AMI.
2. Updating containerd during startup:

    pushd /tmp
    curl -LO https://github.com/containerd/containerd/releases/download/v1.6.0/containerd-1.6.0-linux-amd64.tar.gz
    tar xvzf containerd-1.6.0-linux-amd64.tar.gz
    mv -f bin/containerd bin/containerd-shim-runc-v1 bin/containerd-shim-runc-v2 bin/containerd-shim /usr/bin/
    rm -rf bin containerd-1.6.0-linux-amd64.tar.gz
    popd

4. Wait for containers to start
5. The error appears

Describe the results you received and expected

Expected the node to startup and containers to run, but the node gets terminated, as it never reaches a healthy state

What version of containerd are you using?

containerd-1.6.0 (github release)

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

cat << EOF >> /etc/eks/containerd/containerd-config.toml
[plugins."io.containerd.snapshotter.v1.devmapper"]
    pool_name = "$${POOL_NAME}"
    root_path = "$${DATA_DIR}"
    base_image_size = "15GB"
    discard_blocks = true
    fs_type = "xfs"
EOF

[kzys]

This should be resolved by #6650.

[kzys]

Before closing, since this makes the XFS support basically useless, we may need to backport the fix to 1.6.x.

[estesp]

Marked #6650 to be cherry-picked to release/1.6

[mxpv]

I think this can now be closed. Feel free to reopen if there are still issues.