[BUG] Unable to add disk using Longhorn UI node page.

[khushboo-rancher]

Describe the bug
Unable to add additional disk using Longhorn UI node page for a node.

To Reproduce
Steps to reproduce the behavior:

1. Deploy Longhorn-rc1 on a k8s cluster of aws nodes.
2. Create an ebs volume in aws and attach it to a node.
3. Mount the additional disk on a directory.
4. Add the disk to a node using Longhorn UI and write some data into it.
5. Detach the ebs volume and try to add it to another node.
6. The disk appears as unschedulable and it is unable to detect the storage on the node.

Note:
If I add same disk to a setup with Longhorn v1.1.0, it is getting added successfully.

Expected behavior
The additional disk should be added

Log
Response of nodes api call:

"disk-1": {
				"allowScheduling": true,
				"conditions": {
					"Ready": {
						"lastProbeTime": "",
						"lastTransitionTime": "2021-04-13T20:09:38Z",
						"message": "Disk disk-1(/data/) on node ip- is not ready: disk has same file system ID 251796fd78924e70 as other disks [default-disk-251796fd78924e70 disk-1]",
						"reason": "DiskFilesystemChanged",
						"status": "False",
						"type": "Ready"
					},
					"Schedulable": {
						"lastProbeTime": "",
						"lastTransitionTime": "2021-04-13T20:09:38Z",
						"message": "the disk disk-1(/data/) on the node ip-1is not ready",
						"reason": "DiskNotReady",
						"status": "False",
						"type": "Schedulable"
					}
				},
				"diskUUID": "",
				"evictionRequested": false,
				"path": "/data/",
				"scheduledReplica": {},
				"storageAvailable": 0,
				"storageMaximum": 0,
				"storageReserved": 0,
				"storageScheduled": 0,
				"tags": null
			}
		}

Environment:

• Longhorn version: Longhorn -rc1
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl): kubectl
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: Rke - k8s v1.20.5

[khushboo-rancher]

Updated the steps to reproduce. This is not a direct disk addition case but when disk is moved from one node to another node.

[PhanLe1010]

@khushboo-rancher
I couldn't reproduce the problem following your provided reproduce steps. I was able to move the an EBS disk from node-1 to node-2 and move it back to node-1 without any issue. I also verified the data of replica on the EBS disk after each moving.

Regrading to error that you provided:

"message": "Disk disk-1(/data/) on node ip- is not ready: disk has same file system ID 251796fd78924e70 as other disks [default-disk-251796fd78924e70 disk-1]",

I was able to produce it by mounting the same physical disk to 2 different mount points data-1 and data-2. Then, create 2 new disks in Longhorn UI pointing to those mount points. In this case, the 2 active Longhorn disks have the same physical disk (same filesystem ID) so they fight each other. I think the behavior is ok since the users shouldn't add disks like this. Did you do the same as this? Or you took different steps?

[khushboo-rancher]

@PhanLe1010 The updated exact test steps are as below:

1. Created a cluster with aws instances 3 workers.
2. Deployed Longhorn.
3. Created some volumes.
4. Terminated one instance. That instance become down in Longhorn UI.
5. Moved the root ebs of the downed instance to a new instance and added the instance in the cluster.
6. Tried adding the moved disk for the new instance in the Longhorn UI.

note: Same behavior with v1.1.0

[khushboo-rancher]

This happens only if root disk is moved from one node to another with same template. In that case the filesystem ID are same. Lowering the severity for now.

[PhanLe1010]

This problem happens when users try to add a new disk that have the same filesystem ID as the existing disk on the node.

The VMs created by cloud provider (AWS, DigitalOcean, Linode) using the same template have the same filesystem ID for the root disk. Therefore, user cannot move root disk to different VM with the current implementation of Longhorn.

Since it is uncommon for user to move a root disk to another node as a data disk. We don’t see this is an immediate concern for now

[PhanLe1010]

Note:
we’re using filesystem ID to detect duplicate mounts of the same filesystem. But filesystem IDs can be duplicated in the same node, we might need to use some other mechanism e.g. major/minor number in the future