[BUG] For failed backing image on the disks, the state shows as Healthy on the Longhorn-UI

[khushboo-rancher]

Describe the bug
Create a backing image with dummy image and create a volume using it, the volume stuck is Attaching state and the state of the backing image still shows as Healthy

To Reproduce
Steps to reproduce the behavior:

1. Deploy Longhorn - master on a k8s cluster
2. Create a backing image with dummy URL.
3. Create a volume with that backing image.
4. See the volume stuck in attaching state and on the backing image page the image still shows state as Healthy for failed images on the disks

Expected behavior
The state backing image should change to Failed.

Log

[longhorn-manager-nx87j] time="2021-04-16T17:44:42Z" level=debug msg="Prepare to create instance volume-test-2-r-e66103a8" 
[longhorn-manager-nx87j] time="2021-04-16T17:44:42Z" level=debug msg="Replica volume-test-2-r-e66103a8 is waiting for backing image bi-2 downloading file to node khushboo-test-wk1 disk 979e7789-1682-4231-b7fc-dea07dc8a797 from URL lhhlkjhsdlasdgaisudga, the current state is failed" 
[longhorn-manager-nx87j] time="2021-04-16T17:44:42Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"volume-test-2-r-e66103a8\", UID:\"e88bcf3b-c1d2-4539-b05b-51e20512bebf\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"182815\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts volume-test-2-r-e66103a8" 
[longhorn-manager-nx87j] time="2021-04-16T17:44:43Z" level=debug msg="Current backing image manager is not eligible for pulling" backingImage=bi-2 backingImageManager=backing-image-manager-51e2-979e backingImageUUID=dee94276 controller=longhorn-backing-image-manager diskUUID=979e7789-1682-4231-b7fc-dea07dc8a797 node=khushboo-test-wk1 nodeID=khushboo-test-wk1 url=lhhlkjhsdlasdgaisudga 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=debug msg="Prepare to create instance volume-test-2-r-f93857ac" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=debug msg="Replica volume-test-2-r-f93857ac is waiting for backing image bi-2 downloading file to node khushboo-test1 disk 4b5cda37-251b-4457-8d52-8a3ebe2c481f from URL lhhlkjhsdlasdgaisudga, the current state is failed" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"volume-test-2-r-f93857ac\", UID:\"4f440c8d-b1d7-4189-ab4c-b9e52315923c\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"182828\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts volume-test-2-r-f93857ac" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=debug msg="Start to pull backing image" backingImage=bi-2 backingImageManager=backing-image-manager-51e2-4b5c backingImageUUID=dee94276 controller=longhorn-backing-image-manager diskUUID=4b5cda37-251b-4457-8d52-8a3ebe2c481f node=khushboo-test1 nodeID=khushboo-test1 url=lhhlkjhsdlasdgaisudga 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=warning msg="Error syncing Longhorn backing image manager longhorn-system/backing-image-manager-51e2-4b5c: BackingImageManagerController failed to sync longhorn-system/backing-image-manager-51e2-4b5c: failed to download backing images: rpc error: code = Unknown desc = failed to get file size before pulling: Get \"lhhlkjhsdlasdgaisudga\": unsupported protocol scheme \"\"" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=debug msg="Prepare to create instance volume-test-2-r-f93857ac" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=debug msg="Replica volume-test-2-r-f93857ac is waiting for backing image bi-2 downloading file to node khushboo-test1 disk 4b5cda37-251b-4457-8d52-8a3ebe2c481f from URL lhhlkjhsdlasdgaisudga, the current state is failed" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"volume-test-2-r-f93857ac\", UID:\"4f440c8d-b1d7-4189-ab4c-b9e52315923c\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"182828\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts volume-test-2-r-f93857ac" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=debug msg="Start to pull backing image" backingImage=bi-2 backingImageManager=backing-image-manager-51e2-4b5c backingImageUUID=dee94276 controller=longhorn-backing-image-manager diskUUID=4b5cda37-251b-4457-8d52-8a3ebe2c481f node=khushboo-test1 nodeID=khushboo-test1 url=lhhlkjhsdlasdgaisudga 
[longhorn-manager-qj9v7] E0416 17:44:46.788833       1 backing_image_manager_controller.go:216] BackingImageManagerController failed to sync longhorn-system/backing-image-manager-51e2-4b5c: failed to download backing images: rpc error: code = Unknown desc = failed to get file size before pulling: Get "lhhlkjhsdlasdgaisudga": unsupported protocol scheme "" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=warning msg="Dropping Longhorn backing image manager longhorn-system/backing-image-manager-51e2-4b5c out of the queue: BackingImageManagerController failed to sync longhorn-system/backing-image-manager-51e2-4b5c: failed to download backing images: rpc error: code = Unknown desc = failed to get file size before pulling: Get \"lhhlkjhsdlasdgaisudga\": unsupported protocol scheme \"\"" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=debug msg="Start to pull backing image" backingImage=bi-2 backingImageManager=backing-image-manager-51e2-4b5c backingImageUUID=dee94276 controller=longhorn-backing-image-manager diskUUID=4b5cda37-251b-4457-8d52-8a3ebe2c481f node=khushboo-test1 nodeID=khushboo-test1 url=lhhlkjhsdlasdgaisudga 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=warning msg="Error syncing Longhorn backing image manager longhorn-system/backing-image-manager-51e2-4b5c: BackingImageManagerController failed to sync longhorn-system/backing-image-manager-51e2-4b5c: failed to download backing images: rpc error: code = Unknown desc = failed to get file size before pulling: Get \"lhhlkjhsdlasdgaisudga\": unsupported protocol scheme \"\"" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=debug msg="Start to pull backing image" backingImage=bi-2 backingImageManager=backing-image-manager-51e2-4b5c backingImageUUID=dee94276 controller=longhorn-backing-image-manager diskUUID=4b5cda37-251b-4457-8d52-8a3ebe2c481f node=khushboo-test1 nodeID=khushboo-test1 url=lhhlkjhsdlasdgaisudga 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=warning msg="Error syncing Longhorn backing image manager longhorn-system/backing-image-manager-51e2-4b5c: BackingImageManagerController failed to sync longhorn-system/backing-image-manager-51e2-4b5c: failed to download backing images: rpc error: code = Unknown desc = failed to get file size before pulling: Get \"lhhlkjhsdlasdgaisudga\": unsupported protocol scheme \"\"" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=debug msg="Start to pull backing image" backingImage=bi-2 backingImageManager=backing-image-manager-51e2-4b5c backingImageUUID=dee94276 controller=longhorn-backing-image-manager diskUUID=4b5cda37-251b-4457-8d52-8a3ebe2c481f node=khushboo-test1 nodeID=khushboo-test1 url=lhhlkjhsdlasdgaisudga 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=warning msg="Error syncing Longhorn backing image manager longhorn-system/backing-image-manager-51e2-4b5c: BackingImageManagerController failed to sync longhorn-system/backing-image-manager-51e2-4b5c: failed to download backing images: rpc error: code = Unknown desc = failed to get file size before pulling: Get \"lhhlkjhsdlasdgaisudga\": unsupported protocol scheme \"\"" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=debug msg="Start to pull backing image" backingImage=bi-2 backingImageManager=backing-image-manager-51e2-4b5c backingImageUUID=dee94276 controller=longhorn-backing-image-manager diskUUID=4b5cda37-251b-4457-8d52-8a3ebe2c481f node=khushboo-test1 nodeID=khushboo-test1 url=lhhlkjhsdlasdgaisudga 
[longhorn-manager-qj9v7] E0416 17:44:46.861379       1 backing_image_manager_controller.go:216] BackingImageManagerController failed to sync longhorn-system/backing-image-manager-51e2-4b5c: failed to download backing images: rpc error: code = Unknown desc = failed to get file size before pulling: Get "lhhlkjhsdlasdgaisudga": unsupported protocol scheme "" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=warning msg="Dropping Longhorn backing image manager longhorn-system/backing-image-manager-51e2-4b5c out of the queue: BackingImageManagerController failed to sync longhorn-system/backing-image-manager-51e2-4b5c: failed to download backing images: rpc error: code = Unknown desc = failed to get file size before pulling: Get \"lhhlkjhsdlasdgaisudga\": unsupported protocol scheme \"\"" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=debug msg="Start to pull backing image" backingImage=bi-2 backingImageManager=backing-image-manager-51e2-4b5c backingImageUUID=dee94276 controller=longhorn-backing-image-manager diskUUID=4b5cda37-251b-4457-8d52-8a3ebe2c481f node=khushboo-test1 nodeID=khushboo-test1 url=lhhlkjhsdlasdgaisudga 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=warning msg="Error syncing Longhorn backing image manager longhorn-system/backing-image-manager-51e2-4b5c: BackingImageManagerController failed to sync longhorn-system/backing-image-manager-51e2-4b5c: failed to download backing images: rpc error: code = Unknown desc = failed to get file size before pulling: Get \"lhhlkjhsdlasdgaisudga\": unsupported protocol scheme \"\"" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=debug msg="Start to pull backing image" backingImage=bi-2 backingImageManager=backing-image-manager-51e2-4b5c backingImageUUID=dee94276 controller=longhorn-backing-image-manager diskUUID=4b5cda37-251b-4457-8d52-8a3ebe2c481f node=khushboo-test1 nodeID=khushboo-test1 url=lhhlkjhsdlasdgaisudga 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=warning msg="Error syncing Longhorn backing image manager longhorn-system/backing-image-manager-51e2-4b5c: BackingImageManagerController failed to sync longhorn-system/backing-image-manager-51e2-4b5c: failed to download backing images: rpc error: code = Unknown desc = failed to get file size before pulling: Get \"lhhlkjhsdlasdgaisudga\": unsupported protocol scheme \"\"" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=debug msg="Start to pull backing image" backingImage=bi-2 backingImageManager=backing-image-manager-51e2-4b5c backingImageUUID=dee94276 controller=longhorn-backing-image-manager diskUUID=4b5cda37-251b-4457-8d52-8a3ebe2c481f node=khushboo-test1 nodeID=khushboo-test1 url=lhhlkjhsdlasdgaisudga 
[longhorn-manager-qj9v7] E0416 17:44:46.990876       1 backing_image_manager_controller.go:216] BackingImageManagerController failed to sync longhorn-system/backing-image-manager-51e2-4b5c: failed to download backing images: rpc error: code = Unknown desc = failed to get file size before pulling: Get "lhhlkjhsdlasdgaisudga": unsupported protocol scheme "" 
[longhorn-manager-qj9v7] time="2021-04-16T17:44:46Z" level=warning msg="Dropping Longhorn backing image manager longhorn-system/backing-image-manager-51e2-4b5c out of the queue: BackingImageManagerController failed to sync longhorn-system/backing-image-manager-51e2-4b5c: failed to download backing images: rpc error: code = Unknown desc = failed to get file size before pulling: Get \"lhhlkjhsdlasdgaisudga\": unsupported protocol scheme \"\"" 

Environment:

• Longhorn version: Longhorn-master - 04/16/2021
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Kubectl
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: RKE - K8s v1.20.5
  • Number of management node in the cluster: 1
  • Number of worker node in the cluster: 4
• Node config
  • OS type and version: Ubuntu 20.04
  • CPU per node: 2
  • Memory per node: 4
  • Disk type(e.g. SSD/NVMe): SSD

[innobead]

@shuo-wu is this still a valid issue?

[shuo-wu]

No. The UI won't present the state for the whole backing image now. Let me close this issue.