[BUG] Sometimes v2 volume stuck at attaching because engine error

[chriscchien]

Describe the bug (🐛 if you encounter this issue)

Sometimes v2 volume stuck at attaching state because engine error
From engine describe can see

Events:
  Type     Reason          Age    From                        Message
  ----     ------          ----   ----                        -------
  Warning  FailedStarting  4m22s  longhorn-engine-controller  Error starting vol1-e-f5fba3a5: failed to create instance: rpc error: code = Unknown desc = failed to start SPDK engine: rpc error: code = Unknown desc = failed to stop the mismatching NVMe initiator vol1 before starting: failed to logout target: failed to execute: nsenter [--mount=/host/proc/5734/ns/mnt --net=/host/proc/5734/ns/net nvme disconnect --nqn nqn.2023-01.io.longhorn.spdk:vol1-e-f5fba3a5], output , stderr Failed to scan topoplogy: No such file or directory
: exit status 1                                 

To Reproduce

Steps to reproduce the behavior:

1. Create/Attach a volume in maintenance mode
2. Delete replica trigger offline rebuilding
3. Delete that volume
4. Perform case2
5. Do above steps several times

Expected behavior

V2 volume attach should not stuck at attaching

Log or Support bundle

In longhorn-manager

time="2023-06-21T09:23:54Z" level=info msg="Creating instance vol1-e-f5fba3a5"
time="2023-06-21T09:23:54Z" level=error msg="Error syncing Longhorn engine" controller=longhorn-engine engine=longhorn-system/vol1-e-f5fba3a5 error="failed to sync engine for longhorn-system/vol1-e-f5fba3a5: failed to create instance: rpc error: code = Unknown desc = failed to start SPDK engine: rpc error: code = Unknown desc = failed to stop the mismatching NVMe initiator vol1 before starting: failed to logout target: failed to execute: nsenter [--mount=/host/proc/5734/ns/mnt --net=/host/proc/5734/ns/net nvme disconnect --nqn nqn.2023-01.io.longhorn.spdk:vol1-e-f5fba3a5], output , stderr Failed to scan topoplogy: No such file or directory\n: exit status 1" node=ip-172-31-94-193
time="2023-06-21T09:23:54Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"vol1-e-f5fba3a5\", UID:\"5ea1e409-3b5a-4258-aa37-d011455ebe34\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"17058\", FieldPath:\"\"}): type: 'Warning' reason: 'FailedStarting' Error starting vol1-e-f5fba3a5: failed to create instance: rpc error: code = Unknown desc = failed to start SPDK engine: rpc error: code = Unknown desc = failed to stop the mismatching NVMe initiator vol1 before starting: failed to logout target: failed to execute: nsenter [--mount=/host/proc/5734/ns/mnt --net=/host/proc/5734/ns/net nvme disconnect --nqn nqn.2023-01.io.longhorn.spdk:vol1-e-f5fba3a5], output , stderr Failed to scan topoplogy: No such file or directory\n: exit status 1"

supportbundle_4ed7928e-2dcc-4dc4-8444-681998a94511_2023-06-21T09-29-23Z.zip

Environment

• Longhorn version: master, v1,5,x
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl): kubectl
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3s

Additional context

N/A

[derekbit]

cc @shuo-wu

[shuo-wu]

@chriscchien What's the OS/kernel/nvmecli version of the test env?
I just saw the similar issue in nvme-cli community. Not sure if the root cause is the same.

Besides, I am confused about the reproducing step. What's the relationship between step 1-3 with step 4? Does the reproducing step mean that we need to keep triggering the offline rebuilding?

[chriscchien]

Hi @shuo-wu ,

I tested on Ubuntu 22.04, kernel 5.19.0-1025-aws and nvme version is 1.16, about the test steps, yes it's happening when I am test offline rebuilding.