[BUG] volume attaching, stopping, and detaching

[TobiasS1402]

Describe the bug

When trying to create a volume attached to a stateFulSet longhorn is stuck on attaching, and failing after a while, and retrying.
The error I get is: failed to attach: rpc error: code = Aborted desc = volume pvc-6de20396-e1a7-4588-bc48-22957e28d016 is not ready for workloads

To Reproduce

Steps to reproduce the behavior:

1. Install and configure Longhorn
2. Apply Kubernetes manifest with VolumeClaimTemplate (ReadWriteOnce, 2Gi)
3. Wait for errors to pop up

Expected behavior

I would expect Longhorn to create a volume that is attached to the host of the pods.

Log or Support bundle

If applicable, add the Longhorn managers' log or support bundle when the issue happens.
You can generate a Support Bundle using the link at the footer of the Longhorn UI.

longhorn-support-bundle_1ce5ecf1-d4d1-4cf9-a685-dcc243202457_2022-04-19T05-57-02Z.zip

Environment

• Longhorn version: v1.2.4
• Installation method: Kubectl
• Kubernetes distro k3s v1.22.7+k3s1:
  • Number of management node in the cluster: 1
  • Number of worker node in the cluster: 3
• Node config
  • OS type and version: Debian GNU/Linux 11 in LXC
  • CPU per node: 4
  • Memory per node: 4096
  • Disk type(e.g. SSD/NVMe): SSD
  • Network bandwidth between the nodes: 1Gbit
• Underlying Infrastructure : Proxmox Virtual Environment
• Number of Longhorn volumes in the cluster: 1

Additional context

Add any other context about the problem here.

[derekbit]

@TobiasS1402
pvc-6de20396-e1a7-4588-bc48-22957e28d016 is deleted in support bundle.
For pvc-18a6173b-11ef-4b68-a2e1-6ad1c24f2c7e-e-874a00aa, I noticed the log message.
Have you installed and started the iscsi daemon on each node?
https://longhorn.io/docs/1.2.4/deploy/install/#installing-open-iscsi

2022-04-19T05:53:52.759259932Z time="2022-04-19T05:53:52Z" level=warning msg="pvc-18a6173b-11ef-4b68-a2e1-6ad1c24f2c7e-e-874a00aa: time=\"2022-04-19T05:53:28Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/1/ns/mnt --net=/host/proc/1/ns/net iscsiadm -m discovery -t sendtargets -p 172.17.93.164], output , stderr, iscsiadm: iscsid is not running. Could not start it up automatically using the startup command in the /etc/iscsi/iscsid.conf iscsid.startup setting. Please check that the file exists or that your init scripts have started iscsid.\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: iscsid is not running. Could not start it up automatically using the startup command in the /etc/iscsi/iscsid.conf iscsid.startup setting. Please check that the file exists or that your init scripts have started iscsid.\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""

[TobiasS1402]

@derekbit I reinstalled my cluster and i installed open-iscsi and nfs on every node to make sure and it still is having the same issue. I'll attach another support bundle.
longhorn-support-bundle_11dcf33e-557e-404f-8bae-0dc1c4eb2bcd_2022-04-23T09-14-51Z.zip
.

[derekbit]

Hello @TobiasS1402,
I also saw the iSCSI issue in your longhorn-manager log.
Just found you installed the longhorn on Debian GNU/Linux 11 in LXC. Longhorn cannot work properly in LXC environment and please refer to #2585.

[TobiasS1402]

Thanks for your input @derekbit, I am closing the issue.