iscsi fails to update database with CHAP

[ohthehugemanatee]

Version:
k3s version v1.17.3+k3s1 (5b17a17)

K3s arguments:
curl -sfL https://get.k3s.io | sh -

Describe the bug
iSCSI target which works from the CLI does not mount from kubernetes pod. The error indicates a failure to update the iscsiadm database with CHAP.

To Reproduce

• create an iscsi target with CHAP authentication. Validate that it is accessible from the cluster members with sudo iscsiadm -m discovery -t sendtargets -p 192.168.1.50 && sudo iscsiadm -m node --login
• create a Pod which attempts to use the target as a Volume mount:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nginx
    spec:
      volumes:
        - name: iscsi
          iscsi:
            targetPortal: 192.168.1.50
            iqn: iqn.2003-01.org.linux-iscsi.warehouse:sonarr
            lun: 0
            fsType: ext4
            readOnly: false
            chapAuthDiscovery: true
            chapAuthSession: true
            secretRef:
              name: iscsi-creds
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        volumeMounts:
          - name: iscsi
            mountPath: "/config"

Expected behavior
iSCSI volume is mounted into the pod.

Actual behavior
Pod stays in ContainerCreating state. Event log ends with:

  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Normal   Scheduled               <unknown>          default-scheduler        Successfully assigned default/sonarr-55b4766fbb-jk6qf to cluster2
  Normal   SuccessfulAttachVolume  37s                attachdetach-controller  AttachVolume.Attach succeeded for volume "iscsi"
  Warning  FailedMount             12s (x6 over 29s)  kubelet, cluster2        MountVolume.WaitForAttach failed for volume "iscsi" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to update iscsi node to portal 192.168.1.50:3260 error: iscsi: failed to update node with CHAP, output: iscsiadm: No records found

Additional context / logs
The same deployment worked fine yesterday... today I realized I forgot to set the deployment strategy type. When I updated the deployment, this started happening. It has persisted through restarts of open-iscsi, k3s-agent, and k3s. The mount still works manually.

The fact that it gets to updating the node with CHAP means discovery worked. Running the same commands kubernetes does from the CLI directly works.

pi@cluster2:~ $ sudo iscsiadm -m discovery -t sendtargets -p 192.168.1.50
192.168.1.50:3260,1 iqn.2003-01.org.linux-scsi.warehouse:sonarr
pi@cluster2:~ $ sudo iscsiadm -m node -p 192.168.1.50 -T iqn.2003-01.org.linux-scsi.warehouse:sonarr -o update -n "node.session.auth.authmethod" -v "CHAP"
pi@cluster2:~ $ sudo iscsiadm -m node -p 192.168.1.50 -T iqn.2003-01.org.linux-scsi.warehouse:sonarr --login
Logging in to [iface: default, target: iqn.2003-01.org.linux-scsi.warehouse:sonarr, portal: 192.168.1.50,3260] (multiple)
Login to [iface: default, target: iqn.2003-01.org.linux-scsi.warehouse:sonarr, portal: 192.168.1.50,3260] successful.

[yurinnick]

Is it specific for iscsi volume drives or it's affecting drivers like iscsi-based solutions like OpenEBS as well?

[ohthehugemanatee]

I have no experience with OpenEBS, so I couldn't tell you. :|

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.