[BUG] Longhorn cifs backups cannot find credentials

[DizzieNight]

Describe the bug

Longhorn won't find my credentials for cifs backups. Have also tried changing to nfs but it kept complaining about cifs credentials even though I had the correct credentials passed through.

To Reproduce

Hard to reproduce. Enter the cifs backup target in the gui also with the credentials.

Expected behavior

Should allow me to backup my volumes to my unraid server.

Support bundle for troubleshooting

supportbundle_913f7d1d-6412-427c-87ee-1fe64c171768_2024-05-24T02-39-44Z.zip

Environment

Longhorn version: 1.6.1
Impacted volume (PV): bazarr, deluge-config, deluge-data, flaresolverr, jfa-go, kavita, lidarr, overseerr, radarr, readarr, sonarr, vaultwarden, whisparr
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: K3s 1.27.10+k3s2
    Number of control plane nodes in the cluster: 3
    Number of worker nodes in the cluster: 3
Node config
    OS type and version: Debian Lunar
    Kernel version: 6.2.0-1018-kvm
    CPU per node: 4
    Memory per node: 6GB
    Disk type (e.g. SSD/NVMe/HDD): SSD
    Network bandwidth between the nodes (Gbps): 1Gbps
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Proxmox VMs
Number of Longhorn volumes in the cluster: 19

Additional context

[mantissahz]

Hi @ZanderPittari,

The backup target URL had been modified to nfs and credential secret had been removed.
I believe you have a SMB/CIFS server for backups, right?

The last error log in the longhorn-manager is

2024-05-24T12:38:41.273798359+10:00 E0524 02:38:41.270519 1545448 mount_linux.go:230] Mount failed: exit status 1
2024-05-24T12:38:41.273801804+10:00 Mounting command: mount
2024-05-24T12:38:41.273804717+10:00 Mounting arguments: -t cifs -o soft,<masked>,<masked> //192.168.10.2/longhorn-backups /var/lib/longhorn-backupstore-mounts/192_168_10_2/longhorn-backups
2024-05-24T12:38:41.273807308+10:00 Output: username specified with no parameter
2024-05-24T12:38:41.273809302+10:00

Could you provide the secret smbcreds or check if CIFS_USERNAME and CIFS_PASSWORD are set correctly? (ref: Set up SMB/CIFS Backupstore)

You could empty the Backup Target to clear the error message could not access s3 without credential secret first and reset the backup target URL and credential secret.
(a bit related to #8299)

[DizzieNight]

I think I accidentally uploaded the wrong support bundle. Let me attach a new one:
supportbundle_913f7d1d-6412-427c-87ee-1fe64c171768_2024-05-24T04-05-43Z.zip

[mantissahz]

Still the same error, Could you provide the secret smbcreds or check if CIFS_USERNAME and CIFS_PASSWORD are set correctly? (ref: Set up SMB/CIFS Backupstore)

2024-05-24T14:06:11.397990263+10:00 time="2024-05-24T04:06:11Z" level=info msg="Mounting CIFS share cifs://192.168.10.2/longhorn-backups on mount point /var/lib/longhorn-backupstore-mounts/192_168_10_2/longhorn-backups with optio
ns [soft]" func="cifs.(*BackupStoreDriver).mount" file="cifs.go:115" pkg=cifs
2024-05-24T14:06:11.397994138+10:00 E0524 04:06:11.395709 1571963 mount_linux.go:230] Mount failed: exit status 1
2024-05-24T14:06:11.397997593+10:00 Mounting command: mount
2024-05-24T14:06:11.397999919+10:00 Mounting arguments: -t cifs -o soft,<masked>,<masked> //192.168.10.2/longhorn-backups /var/lib/longhorn-backupstore-mounts/192_168_10_2/longhorn-backups
2024-05-24T14:06:11.398002006+10:00 Output: username specified with no parameter
2024-05-24T14:06:11.398003433+10:00
2024-05-24T14:06:11.398005500+10:00 time="2024-05-24T04:06:11Z" level=fatal msg="Failed to run list system backup command" func=cmd.SystemBackupCmd.SystemBackupListCmd.func4 file="system_backup.go:72" error="cannot mount CIFS share 192.168.10.2/longhorn-backups, options [soft]: mount failed: exit status 1\nMounting command: mount\nMounting arguments: -t cifs -o soft,<masked>,<masked> //192.168.10.2/longhorn-backups /var/lib/longhorn-backupstore-mounts/192_168_10_2/longhorn-backups\nOutput: username specified with no parameter\n"
2024-05-24T14:06:11.398007061+10:00 : exit status 1

Not sure it is about kernel issue of CIFS (#8611)

[DizzieNight]

I know the creds work because I am using the same ones for other shares. The details are below:
CIFS_USERNAME= zander
CIFS_PASSWORD= 2iPr8cCWAP7rPz

[mantissahz]

The error occurred at list system backup for synchronizing the system backup but it would synchronize the backup volumes and backup backing images first (they should fail if the secret setting is incorrect.)

Could you upgrade to v1.6.2 to check if it will occur again because we have modified the flow to get the synchronizing information?

[DizzieNight]

I just upgraded to 1.6.2 but now getting this error. Which looks similar to what you posted above:

[mantissahz]

Could you show the secret information by the command kubectl -n longhorn-system get secret smbcreds -oyaml?

[DizzieNight]

This is what I get. The username and password are encrypted

[mantissahz]

Hi @ZanderPittari,

Please replace the data.username to data.CIFS_USERNAME and data.password to data.CIFS_PASSWORD and then check if the backup target works well. Thanks.

like:

#!/bin/bash

USERNAME=${Username of SMB/CIFS Server}
PASSWORD=${Password of SMB/CIFS Server}

CIFS_USERNAME=`echo -n ${USERNAME} | base64`
CIFS_PASSWORD=`echo -n ${PASSWORD} | base64`

cat <<EOF >>cifs_secret.yml
apiVersion: v1
kind: Secret
metadata:
  name: cifs-secret
  namespace: longhorn-system
type: Opaque
data:
  CIFS_USERNAME: ${CIFS_USERNAME}
  CIFS_PASSWORD: ${CIFS_PASSWORD}
EOF

kubectl apply -f cifs_secret.yml

[DizzieNight]

Hi @ZanderPittari,

Please replace the data.username to data.CIFS_USERNAME and data.password to data.CIFS_PASSWORD and then check if the backup target works well. Thanks.

like:

#!/bin/bash

USERNAME=${Username of SMB/CIFS Server}
PASSWORD=${Password of SMB/CIFS Server}

CIFS_USERNAME=`echo -n ${USERNAME} | base64`
CIFS_PASSWORD=`echo -n ${PASSWORD} | base64`

cat <<EOF >>cifs_secret.yml
apiVersion: v1
kind: Secret
metadata:
  name: cifs-secret
  namespace: longhorn-system
type: Opaque
data:
  CIFS_USERNAME: ${CIFS_USERNAME}
  CIFS_PASSWORD: ${CIFS_PASSWORD}
EOF

kubectl apply -f cifs_secret.yml

You are a legend, it's backing up now. Didn't realise it had to use a specific naming convention