[BUG] Failed to restore a backup to file by the scripts/restore-backup-to-file.sh with a CIFS backup target.

[mantissahz]

Describe the bug

When using scripts/restore-backup-to-file.sh script with CIFS backend, no credentials can be passed to the engine container.

This results in an error when trying to mount CIFS share:

time="2024-03-05T14:35:16Z" level=fatal msg="Error running restore to file command" func=cmd.RestoreToFileCmd.func1 file="restore_to_file.go:66" error="cannot mount CIFS share backup/backup, options [soft]: mount failed: exit status 1\nMounting command: mount\nMounting arguments: -t cifs -o soft,<masked>,<masked> //backup/backup /var/lib/longhorn-backupstore-mounts/backup/backup\nOutput: username specified with no parameter\n"

To Reproduce

• Restore a backup to file by longhorn/scripts/restore-backup-to-file.sh with a CIFS backup target.

Expected behavior

Restoring a backup to file by longhorn/scripts/restore-backup-to-file.sh with a CIFS backup target works well.

Support bundle for troubleshooting

Environment

• Longhorn version:
• Impacted volume (PV):
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl):
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version:
  • Number of control plane nodes in the cluster:
  • Number of worker nodes in the cluster:
• Node config
  • OS type and version:
  • Kernel version:
  • CPU per node:
  • Memory per node:
  • Disk type (e.g. SSD/NVMe/HDD):
  • Network bandwidth between the nodes (Gbps):
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
• Number of Longhorn volumes in the cluster:

Additional context

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:
  [BUG] Failed to restore a backup to file by the scripts/restore-backup-to-file.sh with a CIFS backup target. #8126 (comment)

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at:
  The PR for the chart change is at:
  The PR for the script change is at:
  fix(scripts): Add CIFS credentials options to restore-backup-to-file #8110

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at

• Which areas/issues this PR might have potential impacts on?
  Area
  Issues

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at
  The automation test case PR is at
  The issue of automation test case implementation is at (please create by the template)

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

[chriscchien]

Hi @mantissahz

I got error when using longhornio/longhorn-engine:master to restore cifs backup, but the restore success when using v1.6.2-rc2 engine image, could you help me on this? thank you.

master

> ./restore-backup-to-file.sh --cifs-username "longhorn-cifs-username"  \
          --cifs-password  "longhorn-cifs-password" \
          --backup-url "cifs://10.43.35.102/backupstore?backup=backup-5a0948028f414983&volume=pvc-325033b4-90c9-46b0-92c7-643d7199894a" \
          --output-file "volume.raw" \
          --output-format raw \
          -v master
time="2024-03-22T02:16:54Z" level=info msg="Running restore to file command: backup-url=cifs://10.43.35.102/backupstore?backup=backup-5a0948028f414983&volume=pvc-325033b4-90c9-46b0-92c7-643d7199894a backing-file= output-file=/tmp/restore/volume.raw output-format=raw"
time="2024-03-22T02:16:54Z" level=info msg="Output file path=/tmp/restore/volume.raw"
time="2024-03-22T02:16:54Z" level=info msg="Start to restore cifs://10.43.35.102/backupstore?backup=backup-5a0948028f414983&volume=pvc-325033b4-90c9-46b0-92c7-643d7199894a to backup.img"
time="2024-03-22T02:16:54Z" level=error msg="failed to remove tmp file=backup.img: remove /backup.img: no such file or directory"
time="2024-03-22T02:16:54Z" level=error msg="failed to remove tmp file=backup.img.converted: remove /backup.img.converted: no such file or directory"
time="2024-03-22T02:16:54Z" level=error msg="failed to remove tmp file=backing.img.cp: remove /backing.img.cp: no such file or directory"
time="2024-03-22T02:16:54Z" level=fatal msg="Error running restore to file command: Driver cifs is not supported!"

v1.6.2-rc2

./restore-backup-to-file.sh --cifs-username "longhorn-cifs-username"  \
          --cifs-password  "longhorn-cifs-password" \
          --backup-url "cifs://10.43.35.102/backupstore?backup=backup-5a0948028f414983&volume=pvc-325033b4-90c9-46b0-92c7-643d7199894a" \
          --output-file "volume.raw" \
          --output-format raw \
          -v v1.6.1-rc2
time="2024-03-22T02:16:13Z" level=info msg="Running restore to file command: backup-url=cifs://10.43.35.102/backupstore?backup=backup-5a0948028f414983&volume=pvc-325033b4-90c9-46b0-92c7-643d7199894a backing-file= output-file=/tmp/restore/volume.raw output-format=raw" func=cmd.BackupCmd.RestoreToFileCmd.func4 file="restore_to_file.go:65"
time="2024-03-22T02:16:13Z" level=info msg="Output file path=/tmp/restore/volume.raw" func=cmd.restoreToFile file="restore_to_file.go:141"
time="2024-03-22T02:16:13Z" level=info msg="Start to restore cifs://10.43.35.102/backupstore?backup=backup-5a0948028f414983&volume=pvc-325033b4-90c9-46b0-92c7-643d7199894a to backup.img" func=cmd.restoreToFile file="restore_to_file.go:145"
time="2024-03-22T02:16:13Z" level=info msg="Mounting CIFS share cifs://10.43.35.102/backupstore on mount point /var/lib/longhorn-backupstore-mounts/10_43_35_102/backupstore with options [soft]" func="cifs.(*BackupStoreDriver).mount" file="cifs.go:115" pkg=cifs
time="2024-03-22T02:16:13Z" level=info msg="Loaded driver for cifs://10.43.35.102/backupstore" func=cifs.initFunc file="cifs.go:94" pkg=cifs
time="2024-03-22T02:16:13Z" level=info msg="Saving config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:81" filepath=backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck kind=cifs object=config pkg=backupstore reason=start
time="2024-03-22T02:16:13Z" level=info msg="Saved config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:92" filepath=backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck kind=cifs object=config pkg=backupstore reason=complete
time="2024-03-22T02:16:13Z" level=info msg="Stored lock backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck type 1 on backupstore" func=backupstore.saveLock file="lock.go:190" pkg=backupstore
time="2024-03-22T02:16:15Z" level=info msg="Loading config in backupstore" func=backupstore.LoadConfigInBackupStore file="config.go:56" filepath=backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck kind=cifs object=config pkg=backupstore reason=start
time="2024-03-22T02:16:15Z" level=info msg="Loaded config in backupstore" func=backupstore.LoadConfigInBackupStore file="config.go:67" filepath=backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck kind=cifs object=config pkg=backupstore reason=complete
time="2024-03-22T02:16:15Z" level=info msg="Loaded lock backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck type 1 on backupstore" func=backupstore.loadLock file="lock.go:171" pkg=backupstore
time="2024-03-22T02:16:15Z" level=info msg="Trying to acquire lock backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck" func="backupstore.(*FileLock).canAcquire" file="lock.go:65" lock="{ volume: pvc-325033b4-90c9-46b0-92c7-643d7199894a, name: lock-30f8bd951b3a49b1, type: 1, acquired: false, serverTime: 2024-03-22 02:16:13.6094932 +0000 UTC }" pkg=backupstore
time="2024-03-22T02:16:15Z" level=info msg="backupstore volume pvc-325033b4-90c9-46b0-92c7-643d7199894a contains locks [{ volume: , name: lock-30f8bd951b3a49b1, type: 1, acquired: false, serverTime: 2024-03-22 02:16:13.6094932 +0000 UTC }]" func="backupstore.(*FileLock).canAcquire" file="lock.go:66" pkg=backupstore
time="2024-03-22T02:16:15Z" level=info msg="Acquired lock backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck type 1 on backupstore" func="backupstore.(*FileLock).Lock" file="lock.go:114" pkg=backupstore
time="2024-03-22T02:16:15Z" level=info msg="Saving config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:81" filepath=backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck kind=cifs object=config pkg=backupstore reason=start
time="2024-03-22T02:16:15Z" level=info msg="Saved config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:92" filepath=backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck kind=cifs object=config pkg=backupstore reason=complete
time="2024-03-22T02:16:15Z" level=info msg="Stored lock backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck type 1 on backupstore" func=backupstore.saveLock file="lock.go:190" pkg=backupstore
time="2024-03-22T02:16:15Z" level=info msg="Loading config in backupstore" func=backupstore.LoadConfigInBackupStore file="config.go:56" filepath=backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/volume.cfg kind=cifs object=config pkg=backupstore reason=start
time="2024-03-22T02:16:15Z" level=info msg="Loaded config in backupstore" func=backupstore.LoadConfigInBackupStore file="config.go:67" filepath=backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/volume.cfg kind=cifs object=config pkg=backupstore reason=complete
time="2024-03-22T02:16:15Z" level=info msg="Loading config in backupstore" func=backupstore.LoadConfigInBackupStore file="config.go:56" filepath=backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/backups/backup_backup-5a0948028f414983.cfg kind=cifs object=config pkg=backupstore reason=start
time="2024-03-22T02:16:15Z" level=info msg="Loaded config in backupstore" func=backupstore.LoadConfigInBackupStore file="config.go:67" filepath=backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/backups/backup_backup-5a0948028f414983.cfg kind=cifs object=config pkg=backupstore reason=complete
time="2024-03-22T02:16:15Z" level=info msg="Restoring delta block backup" func=backupstore.RestoreDeltaBlockBackup file="deltablock.go:683" backup_url="cifs://10.43.35.102/backupstore?backup=backup-5a0948028f414983&volume=pvc-325033b4-90c9-46b0-92c7-643d7199894a" event=restore object=snapshot original_volume=pvc-325033b4-90c9-46b0-92c7-643d7199894a pkg=backupstore reason=start snapshot=backup-5a0948028f414983 volume_dev=backup.img
time="2024-03-22T02:16:15Z" level=info msg="Saving config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:81" filepath=backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck kind=cifs object=config pkg=backupstore reason=start
time="2024-03-22T02:16:15Z" level=info msg="Saved config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:92" filepath=backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck kind=cifs object=config pkg=backupstore reason=complete
time="2024-03-22T02:16:15Z" level=info msg="Stored lock backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck type 1 on backupstore" func=backupstore.saveLock file="lock.go:190" pkg=backupstore
time="2024-03-22T02:16:15Z" level=info msg="Truncate backup.img to size 2147483648" func=backupstore.RestoreDeltaBlockBackup.func2 file="deltablock.go:710" pkg=backupstore
0 / 100 [______________________________________________________________________________] 0.00% ? p/stime="2024-03-22T02:16:16Z" level=info msg="Removed lock backupstore/volumes/9c/df/pvc-325033b4-90c9-46b0-92c7-643d7199894a/locks/lock-30f8bd951b3a49b1.lck type 1 on backupstore" func=backupstore.removeLock file="lock.go:180" pkg=backupstore
0 / 100 [______________________________________________________________________________] 0.00% ? p/s0 / 100 [______________________________________________________________________________] 0.00% ? p/s0 / 100 [______________________________________________________________________________] 0.00% ? p/s0 / 100 [______________________________________________________________________________] 0.00% ? p/s0 / 100 [______________________________________________________________________________] 0.00% ? p/s0 / 100 [______________________________________________________________________________] 0.00% ? p/s0 / 100 [______________________________________________________________________________] 0.00% ? p/s0 / 100 [______________________________________________________________________________] 0.00% ? p/s0 / 100 [______________________________________________________________________________] 0.00% ? p/s0 / 100 [______________________________________________________________________________] 0.00% ? p/stime="2024-03-22T02:16:17Z" level=info msg="Done restoring cifs://10.43.35.102/backupstore?backup=backup-5a0948028f414983&volume=pvc-325033b4-90c9-46b0-92c7-643d7199894a to backup.img" func=cmd.restoreToFile file="restore_to_file.go:149"
time="2024-03-22T02:16:17Z" level=info msg="Start ConvertImage (raw) backup.img -> /tmp/restore/volume.raw" func=cmd.ConvertImage file="restore_to_file.go:236"
time="2024-03-22T02:16:17Z" level=info msg="Done ConvertImage (raw) backup.img -> /tmp/restore/volume.raw" func=cmd.ConvertImage file="restore_to_file.go:242"
time="2024-03-22T02:16:17Z" level=error msg="Failed to remove tmp file=backup.img.converted" func=cmd.CleanupTempFiles file="restore_to_file.go:228" error="remove /backup.img.converted: no such file or directory"
time="2024-03-22T02:16:17Z" level=error msg="Failed to remove tmp file=backing.img.cp" func=cmd.CleanupTempFiles file="restore_to_file.go:228" error="remove /backing.img.cp: no such file or directory"
time="2024-03-22T02:16:17Z" level=info msg="Done running restore to file command. Produced image: /tmp/restore/volume.raw" func=cmd.BackupCmd.RestoreToFileCmd.func4 file="restore_to_file.go:70"

[mantissahz]

Hi @chriscchien

./restore-backup-to-file.sh --cifs-username "longhorn-cifs-username"
--cifs-password "longhorn-cifs-password"
--backup-url "cifs://10.43.35.102/backupstore?backup=backup-5a0948028f414983&volume=pvc-325033b4-90c9-46b0-92c7-643d7199894a"
--output-file "volume.raw"
--output-format raw
-v master

I think the error is from the option -v, --version Longhorn version.
This option will pull the image you designate (longhornio/longhorn-engine:${version}) and the longhorn-engine image of the master branch should be master-head.
Could you try again with the option "-v master-head"?

[chriscchien]

Verified pass on longhorn master(83532b)

I can successfully restore a backup form cifs backupstore to /tmp/restore/volume.raw by below command, after mount /tmp/restore/volume.raw, the data was correct.

./restore-backup-to-file.sh --cifs-username "longhorn-cifs-username"  \
          --cifs-password  "longhorn-cifs-password" \
          --backup-url "cifs://10.43.78.230/backupstore?backup=backup-10e9900766a24365&volume=pvc-2871b7b6-8e8a-4207-b96a-ccea753afe6e" \
          --output-file "volume.raw" \
          --output-format raw \
          -v master-head