[BUG]This is a empty backup，but i canot delete。

[goer3]

Describe the bug (🐛 if you encounter this issue)

Expected behavior

I deleted all pv and pvc on --force，then i canot delete this folder. it is empty!

Environment

• Longhorn version:v1.4.1
• Installation method (Helm):

[derekbit]

Can you check the log in instance-manager-r pods? It is usually caused by a permission issue.

[goer3]

i had checked! after i delete again, the instance-manager-r has no log.

[derekbit]

Any clue in longhorn-manager pods?

[goer3]

yes!
this is delete request!

it look like ok! but no effet.

[derekbit]

The deletion is async. The status 200 OK just tells the caller longhorn system has received the request but cannot guarantee the success of the deletion.

Can you provide us with the support bundle?

[goer3]

supportbundle_86280e6d-2b3f-428e-aba3-fca5776adf1f_2023-03-22T17-07-35Z.zip

ok,thanks！

[mantissahz]

Just saw a lot of access denied by server in longhorn-manager of master-01 node

2023-03-23T01:07:46.806906232+08:00 time="2023-03-22T17:07:46Z" level=error msg="Error clean up remote backup volume" backupVolume=pvc-37254b90-9d4c-4348-852c-9e2ebe57bce2 controller=longhorn-backup-volume error="error deleting backup volume: failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.4.1/longhorn [backup rm --volume pvc-37254b90-9d4c-4348-852c-9e2ebe57bce2 nfs://192.168.2.21:/data/backup], output cannot mount nfs 192.168.2.21:/data/backup: vers=4.0: failed to execute: mount [-t nfs4 -o nfsvers=4.0 -o actimeo=1 192.168.2.21:/data/backup /var/lib/longhorn-backupstore-mounts/192_168_2_21/data/backup], output mount.nfs4: access denied by server while mounting 192.168.2.21:/data/backup\n, error exit status 32: vers=4.1: failed to execute: mount [-t nfs4 -o nfsvers=4.1 -o actimeo=1 192.168.2.21:/data/backup /var/lib/longhorn-backupstore-mounts/192_168_2_21/data/backup], output mount.nfs4: access denied by server while mounting 192.168.2.21:/data/backup\n, error exit status 32: vers=4.2: failed to execute: mount [-t nfs4 -o nfsvers=4.2 -o actimeo=1 192.168.2.21:/data/backup /var/lib/longhorn-backupstore-mounts/192_168_2_21/data/backup], output mount.nfs4: access denied by server while mounting 192.168.2.21:/data/backup\n, error exit status 32: Cannot mount using NFSv4\n, stderr, time=\"2023-03-22T17:07:46Z\" level=error msg=\"cannot mount nfs 192.168.2.21:/data/backup: vers=4.0: failed to execute: mount [-t nfs4 -o nfsvers=4.0 -o actimeo=1 192.168.2.21:/data/backup /var/lib/longhorn-backupstore-mounts/192_168_2_21/data/backup], output mount.nfs4: access denied by server while mounting 192.168.2.21:/data/backup\\n, error exit status 32: vers=4.1: failed to execute: mount [-t nfs4 -o nfsvers=4.1 -o actimeo=1 192.168.2.21:/data/backup /var/lib/longhorn-backupstore-mounts/192_168_2_21/data/backup], output mount.nfs4: access denied by server while mounting 192.168.2.21:/data/backup\\n, error exit status 32: vers=4.2: failed to execute: mount [-t nfs4 -o nfsvers=4.2 -o actimeo=1 192.168.2.21:/data/backup /var/lib/longhorn-backupstore-mounts/192_168_2_21/data/backup], output mount.nfs4: access denied by server while mounting 192.168.2.21:/data/backup\\n, error exit status 32: Cannot mount using NFSv4\"\n, error exit status 1" node=master-01

@goer3 Could you help us check the nfs server is running and ACL is correct (this KB might help)? Then could you try to mount 192.168.2.21:/data/backup manually on the worker node and longhorn manager pods?

[goer3]

1. i can mount nfs dir on any nodes.
   mount -t nfs4 -o nfsvers=4.1 -o actimeo=1 192.168.2.21:/data/backup /data/test

2. then, i create a same name volume.
   

3. i create a new backup on the volume, but longhorn create a snapshot. the backup folder still empty.
   

4.finally,Where is this data saved? I'd better delete it directly.

[derekbit]

Your backups failed because the icon is blue rather than green. As @mantissahz said, you need to change the permission of the folder on your NFS server.
Mounting the folder successfully doesn't mean the permission is correct.

[goer3]

i chmod 777 on nfs server，after i backup, the backup folder still empty.

but they were some files in the nfs server dir.

[derekbit]

The backups are generated successfully from your image.

[goer3]

but they cannot display on the backup menu... still empty, so i want to delete this folder directly, but i donot know where is the data save.

[derekbit]

Can you check backup resources' status like kubectl -n longhorn-system get backups?

[goer3]

i find it in the backupvolumes, and then delete it.

[mantissahz]

Could you provide the nfs server setting in /etc/exports?
And get logs from longhorn-manager by this command kubectl -n longhorn-system logs longhorn-manager-gx28w to check if it is still the access denied issue.
And backup information by this command kubectl -n longhorn-system get backups.
Also Check the upper levels of directories ACL such as /data/backup or /data/backup/xx/d1 please.

[goer3]

/etc/exports：
/data/backup 192.168.2.0/24(rw,sync,insecure,no_subtree_check,no_root_squash)

when i run command:
kubectl -n longhorn-system delete backupvolumes.longhorn.io pvc-37254b90-9d4c-4348-852c-9e2ebe57bce2 --force --grace-period=0
but still cannot delete this backupvolumes.

then i edit this backupvolumes,change Finalizers: - longhorn.io to Finalizers: [].
this backupvolumes is deleted.

Thank you very much！ @mantissahz @derekbit @innobead