zfs error: cannot destroy dataset is busy - help to debug

[boospy]

I am using Proxmox 8 with ZFS 2.2.7. Sometimes we have the problem that a subvol (container) cannot be deleted. For example

TASK ERROR: zfs error: cannot destroy 'tank01/subvol-122-disk-0': dataset is busy

We are 100% sure that there is no access to this dataset from such processes. I had the same behavior on another server with a container template. It could not be deleted, but it was never cloned or used. Still no access. We used strace and debug down to the process level:

[<0>] cv_wait_common+0x109/0x140 [spl] -> checken
[<0>] __cv_wait+0x15/0x30 [spl] -> checken
[<0>] rrw_enter_write+0x3c/0xb0 [zfs]
[<0>] rrm_enter+0x2a/0x90 [zfs]
[<0>] zfsvfs_teardown+0x1ee/0x3a0 [zfs]
[<0>] zfs_umount+0x3a/0x110 [zfs]
[<0>] zpl_put_super+0x2c/0x50 [zfs]
[<0>] generic_shutdown_super+0x79/0x180
[<0>] kill_anon_super+0x18/0x50
[<0>] zpl_kill_sb+0x1a/0x30 [zfs]
[<0>] deactivate_locked_super+0x32/0xc0
[<0>] deactivate_super+0x46/0x60
[<0>] cleanup_mnt+0xc6/0x170
[<0>] __cleanup_mnt+0x12/0x20
[<0>] task_work_run+0x5e/0xa0
[<0>] syscall_exit_to_user_mode+0x257/0x260
[<0>] do_syscall_64+0x8d/0x170
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80

We tried to actively reproduce it. To do this, I executed a test program for days that repeatedly created 50 containers, let them run for 5 minutes, rebooted, ran them again for 5 minutes, then shut them down and destroyed them. This happened on a PHY server and also in a virtual environment where resources were so severely restricted that an I/O wait of approx. 80-90% was generated. But I couldn't reproduce it.

If the behavior occurs again on productive servers, how can we debug it to find the error? What is the best way to proceed?

Thanks a lot
Best Regards
Boospy