Unable to Remove Specific Snapshots with Snapper

[realSConway]

Description:
I'm experiencing an issue with the snapper tool where I cannot remove certain snapshots, specifically snapshot 62 (range 63-90 i can delete). When I try to delete this snapshot using the command snapper delete 62, it returns an error message: "Deleting snapshot failed."

Context:

• I'm running a RAID1 setup on microos system.
• I've noticed that I can successfully remove other snapshots without any issues.
• the snapper-cleanup.service is failing to run properly. The status code is exited and the status is 1/FAILURE.

Steps to Reproduce:
Run the command snapper delete 62.
Observe the error message "Deleting snapshot failed."

Expected Behavior:
I expect snapper-cleanup.service to be running so that older snapshots are removed. This didn't happen causing my disk to get full. Additionally using the snapper delete command without encountering any errors.

Additional Information:

• Operating System:

NAME="openSUSE MicroOS"
# VERSION="20230811"
ID="opensuse-microos"
ID_LIKE="suse opensuse opensuse-tumbleweed"
VERSION_ID="20230811"
PRETTY_NAME="openSUSE MicroOS"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:microos:20230811"
BUG_REPORT_URL="https://bugzilla.opensuse.org"
SUPPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:MicroOS"
LOGO="distributor-logo-MicroOS"

• Snapper Version:

# snapper --version 
snapper 0.10.5 
flags btrfs,lvm,no-ext4,xattrs,rollback,btrfs-quota,selinux

# snapper get-config 
Key                    | Value
-----------------------+------
ALLOW_GROUPS           |      
ALLOW_USERS            |      
BACKGROUND_COMPARISON  | yes  
EMPTY_PRE_POST_CLEANUP | yes  
EMPTY_PRE_POST_MIN_AGE | 1800 
FREE_LIMIT             | 0.2  
FSTYPE                 | btrfs
NUMBER_CLEANUP         | yes  
NUMBER_LIMIT           | 2-10 
NUMBER_LIMIT_IMPORTANT | 4-10 
NUMBER_MIN_AGE         | 1800 
QGROUP                 | 1/0  
SPACE_LIMIT            | 0.5  
SUBVOLUME              | /    
SYNC_ACL               | no   
TIMELINE_CLEANUP       | yes  
TIMELINE_CREATE        | no   
TIMELINE_LIMIT_DAILY   | 10   
TIMELINE_LIMIT_HOURLY  | 10   
TIMELINE_LIMIT_MONTHLY | 10   
TIMELINE_LIMIT_WEEKLY  | 0    
TIMELINE_LIMIT_YEARLY  | 10   
TIMELINE_MIN_AGE       | 1800 

• RAID Configuration:

# cat /etc/mdadm.conf
DEVICE containers partitions
ARRAY /dev/md/0 UUID=2749c09e:175660b3:63d9ec2a:1685a914

# cat /proc/mdstat
Personalities : [raid1] 
md0 : active raid1 nvme1n1[0] nvme0n1[1]
      250058944 blocks super 1.0 [2/2] [UU]
      bitmap: 1/2 pages [4KB], 65536KB chunk

• Error Message (snapper-cleanup.service):

# systemctl status snapper-cleanup.service
× snapper-cleanup.service - Daily Cleanup of Snapper Snapshots
     Loaded: loaded (/etc/systemd/system/snapper-cleanup.service; static)
     Active: failed (Result: exit-code) since Wed 2023-08-23 15:38:20 CEST; 17h ago
   Duration: 145ms
TriggeredBy: ● snapper-cleanup.timer
       Docs: man:snapper(8)
             man:snapper-configs(5)
    Process: 20845 ExecStart=/usr/lib/snapper/systemd-helper --cleanup (code=exited, status=1/FAILURE)
   Main PID: 20845 (code=exited, status=1/FAILURE)
        CPU: 27ms

Aug 23 15:38:20 srv01 systemd[1]: Started Daily Cleanup of Snapper Snapshots.
Aug 23 15:38:20 srv01 systemd-helper[20845]: running cleanup for 'root'.
Aug 23 15:38:20 srv01 systemd-helper[20845]: running number cleanup for 'root'.
Aug 23 15:38:20 srv01 systemd-helper[20845]: Deleting snapshot failed.
Aug 23 15:38:20 srv01 systemd-helper[20845]: number cleanup for 'root' failed.
Aug 23 15:38:20 srv01 systemd-helper[20845]: running timeline cleanup for 'root'.
Aug 23 15:38:20 srv01 systemd-helper[20845]: running empty-pre-post cleanup for 'root'.
Aug 23 15:38:20 srv01 systemd[1]: snapper-cleanup.service: Main process exited, code=exited, status=1/FAILURE
Aug 23 15:38:20 srv01 systemd[1]: snapper-cleanup.service: Failed with result 'exit-code'.

Thank you for your assistance in addressing this issue.

[realSConway]

# snapper list
   # | Type   | Pre # | Date                             | User | Used Space | Cleanup | Description             | Userdata     
-----+--------+-------+----------------------------------+------+------------+---------+-------------------------+--------------
  0  | single |       |                                  | root |            |         | current                 |              
 62  | single |       | Fri 05 May 2023 01:42:32 AM CEST | root |   1.10 GiB | number  | Snapshot Update of #61  | important=yes
 91  | single |       | Sun 11 Jun 2023 12:16:52 AM CEST | root | 115.14 MiB | number  | Snapshot Update of #90  | important=yes
 92  | single |       | Mon 12 Jun 2023 12:45:51 AM CEST | root |  63.48 MiB | number  | Snapshot Update of #91  | important=yes
 93  | single |       | Wed 14 Jun 2023 12:49:37 AM CEST | root |  93.15 MiB | number  | Snapshot Update of #92  | important=yes
 94  | single |       | Thu 15 Jun 2023 01:42:01 AM CEST | root | 205.86 MiB | number  | Snapshot Update of #93  | important=yes
 95  | single |       | Sat 17 Jun 2023 12:18:23 AM CEST | root | 173.04 MiB | number  | Snapshot Update of #94  | important=yes
 96  | single |       | Sun 18 Jun 2023 01:04:13 AM CEST | root |  67.96 MiB | number  | Snapshot Update of #95  | important=yes
 97  | single |       | Tue 20 Jun 2023 01:38:08 AM CEST | root |  79.30 MiB | number  | Snapshot Update of #96  | important=yes
 98  | single |       | Wed 21 Jun 2023 01:37:39 AM CEST | root |  95.74 MiB | number  | Snapshot Update of #97  | important=yes
 99  | single |       | Thu 22 Jun 2023 12:14:43 AM CEST | root |  68.48 MiB | number  | Snapshot Update of #98  | important=yes
100  | single |       | Fri 23 Jun 2023 01:35:30 AM CEST | root |  90.18 MiB | number  | Snapshot Update of #99  | important=yes
101  | single |       | Sat 24 Jun 2023 12:16:45 AM CEST | root |  90.17 MiB | number  | Snapshot Update of #100 | important=yes
102  | single |       | Sun 25 Jun 2023 12:36:14 AM CEST | root |  99.01 MiB | number  | Snapshot Update of #101 | important=yes
103  | single |       | Mon 26 Jun 2023 12:51:22 AM CEST | root |  52.78 MiB | number  | Snapshot Update of #102 | important=yes
104  | single |       | Tue 27 Jun 2023 01:24:12 AM CEST | root |  58.21 MiB | number  | Snapshot Update of #103 | important=yes
105  | single |       | Wed 28 Jun 2023 12:45:24 AM CEST | root |  58.21 MiB | number  | Snapshot Update of #104 | important=yes
106  | single |       | Thu 29 Jun 2023 01:11:26 AM CEST | root |  92.40 MiB | number  | Snapshot Update of #105 | important=yes
107  | single |       | Fri 30 Jun 2023 01:35:27 AM CEST | root |  71.29 MiB | number  | Snapshot Update of #106 | important=yes
108  | single |       | Sat 01 Jul 2023 01:56:45 AM CEST | root |  95.39 MiB | number  | Snapshot Update of #107 | important=yes
109  | single |       | Tue 04 Jul 2023 01:45:40 AM CEST | root |  87.02 MiB | number  | Snapshot Update of #108 | important=yes
110  | single |       | Wed 05 Jul 2023 12:20:37 AM CEST | root |  87.05 MiB | number  | Snapshot Update of #109 | important=yes
111  | single |       | Fri 07 Jul 2023 12:50:54 AM CEST | root |  65.33 MiB | number  | Snapshot Update of #110 | important=yes
112  | single |       | Sat 08 Jul 2023 12:41:44 AM CEST | root |  50.43 MiB | number  | Snapshot Update of #111 | important=yes
113  | single |       | Sun 09 Jul 2023 12:49:31 AM CEST | root |  61.52 MiB | number  | Snapshot Update of #112 | important=yes
114  | single |       | Mon 10 Jul 2023 12:04:04 AM CEST | root |  63.74 MiB | number  | Snapshot Update of #113 | important=yes
115  | single |       | Tue 11 Jul 2023 01:23:13 AM CEST | root |  52.82 MiB | number  | Snapshot Update of #114 | important=yes
116  | single |       | Wed 12 Jul 2023 01:24:01 AM CEST | root |  62.85 MiB | number  | Snapshot Update of #115 | important=yes
117  | single |       | Fri 14 Jul 2023 01:35:05 AM CEST | root |  63.61 MiB | number  | Snapshot Update of #116 | important=yes
118  | single |       | Sat 15 Jul 2023 12:56:02 AM CEST | root |  63.11 MiB | number  | Snapshot Update of #117 | important=yes
119  | single |       | Sun 16 Jul 2023 12:12:42 AM CEST | root |  63.57 MiB | number  | Snapshot Update of #118 | important=yes
120  | single |       | Tue 18 Jul 2023 12:40:38 AM CEST | root |  64.20 MiB | number  | Snapshot Update of #119 | important=yes
121  | single |       | Wed 19 Jul 2023 12:26:17 AM CEST | root |  54.16 MiB | number  | Snapshot Update of #120 | important=yes
122  | single |       | Thu 20 Jul 2023 01:15:47 AM CEST | root |  67.05 MiB | number  | Snapshot Update of #121 | important=yes
123  | single |       | Wed 26 Jul 2023 01:22:19 AM CEST | root | 323.79 MiB | number  | Snapshot Update of #122 | important=yes
124  | single |       | Sun 30 Jul 2023 01:30:53 AM CEST | root | 100.80 MiB | number  | Snapshot Update of #123 | important=yes
125  | single |       | Mon 31 Jul 2023 01:43:32 AM CEST | root |  64.16 MiB | number  | Snapshot Update of #124 | important=yes
126  | single |       | Tue 01 Aug 2023 01:07:27 AM CEST | root |  64.05 MiB | number  | Snapshot Update of #125 | important=yes
127  | single |       | Wed 02 Aug 2023 01:08:00 AM CEST | root |  63.93 MiB | number  | Snapshot Update of #126 | important=yes
128  | single |       | Thu 03 Aug 2023 12:47:19 AM CEST | root |  64.81 MiB | number  | Snapshot Update of #127 | important=yes
129  | single |       | Fri 04 Aug 2023 01:18:48 AM CEST | root |  64.37 MiB | number  | Snapshot Update of #128 | important=yes
130  | single |       | Sat 05 Aug 2023 01:08:44 AM CEST | root |  94.40 MiB | number  | Snapshot Update of #129 | important=yes
131  | single |       | Sun 06 Aug 2023 01:13:44 AM CEST | root | 103.21 MiB | number  | Snapshot Update of #130 | important=yes
132  | single |       | Mon 07 Aug 2023 01:21:53 AM CEST | root |  65.65 MiB | number  | Snapshot Update of #131 | important=yes
133  | single |       | Tue 08 Aug 2023 12:50:46 AM CEST | root |  79.47 MiB | number  | Snapshot Update of #132 | important=yes
134  | single |       | Wed 09 Aug 2023 01:03:59 AM CEST | root |  55.95 MiB | number  | Snapshot Update of #133 | important=yes
135  | single |       | Thu 10 Aug 2023 01:48:12 AM CEST | root |  65.30 MiB |         | Snapshot Update of #134 | important=yes
136  | single |       | Fri 11 Aug 2023 12:50:10 AM CEST | root |  67.61 MiB | number  | Snapshot Update of #135 | important=yes
137  | single |       | Sat 12 Aug 2023 01:33:54 AM CEST | root |  65.14 MiB | number  | Snapshot Update of #136 | important=yes
138* | single |       | Sun 13 Aug 2023 01:14:34 AM CEST | root |  11.99 MiB |         | Snapshot Update of #137 |              
139  | single |       | Mon 14 Aug 2023 01:11:02 AM CEST | root |   1.27 GiB | number  | Snapshot Update of #138 |              
140  | single |       | Tue 15 Aug 2023 12:12:37 AM CEST | root |   1.27 GiB |         | Snapshot Update of #138 |              
141  | single |       | Tue 22 Aug 2023 02:56:57 PM CEST | root |  13.71 MiB |         | Snapshot Update of #138 |              
142  | single |       | Tue 22 Aug 2023 03:36:44 PM CEST | root |  13.71 MiB |         | Snapshot Update of #138 |              

[realSConway]

Solved: snapshot 62 contained nested subvolume: 62/snapshot/var/lib/machines/
set snapshot ro false
btrfs property set 62/snapshot ro false
removed nested subvolme
btrfs subvolume delete 62/snapshot/var/lib/machines/
tested to see if I could remove snapshot 62
btrfs subvolume delete /.snapshots/62/snapshot
Then started snapper-cleanup.service
systemctl start snapper-cleanup.service
Now all snapshots are cleaned:

btrfs subvolume list /
ID 256 gen 29 top level 5 path @
ID 257 gen 9661 top level 256 path @/usr/local
ID 258 gen 9661 top level 256 path @/srv
ID 259 gen 9695 top level 256 path @/root
ID 260 gen 9661 top level 256 path @/opt
ID 261 gen 9661 top level 256 path @/home
ID 262 gen 9661 top level 256 path @/boot/writable
ID 263 gen 9661 top level 256 path @/boot/grub2/x86_64-efi
ID 264 gen 9661 top level 256 path @/boot/grub2/i386-pc
ID 265 gen 9693 top level 256 path @/.snapshots
ID 433 gen 7758 top level 265 path @/.snapshots/127/snapshot
ID 434 gen 7781 top level 265 path @/.snapshots/128/snapshot
ID 435 gen 7819 top level 265 path @/.snapshots/129/snapshot
ID 436 gen 7841 top level 265 path @/.snapshots/130/snapshot
ID 437 gen 7893 top level 265 path @/.snapshots/131/snapshot
ID 438 gen 7933 top level 265 path @/.snapshots/132/snapshot
ID 439 gen 7992 top level 265 path @/.snapshots/133/snapshot
ID 440 gen 8041 top level 265 path @/.snapshots/134/snapshot
ID 441 gen 8060 top level 265 path @/.snapshots/135/snapshot
ID 442 gen 8088 top level 265 path @/.snapshots/136/snapshot
ID 443 gen 8140 top level 265 path @/.snapshots/137/snapshot
ID 444 gen 9653 top level 265 path @/.snapshots/138/snapshot
ID 445 gen 8803 top level 265 path @/.snapshots/139/snapshot
ID 446 gen 9482 top level 265 path @/.snapshots/140/snapshot
ID 447 gen 9555 top level 265 path @/.snapshots/141/snapshot
ID 448 gen 9590 top level 265 path @/.snapshots/142/snapshot
ID 449 gen 9657 top level 265 path @/.snapshots/143/snapshot