[Ubuntu 20.04.2 LTS] Sanoid issue for 'blkio.weight', 'oom_kill', and ZFS blocked I/O

[sonaro]

TLDR;
Ubuntu was upgraded to latest kernel recently. Since then, VM had spawned a huge amount of processes that never cleaned up (still investigating), ZFS disk started reporting "Device or resource busy" during shutdown. Any thoughts on the errors (bolded in logs) from sanoid, or others experiencing similar issues?

Summary of errors:

• [zfs had lots of blocking processes that took SIGKILL ~10mins to clear on shutdown]
• Aug 01 08:14:02 data2 systemd[1]: sanoid.service: Failed to set 'blkio.weight' attribute on '/system.slice/sanoid.service' to '500': No such file or directory
• Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory

System info:

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"

uname -a
Linux data2 5.4.0-80-generic #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

zfs version ~
zfs-0.8.3-1ubuntu12.11
zfs-kmod-0.8.3-1ubuntu12.10

sanoid --version
/usr/sbin/sanoid version 2.0.3
(Getopt::Long::GetOptions version 2.50; Perl version 5.30.0)

===== Syslog =====

[480536.528246] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[480536.562827] systemd-shutdown[1]: Sending SIGKILL to PID 3439360 (zfs).
[480536.567122] systemd-shutdown[1]: Sending SIGKILL to PID 3439791 (zfs).
[480536.572498] systemd-shutdown[1]: Sending SIGKILL to PID 3439997 (zfs).
[480536.577079] systemd-shutdown[1]: Sending SIGKILL to PID 3440115 (zfs).
[480536.581165] systemd-shutdown[1]: Sending SIGKILL to PID 3440251 (zfs).
[480536.585546] systemd-shutdown[1]: Sending SIGKILL to PID 3440566 (zfs).
[480536.589871] systemd-shutdown[1]: Sending SIGKILL to PID 3440747 (zfs).
[480536.593844] systemd-shutdown[1]: Sending SIGKILL to PID 3440866 (zfs).
[480536.597691] systemd-shutdown[1]: Sending SIGKILL to PID 3441002 (zfs).
[480536.601177] systemd-shutdown[1]: Sending SIGKILL to PID 3441276 (zfs).
[480536.604095] systemd-shutdown[1]: Sending SIGKILL to PID 3441454 (zfs).
[480536.607883] systemd-shutdown[1]: Sending SIGKILL to PID 3441571 (zfs).
[480536.611376] systemd-shutdown[1]: Sending SIGKILL to PID 3441718 (zfs).
[480536.614112] systemd-shutdown[1]: Sending SIGKILL to PID 3442003 (zfs).
[480536.617430] systemd-shutdown[1]: Sending SIGKILL to PID 3442166 (zfs).
[480536.619771] systemd-shutdown[1]: Sending SIGKILL to PID 3442286 (zfs).
[480536.622179] systemd-shutdown[1]: Sending SIGKILL to PID 3442414 (zfs).
[480536.624539] systemd-shutdown[1]: Sending SIGKILL to PID 3442683 (zfs).
[480536.626523] systemd-shutdown[1]: Sending SIGKILL to PID 3442851 (zfs).
[480536.628995] systemd-shutdown[1]: Sending SIGKILL to PID 3442970 (zfs).
[480536.631624] systemd-shutdown[1]: Sending SIGKILL to PID 3443108 (zfs).
[480536.634699] systemd-shutdown[1]: Sending SIGKILL to PID 3443414 (zfs).
[480536.637480] systemd-shutdown[1]: Sending SIGKILL to PID 3443559 (zfs).
[480536.639630] systemd-shutdown[1]: Sending SIGKILL to PID 3443678 (zfs).
[480536.642392] systemd-shutdown[1]: Sending SIGKILL to PID 3443871 (zfs).
[480536.646808] systemd-shutdown[1]: Sending SIGKILL to PID 3444157 (zfs).
[480536.651874] systemd-shutdown[1]: Sending SIGKILL to PID 3444344 (zfs).
[480536.655276] systemd-shutdown[1]: Sending SIGKILL to PID 3444462 (zfs).
[480536.657785] systemd-shutdown[1]: Sending SIGKILL to PID 3444712 (zfs).
[480536.660888] systemd-shutdown[1]: Sending SIGKILL to PID 3445157 (zfs).
[480536.662899] systemd-shutdown[1]: Sending SIGKILL to PID 3445331 (zfs).
[480536.665620] systemd-shutdown[1]: Sending SIGKILL to PID 3445434 (zfs).
[480536.667922] systemd-shutdown[1]: Sending SIGKILL to PID 3445524 (zfs).
[480536.670160] systemd-shutdown[1]: Sending SIGKILL to PID 3445813 (zfs).
[480536.672224] systemd-shutdown[1]: Sending SIGKILL to PID 3445951 (zfs).
[480536.674219] systemd-shutdown[1]: Sending SIGKILL to PID 3446117 (zfs).
[480536.676132] systemd-shutdown[1]: Sending SIGKILL to PID 3446208 (zfs).
[480536.678185] systemd-shutdown[1]: Sending SIGKILL to PID 3446472 (zfs).
[480536.681074] systemd-shutdown[1]: Sending SIGKILL to PID 3446596 (zfs).
[480536.683307] systemd-shutdown[1]: Sending SIGKILL to PID 3446698 (zfs).
[480536.685652] systemd-shutdown[1]: Sending SIGKILL to PID 3446788 (zfs).
[480536.687800] systemd-shutdown[1]: Sending SIGKILL to PID 3447068 (zfs).
[480536.689824] systemd-shutdown[1]: Sending SIGKILL to PID 3447199 (zfs).
[480536.691421] systemd-shutdown[1]: Sending SIGKILL to PID 3447299 (zfs).
[480536.693465] systemd-shutdown[1]: Sending SIGKILL to PID 3447418 (zfs).
[480536.695307] systemd-shutdown[1]: Sending SIGKILL to PID 3447792 (zfs).
[480536.700693] systemd-shutdown[1]: Sending SIGKILL to PID 3447927 (zfs).
[480536.702479] systemd-shutdown[1]: Sending SIGKILL to PID 3448015 (zfs).
[480536.704501] systemd-shutdown[1]: Sending SIGKILL to PID 3448121 (zfs).
[480536.705960] systemd-shutdown[1]: Sending SIGKILL to PID 3461717 (zfs).
[480536.707364] systemd-shutdown[1]: Sending SIGKILL to PID 3461831 (zfs).
[480536.709516] systemd-shutdown[1]: Sending SIGKILL to PID 3461923 (zfs).
[480536.711551] systemd-shutdown[1]: Sending SIGKILL to PID 3462010 (zfs).
[480536.713672] systemd-shutdown[1]: Sending SIGKILL to PID 3462293 (zfs).
[480536.715672] systemd-shutdown[1]: Sending SIGKILL to PID 3462426 (zfs).
[480536.717569] systemd-shutdown[1]: Sending SIGKILL to PID 3462527 (zfs).
[480536.719543] systemd-shutdown[1]: Sending SIGKILL to PID 3462616 (zfs).
[480536.721531] systemd-shutdown[1]: Sending SIGKILL to PID 3462917 (zfs).
[480536.723442] systemd-shutdown[1]: Sending SIGKILL to PID 3463041 (zfs).
[480536.725426] systemd-shutdown[1]: Sending SIGKILL to PID 3463140 (zfs).
[480536.727270] systemd-shutdown[1]: Sending SIGKILL to PID 3463251 (zfs).
[480536.729255] systemd-shutdown[1]: Sending SIGKILL to PID 3463573 (zfs).
[480536.731159] systemd-shutdown[1]: Sending SIGKILL to PID 3463712 (zfs).
[480536.733238] systemd-shutdown[1]: Sending SIGKILL to PID 3463799 (zfs).
[480536.735137] systemd-shutdown[1]: Sending SIGKILL to PID 3463888 (zfs).
[480536.737220] systemd-shutdown[1]: Sending SIGKILL to PID 3464162 (zfs).
[480536.739146] systemd-shutdown[1]: Sending SIGKILL to PID 3464289 (zfs).
[480536.741125] systemd-shutdown[1]: Sending SIGKILL to PID 3464393 (zfs).
[480536.742969] systemd-shutdown[1]: Sending SIGKILL to PID 3464646 (zfs).
[480536.744946] systemd-shutdown[1]: Sending SIGKILL to PID 3464775 (zfs).
[480536.746761] systemd-shutdown[1]: Sending SIGKILL to PID 3464803 (updatedb.mlocat).
[480536.748945] systemd-shutdown[1]: Sending SIGKILL to PID 3465205 (zfs).
[480536.750818] systemd-shutdown[1]: Sending SIGKILL to PID 3465380 (zfs).
[480536.752887] systemd-shutdown[1]: Sending SIGKILL to PID 3465668 (zfs).
[480536.754802] systemd-shutdown[1]: Sending SIGKILL to PID 3465802 (zfs).
[480536.757585] systemd-shutdown[1]: Sending SIGKILL to PID 3465897 (zfs).
[480536.759494] systemd-shutdown[1]: Sending SIGKILL to PID 3471588 (zfs).
[480536.761698] systemd-shutdown[1]: Sending SIGKILL to PID 3481649 (umount).
[480536.763556] systemd-shutdown[1]: Sending SIGKILL to PID 3481650 (umount).
[480536.765375] systemd-shutdown[1]: Sending SIGKILL to PID 3481651 (umount).
[480536.767298] systemd-shutdown[1]: Sending SIGKILL to PID 3481652 (umount).
[480536.769291] systemd-shutdown[1]: Sending SIGKILL to PID 3481653 (umount).
[480536.771206] systemd-shutdown[1]: Sending SIGKILL to PID 3481654 (umount).
[480536.773481] systemd-shutdown[1]: Sending SIGKILL to PID 3481655 (umount).
[480536.775658] systemd-shutdown[1]: Sending SIGKILL to PID 3481656 (umount).
[480536.777871] systemd-shutdown[1]: Sending SIGKILL to PID 3481658 (umount).
[480536.780060] systemd-shutdown[1]: Sending SIGKILL to PID 3481661 (umount).
[480536.782299] systemd-shutdown[1]: Sending SIGKILL to PID 3481704 ((sd-sync)).
[480546.789927] systemd-shutdown[1]: Waiting for process: zfs, zfs, zfs, zfs, zfs, umount, zfs, zfs, zfs, zfs, umount, zfs, zfs, zfs, zfs, zfs, umount, updatedb.mlocat, zfs, zfs, zfs, umount, zfs, zfs, zfs, zfs, umount, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, umount, zfs, zfs, zfs, zfs, zfs, (sd-sync), zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, umount, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, umount, zfs, zfs, zfs, zfs, zfs, zfs, zfs, zfs, umount, zfs, zfs, zfs, zfs, umount, zfs
[480626.814246] systemd-shutdown[1]: Unmounting file systems.
[480626.820752] systemd-shutdown[1]: Successfully forked off '(sd-remount)' as PID 3481710.
[480626.823959] [3481710]: Remounting '/homezp1/2005/.zfs/snapshot/autosnap_2021-02-21_23:30:34_weekly' read-only in with options 'xattr,noacl'.
[480626.829691] systemd-shutdown[1]: Successfully forked off '(sd-umount)' as PID 3481711.
[480626.832181] [3481711]: Unmounting '/homezp1/2005/.zfs/snapshot/autosnap_2021-02-21_23:30:34_weekly'.
[480626.834657] [3481711]: Failed to unmount /homezp1/2005/.zfs/snapshot/autosnap_2021-02-21_23:30:34_weekly: Device or resource busy
[480626.838009] systemd-shutdown[1]: Unmounting '/homezp1/2005/.zfs/snapshot/autosnap_2021-02-21_23:30:34_weekly' failed abnormally, child process 3481711 aborted or exited non-zero.
[480626.843746] systemd-shutdown[1]: Successfully forked off '(sd-remount)' as PID 3481712.
[480626.847108] [3481712]: Remounting '/homezp1/2005' read-only in with options 'xattr,noacl'.
[480626.850835] systemd-shutdown[1]: Successfully forked off '(sd-umount)' as PID 3481713.
[480626.853543] [3481713]: Unmounting '/homezp1/2005'.
[480626.855493] [3481713]: Failed to unmount /homezp1/2005: Device or resource busy
[480626.858129] systemd-shutdown[1]: Unmounting '/homezp1/2005' failed abnormally, child process 3481713 aborted or exited non-zero.
[480626.862416] systemd-shutdown[1]: Successfully forked off '(sd-remount)' as PID 3481714.
[480626.865362] [3481714]: Remounting '/homezp1' read-only in with options 'xattr,noacl'.
[480626.868825] systemd-shutdown[1]: Successfully forked off '(sd-umount)' as PID 3481715.
[480626.871710] [3481715]: Unmounting '/homezp1'.
[480626.873747] [3481715]: Failed to unmount /homezp1: Device or resource busy
[480626.876284] systemd-shutdown[1]: Unmounting '/homezp1' failed abnormally, child process 3481715 aborted or exited non-zero.
[...]
===== End Syslog =====

===== Systemd Log =====
data2> journalctl -u sanoid.service -u sanoid-prune.service -n1000 -b -f
-- Logs begin at Sat 2021-07-24 20:47:15 EDT. --
Aug 01 08:14:02 data2 systemd[1]: sanoid.service: Trying to enqueue job sanoid.service/start/replace
Aug 01 08:14:02 data2 systemd[1]: sanoid-prune.service: Installed new job sanoid-prune.service/start as 420
Aug 01 08:14:02 data2 systemd[1]: sanoid.service: Installed new job sanoid.service/start as 350
Aug 01 08:14:02 data2 systemd[1]: sanoid.service: Enqueued job sanoid.service/start as 350
Aug 01 08:14:02 data2 systemd[1]: sanoid.service: ConditionFileNotEmpty=/etc/sanoid/sanoid.conf succeeded.
Aug 01 08:14:02 data2 systemd[1]: sanoid.service: Failed to set 'blkio.weight' attribute on '/system.slice/sanoid.service' to '500': No such file or directory
Aug 01 08:14:02 data2 systemd[1]: sanoid.service: Passing 0 fds to service
Aug 01 08:14:02 data2 systemd[1]: sanoid.service: About to execute: /usr/sbin/sanoid --take-snapshots --verbose
Aug 01 08:14:02 data2 systemd[1]: sanoid.service: Forked /usr/sbin/sanoid as 4529
Aug 01 08:14:02 data2 systemd[1]: sanoid.service: Changed dead -> start
Aug 01 08:14:02 data2 systemd[1]: Starting Snapshot ZFS filesystems...
Aug 01 08:14:02 data2 systemd[4529]: sanoid.service: Executing: /usr/sbin/sanoid --take-snapshots --verbose
Aug 01 08:14:05 data2 sanoid[4529]: INFO: cache expired - updating from zfs list.
Aug 01 08:14:52 data2 systemd[1]: sanoid-prune.service: Reinstalled deserialized job sanoid-prune.service/start as 420
Aug 01 08:14:52 data2 systemd[1]: sanoid.service: Reinstalled deserialized job sanoid.service/start as 350
Aug 01 08:14:52 data2 systemd[1]: sanoid.service: Changed dead -> start
Aug 01 08:14:52 data2 systemd[1]: sanoid.service: Failed to set 'blkio.weight' attribute on '/system.slice/sanoid.service' to '500': No such file or directory
Aug 01 08:14:57 data2 systemd[1]: sanoid-prune.service: Reinstalled deserialized job sanoid-prune.service/start as 420
Aug 01 08:14:57 data2 systemd[1]: sanoid.service: Reinstalled deserialized job sanoid.service/start as 350
Aug 01 08:14:57 data2 systemd[1]: sanoid.service: Changed dead -> start
Aug 01 08:14:57 data2 systemd[1]: sanoid.service: Failed to set 'blkio.weight' attribute on '/system.slice/sanoid.service' to '500': No such file or directory
Aug 01 08:17:27 data2 systemd[1]: sanoid-prune.service: Reinstalled deserialized job sanoid-prune.service/start as 420
Aug 01 08:17:27 data2 systemd[1]: sanoid.service: Reinstalled deserialized job sanoid.service/start as 350
Aug 01 08:17:27 data2 systemd[1]: sanoid.service: Changed dead -> start
Aug 01 08:17:27 data2 systemd[1]: sanoid.service: Failed to set 'blkio.weight' attribute on '/system.slice/sanoid.service' to '500': No such file or directory
Aug 01 08:17:29 data2 systemd[1]: sanoid-prune.service: Reinstalled deserialized job sanoid-prune.service/start as 420
Aug 01 08:17:29 data2 systemd[1]: sanoid.service: Reinstalled deserialized job sanoid.service/start as 350
Aug 01 08:17:29 data2 systemd[1]: sanoid.service: Changed dead -> start
Aug 01 08:17:30 data2 systemd[1]: sanoid.service: Failed to set 'blkio.weight' attribute on '/system.slice/sanoid.service' to '500': No such file or directory
Aug 01 08:17:31 data2 systemd[1]: sanoid-prune.service: Reinstalled deserialized job sanoid-prune.service/start as 420
Aug 01 08:17:31 data2 systemd[1]: sanoid.service: Reinstalled deserialized job sanoid.service/start as 350
Aug 01 08:17:31 data2 systemd[1]: sanoid.service: Changed dead -> start
Aug 01 08:17:32 data2 systemd[1]: sanoid.service: Failed to set 'blkio.weight' attribute on '/system.slice/sanoid.service' to '500': No such file or directory
Aug 01 08:17:34 data2 systemd[1]: sanoid-prune.service: Reinstalled deserialized job sanoid-prune.service/start as 420
Aug 01 08:17:34 data2 systemd[1]: sanoid.service: Reinstalled deserialized job sanoid.service/start as 350
Aug 01 08:17:34 data2 systemd[1]: sanoid.service: Changed dead -> start
Aug 01 08:17:34 data2 systemd[1]: sanoid.service: Failed to set 'blkio.weight' attribute on '/system.slice/sanoid.service' to '500': No such file or directory
Aug 01 08:17:40 data2 systemd[1]: sanoid-prune.service: Reinstalled deserialized job sanoid-prune.service/start as 420
Aug 01 08:17:40 data2 systemd[1]: sanoid.service: Reinstalled deserialized job sanoid.service/start as 350
Aug 01 08:17:40 data2 systemd[1]: sanoid.service: Changed dead -> start
Aug 01 08:17:41 data2 systemd[1]: sanoid.service: Failed to set 'blkio.weight' attribute on '/system.slice/sanoid.service' to '500': No such file or directory
Aug 01 08:21:34 data2 sanoid[4529]: INFO: taking snapshots...
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Child 4529 belongs to sanoid.service.
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Main process exited, code=exited, status=0/SUCCESS
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Succeeded.
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Service will not restart (restart setting)
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Changed start -> dead
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Job 350 sanoid.service/start finished, result=done
Aug 01 08:21:34 data2 systemd[1]: Finished Snapshot ZFS filesystems.
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Control group is empty.
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Trying to enqueue job sanoid.service/start/replace
Aug 01 08:21:34 data2 systemd[1]: sanoid-prune.service: Merged sanoid-prune.service/start into installed job sanoid-prune.service/start as 420
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Installed new job sanoid.service/start as 855
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Enqueued job sanoid.service/start as 855
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: ConditionFileNotEmpty=/etc/sanoid/sanoid.conf succeeded.
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Failed to set 'blkio.weight' attribute on '/system.slice/sanoid.service' to '500': No such file or directory
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Passing 0 fds to service
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: About to execute: /usr/sbin/sanoid --take-snapshots --verbose
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Forked /usr/sbin/sanoid as 16253
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Changed dead -> start
Aug 01 08:21:34 data2 systemd[1]: Starting Snapshot ZFS filesystems...
Aug 01 08:21:34 data2 systemd[16253]: sanoid.service: Executing: /usr/sbin/sanoid --take-snapshots --verbose
Aug 01 08:21:34 data2 sanoid[16253]: INFO: taking snapshots...
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Child 16253 belongs to sanoid.service.
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Main process exited, code=exited, status=0/SUCCESS
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Succeeded.
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Service will not restart (restart setting)
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Changed start -> dead
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Job 855 sanoid.service/start finished, result=done
Aug 01 08:21:34 data2 systemd[1]: Finished Snapshot ZFS filesystems.
Aug 01 08:21:34 data2 systemd[1]: sanoid.service: Control group is empty.
Aug 01 08:21:34 data2 systemd[1]: sanoid-prune.service: ConditionFileNotEmpty=/etc/sanoid/sanoid.conf succeeded.
Aug 01 08:21:34 data2 systemd[1]: sanoid-prune.service: Failed to set 'blkio.weight' attribute on '/system.slice/sanoid-prune.service' to '500': No such file or directory
Aug 01 08:21:34 data2 systemd[1]: sanoid-prune.service: Passing 0 fds to service
Aug 01 08:21:34 data2 systemd[1]: sanoid-prune.service: About to execute: /usr/sbin/sanoid --prune-snapshots --verbose
Aug 01 08:21:34 data2 systemd[1]: sanoid-prune.service: Forked /usr/sbin/sanoid as 16279
Aug 01 08:21:34 data2 systemd[1]: sanoid-prune.service: Changed dead -> start
Aug 01 08:21:34 data2 systemd[1]: Starting Prune ZFS snapshots...
Aug 01 08:21:34 data2 systemd[16279]: sanoid-prune.service: Executing: /usr/sbin/sanoid --prune-snapshots --verbose
Aug 01 08:21:35 data2 sanoid[16279]: INFO: pruning snapshots...
Aug 01 08:21:35 data2 sanoid[16279]: INFO: pruning bk2pool1/2003@autosnap_2021-07-27_00:00:32_daily ...
Aug 01 08:21:41 data2 sanoid[16279]: INFO: removing destroyed snapshots from cache.
Aug 01 08:21:48 data2 sanoid[16279]: INFO: pruning bk2pool1/2005@autosnap_2021-07-29_17:00:32_hourly ...
Aug 01 08:21:50 data2 sanoid[16279]: INFO: deferring pruning of bk2pool1/2005@autosnap_2021-07-30_01:00:29_hourly - bk2pool1/2005 is currently in zfs send or receive.
[...]
Aug 01 08:21:55 data2 systemd[1]: sanoid-prune.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
Aug 01 08:21:55 data2 systemd[1]: sanoid-prune.service: Child 16279 belongs to sanoid-prune.service.
Aug 01 08:21:55 data2 systemd[1]: sanoid-prune.service: Main process exited, code=exited, status=0/SUCCESS
Aug 01 08:21:55 data2 systemd[1]: sanoid-prune.service: Succeeded.
Aug 01 08:21:55 data2 systemd[1]: sanoid-prune.service: Service will not restart (restart setting)
Aug 01 08:21:55 data2 systemd[1]: sanoid-prune.service: Changed start -> dead
Aug 01 08:21:55 data2 systemd[1]: sanoid-prune.service: Job 420 sanoid-prune.service/start finished, result=done
Aug 01 08:21:55 data2 systemd[1]: Finished Prune ZFS snapshots.
Aug 01 08:21:55 data2 systemd[1]: sanoid-prune.service: Control group is empty.
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Trying to enqueue job sanoid.service/start/replace
Aug 01 08:30:02 data2 systemd[1]: sanoid-prune.service: Installed new job sanoid-prune.service/start as 1000
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Installed new job sanoid.service/start as 930
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Enqueued job sanoid.service/start as 930
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: ConditionFileNotEmpty=/etc/sanoid/sanoid.conf succeeded.
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Failed to set 'blkio.weight' attribute on '/system.slice/sanoid.service' to '500': No such file or directory
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Passing 0 fds to service
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: About to execute: /usr/sbin/sanoid --take-snapshots --verbose
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Forked /usr/sbin/sanoid as 28554
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Changed dead -> start
Aug 01 08:30:02 data2 systemd[1]: Starting Snapshot ZFS filesystems...
Aug 01 08:30:02 data2 systemd[28554]: sanoid.service: Executing: /usr/sbin/sanoid --take-snapshots --verbose
Aug 01 08:30:02 data2 sanoid[28554]: INFO: taking snapshots...
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Child 28554 belongs to sanoid.service.
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Main process exited, code=exited, status=0/SUCCESS
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Succeeded.
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Service will not restart (restart setting)
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Changed start -> dead
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Job 930 sanoid.service/start finished, result=done
Aug 01 08:30:02 data2 systemd[1]: Finished Snapshot ZFS filesystems.
Aug 01 08:30:02 data2 systemd[1]: sanoid.service: Control group is empty.
Aug 01 08:30:02 data2 systemd[1]: sanoid-prune.service: ConditionFileNotEmpty=/etc/sanoid/sanoid.conf succeeded.
Aug 01 08:30:03 data2 systemd[1]: sanoid-prune.service: Failed to set 'blkio.weight' attribute on '/system.slice/sanoid-prune.service' to '500': No such file or directory
Aug 01 08:30:03 data2 systemd[1]: sanoid-prune.service: Passing 0 fds to service
Aug 01 08:30:03 data2 systemd[1]: sanoid-prune.service: About to execute: /usr/sbin/sanoid --prune-snapshots --verbose
Aug 01 08:30:03 data2 systemd[1]: sanoid-prune.service: Forked /usr/sbin/sanoid as 28565
Aug 01 08:30:03 data2 systemd[1]: sanoid-prune.service: Changed dead -> start
Aug 01 08:30:03 data2 systemd[1]: Starting Prune ZFS snapshots...
Aug 01 08:30:03 data2 systemd[28565]: sanoid-prune.service: Executing: /usr/sbin/sanoid --prune-snapshots --verbose
Aug 01 08:30:03 data2 sanoid[28565]: INFO: pruning snapshots...
Aug 01 08:30:03 data2 sanoid[28565]: INFO: pruning bk2pool1/2005@autosnap_2021-05-03_00:00:29_daily ...
Aug 01 08:30:03 data2 sanoid[28565]: INFO: removing destroyed snapshots from cache.
[...]
Aug 01 08:30:03 data2 sanoid[28565]: INFO: pruning bk2pool1/2005@autosnap_2021-07-30_01:00:29_hourly ...
Aug 01 08:30:06 data2 sanoid[28565]: INFO: removing destroyed snapshots from cache.
Aug 01 08:30:06 data2 systemd[1]: sanoid-prune.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
Aug 01 08:30:06 data2 systemd[1]: sanoid-prune.service: Child 28565 belongs to sanoid-prune.service.
Aug 01 08:30:06 data2 systemd[1]: sanoid-prune.service: Main process exited, code=exited, status=0/SUCCESS
Aug 01 08:30:06 data2 systemd[1]: sanoid-prune.service: Succeeded.
Aug 01 08:30:06 data2 systemd[1]: sanoid-prune.service: Service will not restart (restart setting)
Aug 01 08:30:06 data2 systemd[1]: sanoid-prune.service: Changed start -> dead
Aug 01 08:30:06 data2 systemd[1]: sanoid-prune.service: Job 1000 sanoid-prune.service/start finished, result=done
Aug 01 08:30:06 data2 systemd[1]: Finished Prune ZFS snapshots.
Aug 01 08:30:06 data2 systemd[1]: sanoid-prune.service: Control group is empty.
===== End Systemd Log =====

[phreaker0]

not a sanoid issue