Latency Spikes in realtime threads while IO on ZFS

[poelzi]

System information

Type	Version/Name
Distribution Name	NixOS
Distribution Version	21.11
Kernel Version	5.10.94
Architecture	x64
OpenZFS Version	zfs-2.1.2-1

Describe the problem you're observing

I'm using mixxx with jackd. I had used this machine mit lvm+ext4 for 5 years and switched recently to zfs.
I notice a lot of underruns in the realtime thread now, especially when the system performs IO that did not happen with stock kernel and ext4.

Describe how to reproduce the problem

Doing zfs IO while realtime thread does work.

Include any warning/errors/backtraces from the system logs

[szubersk]

@poelzi, this issue report is incomplete. Would you mind providing more details?

[IvanVolosyuk]

I was using qemu/kvm/vfio with realtime priority and ZFS caused audio glitches and video stuttering. I was trying isol_cpus which ZFS ignored, so I was maintaining my own version of CPU isolation in ZFS, but it was leading to system deadlocks in some situations. In the end I have found out that preemptive kernel solves the latency spikes I observed with ZFS. I am using it successfully for more than a year.

[poelzi]

@IvanVolosyuk I switched to a custom kernel with full preemtive kernel and it is such a difference. Running ZFS on voluntary preemtive kernel on a desktop system is a really bad expericene.
Even with 5.16.20 and zfs-2.1.4-1

I will add a NixOS ticket so the combination of zfs and preemtive get solved, but I think it would be good on the long run if ZFS would behave like other FS on PREEMPT_VOLUNTARY and not cause such latency spikes.

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

[IvanVolosyuk]

still relevant

[IvanVolosyuk]

Actually, what would be the way to debug / profile the ZFS code to find out in which stacks ZFS takes a long time in kernel threads between calls to schedule() or similar situations?