feat(I/O): change default SSD and HDD schedulers to remove system lag

[kevinpbaker]

Problem

I was running into some issues where I thought Omarchy was slow/laggy. This didn't make sense, so I decided to troubleshoot. I discovered that heavy IO tasks will kill UI responsiveness (if you're not running NVMe drives). I'll try to explain why below.

All data drives (SSDs and HHDs) currently default to using none as their selected scheduler. Using none can lead to poor interactive latency on desktop systems. Background workloads such as large file copies, indexing, or builds can dominate I/O submission, causing foreground tasks like UI redraws, audio playback, or terminal input to stall. While throughput remains high, the system feels unresponsive because the kernel is not prioritizing latency-sensitive processes.

Therefore, I propose a change is put in place to switch all SSDs and HDDs to default to BFQ (Budget Fair Queueing) scheduling. It addresses the problem stated above by introducing per-process I/O fairness and explicit latency control. BFQ ensures that interactive applications receive timely access to storage even under heavy background I/O, which significantly improves perceived system responsiveness. So there is slightly more scheduling overhead in exchange for a smoother, more predictable user experience.

I had large downloads running in the background, and kept getting "program is unresponsive" pop ups, and my mouse would freeze up, or typing input would lag. I'd hate for someone to think Linux is slow/laggy, just because the wrong IO schedulers are being used for their data drive types. I think it's highly likely that the average person will install Omarchy on an SSD or HDD, because they don't want to nuke their daily driver OS (until they see how epic Omarchy is), which is likely on their NVMe. So they'll look for an old SSD or HDD to install Omarchy onto.

Solution

Add a new shell script to boot process (not sure where? perhaps under the hardware folder?) that creates a new udev rule for automatically applying BFQ to SSDs and HDDs. These rules are stored in a file under /usr/lib/udev/rules.d (for vendor / distro policies).

The file would be named 60-auto-io-scheduler.rules. 60 is the rule load priority where udev executes rules in lexical order. I was reading that rules 10-50 are used for kernal and drivers, while 60 is the sweet spot for hardware policies.

The rule inside the file would look like this:

# Automatically assign I/O schedulers based on device type.
# These defaults are tuned for desktop and interactive workloads.

# NVMe devices default to the 'none' scheduler.
# - NVMe controllers implement their own deep, multi-queue scheduling in hardware.
# - Avoids redundant software scheduling and minimizes CPU overhead.
ACTION=="add|change", KERNEL=="nvme*", SUBSYSTEM=="block", ATTR{queue/scheduler}="none"

# SATA SSDs use the BFQ scheduler.
# - BFQ provides per-process I/O budgeting and prioritizes latency-sensitive tasks.
# - Improves desktop responsiveness during heavy background I/O.
ACTION=="add|change", KERNEL=="sd*", SUBSYSTEM=="block", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="bfq"

# Rotational HDDs use the BFQ scheduler.
# - HDDs benefit significantly from request fairness and seek-aware scheduling.
# - Prevents large sequential workloads from monopolizing the disk.
ACTION=="add|change", KERNEL=="sd*", SUBSYSTEM=="block", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"

Users can implement an override to the rule inside /etc/udev/rules.d/ if they want to use a different scheduler.

TL;DR

Not using a software scheduler for IO causes significant system lag when spamming a drive with work, which can render UI as frozen, or unresponsive. Therefore, add appropriate schedulers for specific drive types to increase perceived system performance.

[kevinpbaker]

I'd be happy to implement this and make a PR, but would prefer to get confirmation from a project owner or contributor before doing so. If this gets a green light, then I'd like to be educated on where the script should live. After parsing through the file structure of the repo, I would guess install/config/hardware, but I'm not positive.

[rudahssilva]

ive tried that with no luck, still the same, wa around 40 to 80%. Im using nvme drive

[kevinpbaker]

Note that the solution I proposed doesn't alter the scheduling for NVMe drives. It only adds bfq to SATA SSDs and HDDs. So to confirm: Did you alter the above code to use bfq scheduling for your NVMe drive, and the system still freezes?

[rudahssilva]

Yes i changed ir to btq with no luck , still the samen

[drwatson85]

I've been running into this problem pretty heavily.
Whenever there is any load on my SSD, my system is unusable. It basically freezes completely until i.e. Steam is done patching a game, etc.

[rudahssilva]

im having the same issue here, other distros dont do that, ive tride in arch with no issues, but in Omarhcy any internet download freeze the system, i work with video editing andc cant even export a video to the internal nvme , where the system are too. And the problema is that my laptop has just two nvme slot, nothing else. Can someone please help me?

[nickjj]

[kevinpbaker]

Hmm. It could be related. Your disk writes might be blocking your network, and then the packets might be lost (this is an assumption, don't know if that's true). If you add the 60-auto-io-scheduler.rules file to /etc/udev/rules.d/, and paste the code in from the original post, does it fix your problem?

[nickjj]

I haven't tested it yet because without researching it more since I don't know about the side effects this type of change may introduce. I don't think your intent is malicious, I just didn't want to blindly start modifying system level parameters without knowing why or how they work.

But with that said, coming from Windows with the same hardware I am seeing all sorts of "Linux is laggy" situations:

• Typing into Neovim is very slow with all terminals, especially with multiple splits on a 4k monitor
• I had to tweak the power settings on my SATA SSD, otherwise my whole machine would lock up for 30 seconds on a regular basis
• Games generally feel less smooth than Windows, lots of jittering
• I lose internet access when building Docker images (no IRQs are shared based on lspci -vv not showing any)
• Audio sometimes cuts out for a second on windows that aren't in direct view (such as a 2nd workspace)

Literally 11 years of using Windows with the same hardware produced none of these issues.

[kevinpbaker]

Problem Update

I've noticed that this heavy read/write freezing seems to be centered around applications that block their main threads while waiting for disk reads/writes. Therefore, web based applications are heavily affected by this. The UI for web apps or chromium will lock up, input from the mouse and keyboard becomes frozen, and only unfreezes after the heavy read/writes finish. However, while my web apps are frozen, I can still interact with my integrated development environment (Zed), where it has zero lag, and input from mouse/keyboard still works.

Chromium - Synchronous Blocking Reads/Writes

Supposedly Chromium writes often synchronously, so each tab will block the thread of execution while waiting for disk. Even if the system isn't frozen, the tab process will hang until the disk flush completes.

Zed - Asynchronous Non Blocking Reads/Writes

While Zed buffers writes in memory, which doesn't block the main thread on every I/O.

Conclusion

I'm not sure how to fix this problem. I think it has to do with btrfs file compression mixed with LUKS2 disk encryption causing overhead during heavy read/writes to disk, which causes some programs to hang while they wait for their turn to use the storage device.

[NicolasDorier]

@kevinpbaker it may be related to Ghosty #4507

[kevinpbaker]

Hmm, that definitely could be it, and confirms my understanding that I/O pressure is indeed my problem. The bug mentioned above states that Ghosty has a known problem where it will skyrocket your I/O pressure.

Note

Ghosty is the default terminal for omarchy at the time of this post.

One can watch the I/O pressure with: watch cat /proc/pressure/io.

If the average over 10 seconds goes above 90%, then all my web apps enter into the "App Not Responding" state that I've described above. I can replicate this pressure by using stress-ng --hdd 4 --hdd-bytes 20G --temp-path / --timeout 5m. I'll make the ghosty change from your PR on my system, and report back after weeks of use.

I'll also do some research on if there is a way to prevent I/O pressure from going above 85%, which would delete the root cause of the system lag (that's probably not a thing that can be done, but I'll look into it anyways).

[blazskufca]

I managed to partially mitigate this problem by doing cryptdevice=PARTUUID=<your partuuid>:root:allow-discards:no-read-workqueue:no-write-workqueue in /etc/default/limine. Run limine-updateand reboot. no-read-workqueue:no-write-workqueue will cost you some IOPS but pressure was reduced significantly for me

[kevinpbaker]

https://wiki.archlinux.org/title/Dm-crypt/Specialties#Disable_workqueue_for_increased_solid_state_drive_(SSD)_performance

Solid state drive users should be aware that, by default, discarding internal read and write workqueue commands are not enabled by the device-mapper, i.e. block-devices are mounted without the no_read_workqueue and no_write_workqueue option unless you override the default.

The no_read_workqueue and no_write_workqueue flags were introduced by internal Cloudflare research Speeding up Linux disk encryption made while investigating overall encryption performance. One of the conclusions is that internal dm-crypt read and write queues decrease performance for SSD drives. While queuing disk operations makes sense for spinning drives, bypassing the queue and writing data synchronously doubled the throughput and cut the SSD drives' IO await operations latency in half. The patches were upstreamed and are available since linux 5.9 and up [5].

---

Nice. Thanks for sharing this. I had todo the following in order to turn off the default queuing mechanism for LUKS encryption:

1. sudo cryptsetup --perf-no_read_workqueue --perf-no_write_workqueue --persistent refresh root
2. confirm it worked with sudo dmsetup table
3. reboot my system

[kevinpbaker]

A summarization of the this entire thread:

1. I updated my HDDs and SSDs to use the BFQ (Budget Fair Queueing) system
2. I stopped using the Ghosty terminal and switched to Foot
3. I turned OFF the read/write queuing system that LUKS has defaulted to on
4. I use a secondary unencrypted drive to store large files: videos, music, photos, games, etc

I've seen much better system performance because of this, and the OS hasn't locked up on me like I described in the original post.