Real-time scheduling. Hyprland dies by SIGKILL when reconnecting HDMI

[freevatar]

Hyprland version

v0.56.0 built from branch main at commit a1c2428

Is this a regression?

Unsure

What's the bug?

LG C1 is the only display connected to RTX 5090. When I unplug HDMI and reconnect it, Hyprland dies and restarts in recovery mode.

There's no crash report or core dump and strace showed SIGKILL.

Workaround

Before workaround Hyprland was using SCHED_RR with priority 1.

Probably related: #15465

Runtime workaround

Switch Hyprland to normal scheduling with chrt -o -R -p 0 <PID>

Permanent workaround

Set HYPRLAND_NO_RT=1

Setup

• AMD Ryzen 9 9950X3D
• NVIDIA RTX 5090, driver 615.78.08
• LG C1 77
• Fedora 44, kernel 7.2.11-200.fc44.x86_64
• Aquamarine 0.15.1 built from latest commit hyprwm/aquamarine@163b138

Reproduction steps

1. Turn off TV
2. Wait for 15 min
3. Turn on TV
4. Crash

OR

1. Unplug HDMI cable
2. Wait a bit
3. Plug HDMI cable back in
4. Crash

Attachments

failure-hotplug-excerpt.txt
launcher-exit-trace.txt
normal-scheduling-hotplug-excerpt.txt
normal-scheduling-metadata.txt

I acknowledge that:

• I have searched all (open and closed) discussions and issues and my problem is not there.

• I have read the Code of Conduct and agree to follow it.

• I wrote this request with my own hands, my own keyboard, and in my own words.

[freevatar]

Ok, with the help from AI I traced SIGKILL to the kernel enforcing RLIMIT_RTTIME: Hyprland had 150ms soft limit and 200ms hard limit.

On reconnect, one drm_mode_getconnector call spent 242ms on CPU without blocking. NVIDIA profiling showed 229 GSP requests inside that call totalling 241ms: about 90ms for HDMI link training, 43ms for EDID reading and 108ms for remaining requests.

Query returned successfully then SIGKILL arrived about 7µs later. No SIGXCPU was generated so Hyprland's fallback never got a chance to demote the thread.

[freevatar]

working on fix

[freevatar]

Fix: hyprwm/aquamarine#459

[iovis]

I have a similar issue on an HDMI monitor when turning dpms off, also traced to a SIGKILL from RLIMIT_RTTIME and HYPRLAND_NO_RT=1 serving as a workaround

[iovis]

Same hardware as you, but on Arch and on v0.57.0 stable

[shawnyeager]

Same thing here on Intel, with a different trigger. That makes me think a fix in the connector-probe path alone won't cover it.

Setup: Hyprland 0.57.0 (0aa6745, Arch package), aquamarine 0.15.1, kernel 7.2.8 (Omarchy build, CONFIG_POSIX_CPU_TIMERS_TASK_WORK=y), Intel Panther Lake / Arc B390 on xe, mesa 26.2.6. Displays are eDP 2880x1800@120 and a 5K DP monitor at 60 Hz. Launched by uwsm through start-hyprland.

What happens: this started the same day I upgraded from 0.56.2 to 0.57.0. I had six compositor deaths that day. Four were with Steam running, one of them under a minute after launching a game. One was about 2.5 minutes after unlocking from idle (DPMS on), while a big system update was running. No hotplug was involved in any of them. Every one looks the same:

• No coredump, no Hyprland crash report, nothing in the kernel log.
• start-hyprland logs Couldn't waitpid for hyprland: No child processes and then Hyprland exit not-cleanly, restarting.
• hyprland.log just stops in the middle of normal output.
• Realtime budget exceeded (SIGXCPU) appears zero times in the journal across all three boots. stderr does go to the journal, so the SIGXCPU handler never ran.

On the running instance:

$ chrt -p $(pgrep -x Hyprland)
current scheduling policy: SCHED_RR|SCHED_RESET_ON_FORK
current scheduling priority: 1
$ grep -i 'realtime timeout' /proc/$(pgrep -x Hyprland)/limits
Max realtime timeout      150000               200000               us

In one case, before the death at 20:58:36, both outputs went through DPMS off/on. At 20:58:29 an Xwayland client logged Key repeat discarded, Wayland compositor doesn't seem to be processing events fast enough!, and seven seconds later the process was gone.

I haven't caught the exact kernel path the way you did. On xe my guesses are the modeset on DPMS wake, or xe_exec_ioctl/reclaim under load (see #16644 for xe main-thread stalls in drm_exec_lock_obj). The pattern matches yours, though: with task-work CPU timers, the limits are only checked on return to userspace, and the hard limit is checked before the soft one. If the thread is still inside a single syscall when it crosses the 150 to 200 ms window, the first check it hits is the SIGKILL. So the SIGXCPU fallback can't be relied on for any long kernel call, whether it's a connector probe on NVIDIA or a modeset or exec on Intel.

I'm switching to HYPRLAND_NO_RT=1 now and will report back on whether the deaths stop.

[andresilva]

The root issue here is that RLIMIT_RTTIME is only checked when the thread returns to userspace, and the hard limit is checked first. We already have a SIGXCPU handler that demotes the main thread when it goes over the soft limit, but it can only run in userspace. In these cases the thread is stuck in the kernel (on connector probes). By the time the thread returns, the count is already past the 200 ms hard limit, so the kernel sends SIGKILL and the handler never gets a chance to demote the main thread.

Moving the slow call off the realtime thread, like hyprwm/aquamarine#459 does for connector probes, fixes that one path but not the underlying problem. Any kernel call on the main thread that spins for too long can trigger the same SIGKILL, and which ones do depends on the driver and the hardware. Chasing each one down is whack-a-mole, and the next one we miss crashes the compositor again.

The fix I'll try is a small watchdog thread that checks every ~20 ms whether the main thread has slept recently. If it goes about 150 ms without sleeping, the watchdog demotes it from the outside. That works even while it's stuck in a syscall. The compositor then stays at the highest normal priority for the rest of the session. The same watchdog thread could then repromote the main thread to realtime once it has seen it sleep again after a hold-off, and only fall back to high priority for good if the stalls keep coming (eg. a second one within 10 minutes). I won't implement the repromote logic for now.

[andresilva]

If you could test #16737.