[RFC] Recovering from Xid 79 / GPU fallen off the bus without rebooting Linux

[edwardgushchin]

Hi NVIDIA maintainers,

Per CONTRIBUTING.md, I am opening this RFC before turning a local prototype into pull requests.

The problem

A discrete GPU can occasionally disappear from PCIe and report Xid 79: GPU has fallen off the bus. When that GPU drives the desktop, the result is much larger than one failed application: the monitors go black, remote desktop may remain connected without receiving new frames, and every GPU-facing process becomes unusable.

At the same time, Linux itself can still be alive. Networking, storage, SSH, and unrelated system services may continue to work normally. Rebooting the whole operating system is therefore a very expensive recovery boundary for a failure isolated to one PCIe device and its clients.

The goal of this work is simple: when the GPU fails, recover as much as the hardware state safely allows, keep the Linux boot and non-GPU services alive, and restore the graphical stack without requiring an OS reboot.

How the prototype solves it

The key was to stop treating every GPU failure as the same kind of reset. The prototype classifies the state after an Xid/RM recovery decision and chooses one of two recovery paths.

1. The GPU is still reachable and supports FLR

When PCI configuration is readable and RM selects the gpu-pf-flr recovery action, the current driver object tree can be preserved:

1. A delayed worker takes the existing global power-state and PM write locks.
2. NvKMS, UVM, and user channels are quiesced through the normal system-PM suspend path.
3. The PCI function receives a Function Level Reset.
4. The normal resume path restores RM/GSP, UVM, modeset, and clients.
5. PCI identity and driver state are verified before the transaction is declared complete.
6. A session helper reapplies the compositor's unchanged current monitor layout so DRM scanout and remote capture are recreated.

The Xid callback never resets hardware directly. It records the primary event and schedules classification after a short settle window, allowing a following Xid 154 and the final RM recovery action to participate in the decision.

2. The GPU has genuinely fallen off the bus

After Xid 79, RM has already marked the GPU disconnected/lost and invalidated its channels. Restoring PCIe connectivity alone cannot make the old RM/GSP object tree valid again, so the prototype deliberately does not attempt an in-place resume.

Instead, the kernel publishes a recovery request to a privileged host supervisor. The supervisor:

1. remembers which GPU-facing services were active and drains their device users;
2. unbinds every function belonging to the GPU slot;
3. verifies that the slot is directly behind an isolated Root Port with no unrelated devices in reset scope;
4. performs a Secondary Bus Reset;
5. rebinds the GPU functions, recreating fresh driver/RM state;
6. restarts only the graphical and GPU-facing services that were active before recovery.

This path cannot preserve in-flight GPU work or the old graphical session, but it preserves the Linux boot, SSH, storage, networking, and unrelated services. No OS reboot is requested or performed.

Result

On the test system (GA102 / RTX 3090 Ti, open kernel modules 610.57.04, GNOME Wayland), rebuilding the GPU stack and replaying the current compositor layout returned all three physical displays and remote desktop frames while the Linux boot and SSH connection stayed alive.

The current prototype combines the delayed kernel classifier, PM/FLR coordinator, topology-gated host supervisor, recovery telemetry, and compositor watcher into one automatic policy. It fails closed when the recovery action or PCIe topology is ambiguous, and none of the reset paths use hard-coded PCI addresses.

The implementation also keeps the mechanism/policy boundary explicit: the kernel classifies the RM/GPU state and performs the PM-preserving transaction, while destructive host orchestration and desktop-specific layout replay remain in userspace.

Proposed upstream shape

I would like to split the work into reviewable pieces rather than submit the complete prototype as one patch:

1. accurate PM resume error propagation and recovery telemetry;
2. the PM/FLR recovery coordinator;
3. delayed Xid/RM classification and a generic kernel-to-userspace lost-link notification;
4. keep compositor integration and host service policy outside the production kernel patch.

Questions for maintainers

1. Is preserving Linux uptime after Xid 79, with an explicit loss of GPU clients but no OS reboot, within scope for this repository?
2. What kernel-to-userspace mechanism would be appropriate for the lost-link handoff: uevent, generic netlink, devlink health reporting, or an existing NVIDIA interface?
3. Is the proposed split suitable for review, or should the first change be limited to telemetry/error propagation?
4. Is there an existing RM recovery/reinitialization path or locking rule that should replace any part of this design?
5. What hardware matrix and recovery evidence would you require for the first code PR?

I can provide the source audit, state-machine traces, recovery telemetry, and hardware logs. I would prefer to align the design with maintainer guidance before publishing the patch series.

Thanks.