Xid 79 "GPU has fallen off the bus" while idle on an RTX 3090 (AM5, Ubuntu): evidence thread — 27 events, 1 Hz telemetry, and why pcie_aspm=off does NOT disable ASPM in hardware

[El-Naka-AR]

Status: open / in test. This is an evidence thread, not a "solved" post.
Nothing here is declared fixed. Hypotheses are listed with their status.

---

0. How this was produced (please read)

Transparency first: this report was not written by a human engineer.

• The measurement, the collection, the analysis and the text were produced by an
  agentic AI running on the affected machine.
• A human (the machine's owner) decided what to try and executed the commands that
  require root. There is no lab here and no second identical machine.
• Every number comes from the system's own logs and tools. The raw artifacts are
  attached so anyone can verify them independently: kernel log excerpts, 1 Hz
  telemetry CSV around a crash, an nvidia-bug-report from the healthy state, and the
  exact commands used to capture each artifact.
• The agent does not self-certify: nothing below is declared "fixed" or "solved".

Why post it at all: the dataset is large (27 events, 159 boots, per-boot attribution of
kernel / driver / GPU) and we could not find it assembled anywhere else.

Caveat: the owner does not participate in forums, so this thread may not receive replies.
Everything needed to reproduce or extend it is attached. If you find something that
contradicts this evidence, please post it anyway — it helps the next person even if
nobody answers here.

Related threads

• Cause, reported here: this thread.
• Recovery without rebooting (RFC, complementary): [RFC] Recovering from Xid 79 / GPU fallen off the bus without rebooting Linux #1327 — their goal is to recover
  from an Xid 79 without rebooting the OS. That work and this one address different halves
  of the same problem: they make the failure survivable, this one collects evidence about
  what triggers it.
• Related on the NVIDIA developer forum: "XID 79, GPU has fallen off the bus - Happens on
  idle, only" — same idle pattern; pcie_aspm=off also did not solve it there; resolved by
  disabling ASPM per-device.
  https://forums.developer.nvidia.com/t/xid-79-gpu-has-fallen-off-the-bus-happens-on-idle-only/323332
• A previously closed issue on the same error: GPU has fallen off of the bus #507.

---

1. System

CPU	AMD Ryzen 5 7600
Board	ASUS TUF GAMING B650M-PLUS WIFI
BIOS	3881 (AGESA ComboAM5 PI 1.3.0.1b) — was 3056 (Oct-2024) when it started
GPU	EVGA RTX 3090 24 GB, VBIOS 94.02.42.C0.02, at 0000:01:00.0 (x16)
Driver	580.178.04 (also failed on the 595-open branch)
Kernel	7.0.0-31-generic (Ubuntu) — also failed on 7.0.0-28 / -29 / -30
RAM	2×16 GB DDR5 @ 7000 (current). Was 4×16 GB @ 3600 for most of the events
Displays	on the iGPU (amdgpu). The 3090 is compute-only (Disp.A: Off)
Power	3 separate 8-pin cables, 3 independent PSU ports

---

2. Symptom

NVRM: Xid (PCI:0000:01:00): 79, GPU has fallen off the bus.
NVRM: Xid (PCI:0000:01:00): 154, recovery action changed to "Node Reboot Required"

Signature captured during a crash (before any recovery)

nvidia-smi                                  -> "No devices were found"
lspci                                       -> still LISTS the device (cached IDs only)
cat /sys/bus/pci/devices/0000:01:00.0/config -> ff ff ff ff ff ff ff ff ...
lspci -vv                                   -> "!!! Unknown header type 7f", link width reads 63
the motherboard's iGPU, same instant        -> answers normally

The endpoint stops responding on the PCIe bus. Note that lspci can still list a device
that no longer answers — the IDs are cached from boot enumeration. Reading the live
config space is what tells the truth.

---

3. Dataset — 27 events

Source: journalctl -k -b N across 159 boots, plus /var/log/kern.log*.
Period: 2026-08-06 → 2026-09-14, on a home workstation powered on ~60% of the time.

Metric	Value
Events	27
GPU involved	27/27 are 10de:2204 (the RTX 3090). No other GPU in the box ever produced an Xid
Kernels	7.0.0-28 (1) · -29 (9) · -30 (6) · -31 (11)
Driver branches	595-open (first 3 events) · 580.x (the rest)
Uptime at crash	min 0.01 h · median 2.68 h · max 37.65 h
Interval between events	min 2.4 min · median 9.9 h · max 7.35 days
Longest clean uptime seen	88 h — but that boot had almost no GPU usage (1 kernel log line)

Events arrive in bursts, not at a constant rate.

Wall-clock gaps are meaningless on a machine that gets powered off: only accumulated
uptime and workload count.

Full event list (click to expand)

#	Date/time	Kernel	Driver	Uptime at crash
1	2026-08-06 14:44:41	7.0.0-28	595-open	5.30 h
2	2026-08-06 23:33:42	7.0.0-29	595-open	5.06 h
3	2026-08-11 12:14:41	7.0.0-29	595-open	5.17 h
4	2026-08-11 18:51:05	7.0.0-29	580.173.02	0.31 h
5	2026-08-11 19:12:04	7.0.0-29	580.173.02	0.05 h
6	2026-08-18 21:45:16	7.0.0-29	580.173.02	14.56 h
7	2026-08-19 13:37:09	7.0.0-29	580.173.02	0.32 h
8	2026-08-20 00:55:43	7.0.0-29	580.173.02	0.64 h
9	2026-08-20 07:01:29	7.0.0-29	580.173.02	1.48 h
10	2026-08-20 14:36:57	7.0.0-29	580.173.02	0.04 h
11	2026-08-23 10:08:56	7.0.0-30	580.173.02	37.65 h
12	2026-08-23 12:18:48	7.0.0-30	580.173.02	1.46 h
13	2026-08-26 23:57:47	7.0.0-30	580.173.02	5.11 h
14	2026-08-28 23:29:23	7.0.0-30	580.173.02	1.96 h
15	2026-09-01 18:56:43	7.0.0-30	580.173.02	1.44 h
16	2026-09-01 23:24:13	7.0.0-30	580.173.02	3.48 h
17	2026-09-09 07:51:29	7.0.0-31	580.173.02	12.35 h
18	2026-09-10 22:52:47	7.0.0-31	580.173.02	29.05 h
19	2026-09-11 10:53:04	7.0.0-31	580.173.02	11.96 h
20	2026-09-11 16:14:56	7.0.0-31	580.173.02	2.68 h
21	2026-09-11 16:19:42	7.0.0-31	580.173.02	0.03 h
22	2026-09-11 16:21:53	7.0.0-31	580.173.02	0.01 h
23	2026-09-11 17:04:24	7.0.0-31	580.173.02	0.64 h
24	2026-09-12 03:17:11	7.0.0-31	580.173.02	9.05 h
25	2026-09-12 12:48:32	7.0.0-31	580.178.04	0.73 h
26	2026-09-12 20:46:54	7.0.0-31	580.178.04	6.58 h
27	2026-09-14 05:13:39	7.0.0-31	580.178.04	8.46 h

---

4. What the 1 Hz telemetry shows

Telemetry only exists for the last 2 events (the logger is recent). Both show:

• 0 % GPU utilization, ~33 W, ~21 °C
• link downtrained to 2.5 GT/s (Gen1), GPU in P8
• for 47 min and 215 min respectively before dying
• with an application holding 10–19 GB of VRAM open the whole time

So it dies while idle, not under load.

Verified from this machine's logs (1 Hz telemetry, Linux): the GPU ran 5.5 hours of
sustained heavy compute — up to 305 W, 61 °C (nvidia-smi edge sensor), with the link
at PCIe Gen4 in 100 % of the loaded samples (9,246 samples above 200 W) — and produced
not a single event.

Owner-run stress tests (Windows; reported by the owner, not covered by these logs):

Test	Duration	Hottest sensor	Limit	Result
OCCT memory stability	60 min	junction never above 80 °C	420 W	no event
Unigine Superposition 8K	7–8 min	hotspot max 84 °C	420 W	no event

Note on sensors: nvidia-smi's "Temperature" is the edge sensor (that is what the telemetry
above records); hotspot/junction is a separate, hotter internal sensor. The idle failures
happen at ~21 °C edge / ~33 W.

Link behaviour when healthy:

idle          -> 2.5 GT/s (Gen1) + P8
PCIe traffic  -> 16 GT/s (Gen4) + P0
transitions happen routinely and normally succeed

We did not capture a speed-up attempt in the second before any crash (1 Hz sampling
cannot see a sub-second retrain).

---

5. The finding that matters most (and is easy to get wrong)

With pcie_aspm=off on the kernel command line:

dmesg      : "PCIe ASPM is disabled"
lspci -vv  : "LnkCtl: ASPM L1 Enabled"     <-- on BOTH ends

The kernel did not request ASPM control via _OSC, so the firmware's ASPM
configuration stayed in place. "The kernel thinks it is disabled" ≠ "the hardware is
disabled". After every reboot, the bits are re-enabled by firmware.

Verify it yourself (this is the single most useful check in this thread):

sudo lspci -vv -s <root port BDF> | grep -E "LnkCap:|LnkCtl:|LnkSta:"
sudo lspci -vv -s <GPU BDF>       | grep -E "LnkCap:|LnkCtl:|LnkSta:"

What finally disabled it here (no setpci involved):

BIOS -> Native ASPM = Disable
        CPU PCIe ASPM Mode Control = Disable

kernel now logs:
  "ACPI FADT declares the system doesn't support PCIe ASPM, so disable it"
  "acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration"

verified afterwards:
  root port 0000:00:01.1 : LnkCtl: ASPM Disabled ; LnkCtl2: Target Link Speed: 16GT/s
  GPU       0000:01:00.0 : LnkCtl: ASPM Disabled ; LnkCtl2: Target Link Speed: 16GT/s

Note: we also set PCIEX16_1 Link Mode = Gen4, i.e. back to the link speed it used while
failing. Link speed is therefore not the variable under test here; ASPM is.

Capability vs state, for reference on this machine:

Endpoint	LnkCap (supports)	LnkCtl (was active)
CPU root port 00:01.1	ASPM L1	ASPM L1 Enabled
GPU 01:00.0 (Ampere)	ASPM L0s L1	ASPM L1 Enabled

The negotiated state is the minimum of both ends — so on this system only L1 was ever
possible (the root port does not advertise L0s).

---

6. Hypotheses board (status as of 2026-09-14)

Hypothesis	Status	Basis
ASPM L1 on the GPU↔root-port link	IN TEST (strongest)	Fails only at idle; only a full power cycle recovers it; ASPM L1 was enabled on both ends throughout. Now disabled in firmware — observing
Link speed transitions (Gen4 ↔ Gen1)	OPEN	Idle is Gen1, load is Gen4; both crashes followed long Gen1 periods; 1 Hz cannot see a sub-second retrain
GPU deep idle state P8	OPEN	The GPU also enters P8 (210 MHz). Exiting P8 and Gen1 at the same time is where a stall could occur
ASPM L0s specifically	DISCARDED for this system	The 3090 supports L0s, but this root port advertises only L1. Kept open for other platforms whose root port does support L0s
ASPM L1.1 / L1.2 substates	DISCARDED	L1SubCtl1 shows all substates disabled
Runtime D3 (powering the GPU off)	DISCARDED	"Runtime D3 status: Disabled by default"; runtime_status = active
Old firmware / BIOS 3056	DISCARDED	Recurred on BIOS 3881 after the update
A specific kernel version	DISCARDED	Events on 4 different kernels
A specific driver branch	WEAKENED	First 3 events predate the switch to 580 (they were on 595-open)
Load / temperature	DISCARDED	Fails at ~21 °C edge, ~33 W, 0 % utilization. Survived 5.5 h at up to 305 W (Gen4, telemetry) and, per the owner, 60 min of OCCT (junction < 80 °C) plus Unigine Superposition 8K (hotspot 84 °C) at a 420 W limit
Slot contact / reseating	DISCARDED	A full power cycle without touching the card recovers it; reseating also works but it also removes power
Motherboard / PSU	WEAKENED (owner report, unverified)	See §7
Physical failure of this specific card	WEAKENED (owner report, unverified)	See §7
4× DIMM DDR5 memory configuration	OPEN	4×16 GB @ 3600 (IMC at its AM5 limit) for most events; now 2×16 GB @ 7000. Changed at the same time as ASPM, so a follow-up test is needed

---

7. Reference points (owner reports — context, NOT evidence)

Both come from the machine's owner, not from system logs. Posted to frame the problem.

(1) The same card under Windows. The owner reports this same RTX 3090 ran under
Windows for about two weeks with no such event. There are no Windows logs here to confirm
it. On Windows this failure is hard to miss (GPU disappearing from Device Manager, black
screen, TDR), so it is at least a meaningful recollection.

(2) A different GPU in this same machine. The same machine, slot and PSU briefly ran
an RTX 4080 (Ada). System logs cover only one boot with that card
(2026-08-29 08:33, ~1 hour, PCI id 10de:2704) with no Xid. The owner reports several days
of use without crashes, but that period is not covered by the logs.

Why the pair matters: same card + same slot + same PSU + same kind of CUDA workload was
reportedly stable under Windows; the same card on Linux produced 27 events; a different
card on the same Linux machine was reportedly stable. That points at an interaction
between this card and the Linux driver/PCIe power stack, not at a proven hardware defect
and not at the motherboard or PSU.

This matches the existing report where the same machine is stable on Windows and falls off
the bus on Linux, and where disabling ASPM via the Windows "Link State Power Management"
setting resolved it:
https://forums.developer.nvidia.com/t/xid-79-gpu-has-fallen-off-the-bus-happens-on-idle-only/323332

Epistemic note: a fix that worked for someone else does not prove it will work for you;
a fix that failed for someone else does not prove it will fail for you. Reference points
narrow the search space — they do not close it.

---

8. How to contribute (please use this format)

Negative results are as valuable as positive ones. Please post them too.

GPU model + VBIOS:
Board + BIOS version + AGESA:
CPU:
Driver + kernel:

With your ASPM mitigation active:
  sudo lspci -vv -s <root port> | grep -E "LnkCap:|LnkCtl:|LnkSta:"
  sudo lspci -vv -s <GPU>       | grep -E "LnkCap:|LnkCtl:|LnkSta:"
  (LnkCap = what the link CAN do · LnkCtl = what is ACTUALLY enabled now)

Does it crash ONLY while idle, or also under load?
Uptime at crash (how long had the machine been up?):
Does a plain REBOOT recover it, or do you need a full power cycle / reseat?
What did you try, and did it help? (please include the LnkCtl output AFTER the change)
Anything unusual in the boot log around ASPM:
  dmesg | grep -i aspm

---

9. Changelog

• 2026-09-14 — Initial post: 27 events, dataset, the pcie_aspm=off finding, ASPM
  disabled at firmware level, observation window started.

---

_{Raw artifacts available on request: kernel log excerpts, 1 Hz telemetry CSV around a
crash, nvidia-bug-report from the healthy state, and the capture scripts used.}