Bridged 2x RTX 3090, both unlocked: Xid 74 (NVLink fatal) on all 4 links under load — concurrent unlock as the suspect

[hybridindie]

TL;DR — 2× RTX 3090 on an NV4 bridge, both cards unlocked, 615.71.09 open modules. Boots clean, both cards give full speed. Then a dual-card GEMM takes Xid 74 "NVLink: fatal error detected" on all four links about 4½ minutes in, and the box hard-locks until reset. Unlocking only the headless card has been stable for days. The one structural difference I can find versus the known-good bridged 2×3090 reports is that both cards are unlocked concurrently at boot. Serialising the unlocks gave me one clean 10-minute soak — which I don't consider proof, so I'm posting for data points and a sanity check on the mechanism.

Setup

• 2× RTX 3090 24 GB (10de:2204), VBIOS 94.02.42.00.A8, stock, 390 W each (driver default, no cap service).
• 0b:00.0 drives GNOME, 0c:00.0 is headless. NV4 bridge: nvidia-smi topo -m → GPU0 ↔ GPU1 = NV4. Each card on its own CPU root port, no PCIe switch. Display slot trains Gen4 x8.
• Fedora 45, kernel 7.2.15-300.fc45.x86_64, NVIDIA 615.71.09 open kernel modules (akmods), dracut, Secure Boot off.
• gpu-unlock 7eb51bc, built with make rpm + DKMS, firmware 610.43.03 in the initramfs, rd.driver.pre=gpu_unlock, nvidia.ko deliberately absent from the initramfs.
• Cmdline: ... gpu_unlock.targets=0000:0b:00.0,0000:0c:00.0

Symptom

Boot and unlock are clean (stage=handoff ret=0, unlocked, both cards), 115–119 TFLOPS BF16 each. Then a sustained dual-card BF16 GEMM — plain torch.matmul 8192³, no P2P, no NCCL, so NVLink carries no traffic at all — does this at 4 m 38 s:

kernel: NVRM: GPU at PCI:0000:0c:00: GPU-4f824799-...
kernel: NVRM: Xid (PCI:0000:0c:00): 74, NVLink: fatal error detected on link 0(0x10010000, 0x0, ...)
kernel: NVRM: Xid (PCI:0000:0c:00): 74, NVLink: fatal error detected on link 1(0x10010000, ...)
kernel: NVRM: Xid (PCI:0000:0c:00): 74, NVLink: fatal error detected on link 2(0x10010000, ...)
kernel: NVRM: Xid (PCI:0000:0c:00): 74, NVLink: fatal error detected on link 3(0x10010000, ...)
kernel: NVRM: Xid (PCI:0000:0c:00): 45, pid=35221, name=python, channel 0x00000002 caused by previous Xid 45
   (32× Xid 45 across channels 2–10, same pid)
kernel: NVRM: kgmmuInvalidateTlb_GM107: TLB invalidation failed waiting for prior invalidate (status=0x0000000f)
kernel: NVRM: nvAssertFailedNoLog: Assertion failed: 0 @ mmu_walk_unmap.c:62
kernel: NVRM: GPU1 virtmemDestruct_IMPL: VirtualMemory memmgrFree failed, client: ..., gpu: 1

The worker dies with CUBLAS_STATUS_EXECUTION_FAILED, the GNOME session goes into a shutdown loop, NVRM then logs _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 76 every 30 s, and the journal stops with no systemd-shutdown lines — hard reset required.

The bridge itself is fine afterwards: nvidia-smi nvlink -s → 4×14.062 GB/s on both cards, nvlink -e → 0 replay / CRC / recovery errors.

I don't think it's power

This machine previously had a 1000 W PSU that hard-latched (OCP, wall-switch recovery) seconds after unlocking both cards — that was power, and it's gone since a 1600 W unit went in. During this failure the rails never reached the limit:

GPU0 (0b, display)  avg 375.0 W  max 389.9 W  max 93 °C  SM 1305 MHz
GPU1 (0c, headless)  avg 381.0 W  max 390.1 W  max 74 °C  SM 1440 MHz
combined ~756 W avg, 140 samples at 2 Hz, none above 800 W

And the box kept running for seven minutes after the Xid rather than losing power.

Stable vs unstable, same hardware, same day

Config	5-min dual-card GEMM soak
0c unlocked, 0b locked	stable for days, including the soak (~744 W avg / 780 W max)
both unlocked	Xid 74 on all 4 links at 4 m 38 s, hard reset

The failing run is lower in total power than the passing one, and carries no NVLink traffic.

The one difference I can see

main.c schedules every selected card on one shared async domain, so the whole set is reset concurrently. Each unlock ends in pcie_flr() + pci_restore_state(). The boot log shows both stage=claim at 12:44:16 and both handoff/unlocked at 12:44:17:

pci 0000:0b:00.0: gpu_unlock: stage=claim ...
pci 0000:0c:00.0: gpu_unlock: stage=claim ...
pci 0000:0c:00.0: gpu_unlock: stage=handoff ret=0 ... / unlocked
pci 0000:0b:00.0: gpu_unlock: stage=handoff ret=0 ... / unlocked

So two PF-FLRs plus two PCIe state restores land inside the same second on the two ends of the bridge. I'm aware of the #260 analysis that on GA102 the module writes no link registers and that this FLR is the one VFIO does on every passthrough — which is exactly why I can't explain how it would still matter minutes later, after nvidia.ko has trained the links. That's question 1 below.

What I tried

Serialised the unlocks — firmware still loaded once up front, then each card unlocked before the next starts. Boot log is now strictly sequential, one second apart:

13:38:59 0000:0b:00.0 stage=claim
13:39:00 0000:0b:00.0 unlocked
13:39:00 0000:0c:00.0 stage=claim
13:39:01 0000:0c:00.0 unlocked

Result: 10-minute dual-card soak completed — 618 samples, 727 W avg / 779 W max, 0 Xid, and nvidia-smi nvlink -e sampled every 20 s stayed at zero replay/CRC/recovery errors throughout. Per-card: 116.3 TFLOPS on the headless card, 94.3 on the display card (thermally throttled at 93 °C).

-static ASYNC_DOMAIN_EXCLUSIVE(unlock_domain);
-
 /* Prepare all selected firmware families, then run the ready cards concurrently. */
 static void unlock_cards(async_func_t unlock)
 {
 	...
 	list_for_each_entry(card, &cards, node) {
 		if (card->prepared)
-			async_schedule_domain(unlock, card, &unlock_domain);
+			unlock(card, 0);
 	}
-	async_synchronize_full_domain(&unlock_domain);
 	gpu_unlock_610_release_shared(&shared);
 }

Caveats I'd raise against my own result:

• n=1 per arm, and boots are the trial unit, not soaks — I have one patched boot and one unpatched boot.
• Not thermally identical: the display card averaged 342 W patched vs 375 W unpatched (it throttled harder at 93 °C), so "less heat at the bridge" is a live alternative explanation.
• The fault is intermittent: of three unpatched boots on this box a while back, two crashed and one survived. One clean run is therefore weak evidence.
• No mechanism — I can't explain the causal chain from a boot-time reset ordering to a fault minutes later.

Questions

1. Mechanism: could a PF-FLR on both ends of a bridge within ~1 s leave the link in a state that only faults later under load? Or is reset ordering irrelevant because the links are trained later by the driver anyway?
2. Signature: is Xid 74 on all four links at once (rather than one link degrading first) a link-hardware signature, or something the driver/GSP can produce on its own?
3. Anyone running bridged pairs unlocked and stable? Especially 2 cards on one bridge. Your boot-log timestamps would tell us whether you're effectively running concurrent or serialised unlocks — just journalctl -k -b | grep -E "gpu_unlock: (stage=claim|unlocked)".
4. Anyone seen Xid 74 (not 79) with the unlock in the picture?

What would settle it

Interleaved cold-boot A/B — unpatched soak, patched soak, three pairs — with success defined as patched 3/3 clean and unpatched at least 1/3 faulting, which also pins the base rate. Then repeat both arms with the display card capped at 300 W to decouple heat from the patch: if unpatched stops faulting under the cap, thermals own the problem and the serialisation is incidental.

I'll report back either way. I haven't opened a PR — happy to if there's interest in serialising (it costs ~1 s of boot per extra card, which may be unacceptable on 8-GPU boxes, so a module parameter might be the better shape).