Kernel use-after-free panic: mst_ioctl() dereferences stale struct pci_dev after PCIe hotplug removal (mstflint_access)

[ActiveInAI]

Summary

On NVIDIA DGX Spark, running mstflint after a ConnectX-7 cable hotplug event reliably panics the kernel.

The mstflint_access kernel module caches a struct pci_dev * in struct mst_dev_data and never invalidates it. NVIDIA's own hotplug handler (dgx-spark-mlnx-hotplug) responds to a CX7 cable event by writing to /sys/bus/pci/devices/<bdf>/remove, which frees those struct pci_dev objects. The character devices (/dev/<bdf>_mstconf) survive the removal. The next MST_PARAMS ioctl dereferences the freed pointer → use-after-free → Oops → panic_on_oops=1 → unplanned reboot.

Both components involved are shipped by NVIDIA, and the module is force-loaded at boot on every DGX Spark, so every unit is exposed by default.

Impact

• Kernel use-after-free → panic → unplanned reboot (data loss on a running appliance).
• Triggered by a supported workflow: replug a CX7 cable, then query NIC firmware with mstflint.
• Reachable as root only (/dev/*_mstconf is crw------- root:root), so this is a reliability / kernel-memory-safety bug, not a privilege boundary crossing. A read of freed slab memory is nonetheless undefined behavior and the faulting pointer is fully attacker-influenced by whatever reuses the slab.

Environment

Item	Value
Platform	NVIDIA NVIDIA_DGX_Spark/P4242 (aarch64)
BIOS	5.36_0ACUM018 08/06/2025
Kernel	6.17.0-1021-nvidia #21-Ubuntu (linux-image-nvidia-hwe-24.04 = 6.17.0-1021.21)
Faulting module	mstflint_access 2.0.0, shipped inside linux-modules-6.17.0-1021-nvidia (not DKMS, not the Ubuntu mstflint deb)
Autoload	nvidia-mstflint-loader 23.03-1 → /etc/modules-load.d/nvidia-mstflint.conf
Hotplug handler	dgx-spark-mlnx-hotplug 26.01-1 → /lib/udev/rules.d/90-mtk-hotplug.rules → /opt/nvidia/dgx-spark-mlnx-hotplug/mtk-hotplug-handler.sh
Hotplug driver	cx7-pcie-hotplug (mtk_pcie_hotplug), ACPI dev MTKP0001:00
Userspace	mstflint 4.26.0 (Git SHA 9f7f49c), deb 4.26.0+1-2ubuntu3
NIC	4× ConnectX-7 MT2910 [15b3:1021] @ 0000:01:00.{0,1}, 0002:01:00.{0,1}
NIC firmware	28.45.4028

Reproduction

Observed once in production (uptime 9 days). Not deliberately re-run — it panics the host. The sequence recovered from logs:

1. Boot. mstflint_access is auto-loaded 8 s in; it creates /dev/0000:01:00.0_mstconf etc. and caches each struct pci_dev *.
2. Unplug the ConnectX-7 cable.
   cx7-pcie-hotplug MTKP0001:00: Cable removal
   udev runs mtk-hotplug-handler.sh removal, whose remove_devices_in_domain() does:

   echo 1 > /sys/bus/pci/devices/${device_bdf}/remove

   → every struct pci_dev in the domain is freed. mstflint_access is not notified and its char devices remain.
3. Replug the cable (13 s later).
   cx7-pcie-hotplug MTKP0001:00: Cable plugin → pci_rescan_bus() → mlx5_core re-probes, new pci_dev objects are allocated.
   (PCIe AER also logs Multiple Uncorrectable (Fatal) error message received on both root ports during this window.)
4. ~4 minutes later, run mstflint -d <bdf> q.
   → open("/dev/<bdf>_mstconf") → MST_PARAMS ioctl → dereference of the stale pci_dev → Oops.

Root cause

Upstream kernel/mst_main.c (built as mstflint_access.ko):

• struct mst_dev_data holds struct pci_dev *pci_dev;
• mst_ioctl() case MST_PARAMS reads pci_dev->bus->number, pci_dev->devfn, pci_dev->device, …
• There is no struct pci_driver with .remove(), no PCI bus notifier, and no pci_error_handlers. Nothing invalidates or refcounts the cached pointer when the device disappears.

The register state proves exactly this chain:

pc : mst_ioctl+0x960/0x1810 [mstflint_access]
Code: 97fffc4a f9402260 52800003 f9400801 (b940dc22)
                                           ^^^^^^^^ ldr w2, [x1, #0xdc]

x19 = ffff0000939b2c00   ; struct mst_dev_data *
x0  = ffff000081dae000   ; = [x19 + 0x40]  -> dev->pci_dev   (freed slab, still mapped)
x1  = 00000b6c000002e4   ; = [x0  + 0x10]  -> pci_dev->bus   (garbage: slab reused)
fault addr = 00000b6c000003c0  ; = x1 + 0xdc  -> bus->number

0x2e4 + 0xdc = 0x3c0 — the faulting address is exactly pci_dev->bus + 0xDC. Offset 0x10 into struct pci_dev is struct pci_bus *bus (immediately after the 16-byte bus_list). x0 still points into a valid slab page, but its contents have been overwritten by whatever reallocated the freed pci_dev — a textbook use-after-free read (WnR=0, FSC=0x04 level 0 translation fault).

Suggested fix

Any one of these closes it; (1)+(2) together is the clean fix:

1. Invalidate on removal. Register a PCI bus notifier and, on BUS_NOTIFY_DEL_DEVICE / BUS_NOTIFY_REMOVED_DEVICE, take the device lock, set mst_dev_data->pci_dev = NULL, and have every mst_ioctl case return -ENODEV when it is NULL. (Deleting/hiding the char device at the same time would be better still.)
2. Hold a reference. pci_dev_get() when binding the char device, pci_dev_put() on release, so the struct pci_dev cannot be freed while a handle is open. Note that a refcount alone is not sufficient — the device is still removed, and pci_dev->bus may be torn down — so this must be combined with (1).
3. Become a real PCI driver with a .remove() callback instead of a bare chardev that squirrels away a pointer it does not own.

On the DGX Spark integration side, mtk-hotplug-handler.sh unbinds/removes PCI devices out from under a module that NVIDIA also force-loads at boot via /etc/modules-load.d/nvidia-mstflint.conf. Even with the module fixed, it is worth making the hotplug handler aware of consumers of those devices.

Workaround until fixed: after any CX7 cable hotplug event, do not run mstflint / mstconfig / anything opening /dev/*_mstconf until the host has been rebooted (or rmmod mstflint_access first, then reload).

Full Oops

Unable to handle kernel paging request at virtual address 00000b6c000003c0
Mem abort info:
  ESR = 0x0000000096000004
  EC = 0x25: DABT (current EL), IL = 32 bits
  SET = 0, FnV = 0
  EA = 0, S1PTW = 0
  FSC = 0x04: level 0 translation fault
Data abort info:
  ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
  CM = 0, WnR = 0, TnD = 0, TagAccess = 0
  GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
user pgtable: 4k pages, 48-bit VAs, pgdp=000000080cf97000
[00000b6c000003c0] pgd=0000000000000000, p4d=0000000000000000
Internal error: Oops: 0000000096000004 [#1]  SMP
CPU: 13 UID: 0 PID: 1768087 Comm: mstflint Tainted: G        W  O        6.17.0-1021-nvidia #21-Ubuntu PREEMPT(none)
Tainted: [W]=WARN, [O]=OOT_MODULE
Hardware name: NVIDIA NVIDIA_DGX_Spark/P4242, BIOS 5.36_0ACUM018 08/06/2025
pstate: 63400009 (nZCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
pc : mst_ioctl+0x960/0x1810 [mstflint_access]
lr : mst_ioctl+0x954/0x1810 [mstflint_access]
sp : ffff8000c65db8e0
x29: ffff8000c65dbb50 x28: ffffbc05cebe50a0 x27: 0000000000000000
x26: 0000000000000000 x25: 0000000000000000 x24: ffff000096a57808
x23: c8e5bc05cebe2de8 x22: 000000008028d001 x21: ffff8000c65db910
x20: 0000ffffcaceb428 x19: ffff0000939b2c00 x18: ffff80009fdc0040
x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
x11: 0000000000000000 x10: 0000000000000000 x9 : ffffbc05cebe1efc
x8 : 0000000000000000 x7 : 0000000000000000 x6 : 0000000000000000
x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
x2 : 0000000000000000 x1 : 00000b6c000002e4 x0 : ffff000081dae000
Call trace:
 mst_ioctl+0x960/0x1810 [mstflint_access] (P)
 unlocked_ioctl+0x30/0x60 [mstflint_access]
 __arm64_sys_ioctl+0xd0/0x160
 invoke_syscall+0x74/0x128
 el0_svc_common.constprop.0+0x4c/0x140
 do_el0_svc+0x28/0x58
 el0_svc+0x40/0x198
 el0t_64_sync_handler+0xc0/0x108
 el0t_64_sync+0x1b8/0x1c0
Code: 97fffc4a f9402260 52800003 f9400801 (b940dc22)
---[ end trace 0000000000000000 ]---

Preceding hotplug window (same boot, 4 minutes earlier)

cx7-pcie-hotplug MTKP0001:00: Cable removal            (T-4m25s)
cx7-pcie-hotplug MTKP0001:00: Cable plugin             (T-4m12s)
pcieport 0000:00:00.0: AER: Multiple Uncorrectable (Fatal) error message received from 0000:ff:1f.7
pcieport 0002:00:00.0: AER: Multiple Uncorrectable (Fatal) error message received from 0002:ff:1f.7
mlx5_core 0000:01:00.0: enabling device (0000 -> 0002)   ... (re-probe of all 4 functions)

A secondary, non-fatal issue in the same window (kernel tainted W, likely unrelated to the panic but worth a look):

mtk-hotplug-han: page allocation failure: order:8, mode:0x40dc0(GFP_KERNEL|__GFP_ZERO|__GFP_COMP)
Call trace:
 ...
 mlx5_fw_tracer_allocate_strings_db+0x50/0x160 [mlx5_core]
 mlx5_fw_tracer_create+0x1fc/0x368 [mlx5_core]
 mlx5_init_once+0x170/0x540 [mlx5_core]
 probe_one+0x104/0x248 [mlx5_core]
 ...
 pci_rescan_bus+0x3c/0x68
 dev_rescan_store+0xac/0xc8

mlx5_fw_tracer_create() requests an order-8 (1 MB) contiguous allocation during hotplug re-probe and does not abort probe when it fails. The host had 26 GB free at the time — this is high-order fragmentation, not memory exhaustion.

Notes on evidence

kdump is non-functional on the stock DGX Spark image (crashkernel=1G-:0M reserves 0 MB; USE_KDUMP unset in /etc/default/kdump-tools), so no vmcore exists. The Oops above was recovered from efi_pstore via systemd-pstore's archive at /var/lib/systemd/pstore/<epoch>/001/dmesg.txt. Suggest fixing the shipped crashkernel= value.

[ActiveInAI]

Follow-up: this use-after-free is triggered automatically, with no user action.

The original report described the sequence as "hotplug a ConnectX-7 cable, then run mstflint". That framing understates it. On a stock DGX Spark, nobody runs mstflint — NVIDIA's own dgx-dashboard-admin daemon (root, Restart=always) does, on an hourly ticker:

dgx-dashboard-admin  (hourly, anchored to service start)
  └─ refreshUpdatesCache
       └─ /opt/nvidia/spark-ota-check/check_ota_status.py
            └─ ota_checker.py:155  → check_firmware.py:176
                 run_cmd(["mstflint", "-d", pci_addr, "q"])

/opt/nvidia/spark-ota-check/metadata/ holds 10 recipe files, so each pass runs mstflint 10×.

Measured. A bpftrace probe on tracepoint:sched:sched_process_exec filtered to comm == "mstflint", left running 19 hours on an idle host, recorded 570 executions of /usr/bin/mstflint -d 0000:01:00.0 q — 57 short-lived parent PIDs × 10 each, 30/hour, in one burst per hour. None were human-initiated.

The panic occurred inside one of those passes. The daemon logs each cycle. Across the crash boot's log there are 217 refreshUpdatesCache: starting and 216 refreshUpdatesCache: completed. The single unmatched one is the last thing in the file:

 1:32PM INF refreshUpdatesCache: completed ...
 2:30PM INF refreshUpdatesCache: starting
 2:30PM INF Network connectivity confirmed
<end of file>

(12-hour timestamps; 2:30PM = 14:30. The Oops in mst_ioctl+0x960/0x1810 [mstflint_access] is at 14:31:39.) On the next, healthy boot the same log shows 21 starting / 21 completed.

Negative control. That healthy boot had no cx7-pcie-hotplug Cable removal/Cable plugin and no AER: ... Uncorrectable (Fatal). All 570 mstflint runs completed normally. So mstflint is not the hazard — the stale struct pci_dev * cached by mstflint_access is, and the hotplug → AER-fatal → pci_rescan_bus() sequence is what creates it:

14:27:14  cx7-pcie-hotplug MTKP0001:00: Cable removal
14:27:27  cx7-pcie-hotplug MTKP0001:00: Cable plugin
14:27:29  pcieport 0000:00:00.0: AER: Multiple Uncorrectable (Fatal) error message received
14:27:29  pcieport 0002:00:00.0: AER: Multiple Uncorrectable (Fatal) error message received
14:30:xx  refreshUpdatesCache: starting        (hourly, unattended)
14:31:39  Oops: mst_ioctl+0x960/0x1810 [mstflint_access]  → panic_on_oops=1 → reboot

Consequences for this issue:

1. Severity is higher than "root runs a diagnostic tool and crashes the box". Reseating a QSFP cable — routine maintenance, cable management, a rack move — arms the UAF, and it fires by itself within the hour. The window between arming and detonation is bounded by the ticker, not by operator behaviour.
2. The workaround "don't run mstflint after a hotplug event" does not work. The only effective mitigations are systemctl stop dgx-dashboard-admin, rmmod mstflint_access (refcnt is 0 on an idle system), or blacklist mstflint_access.
3. The module identifies itself as version: 2.0.0 (Nov-27-2012) and registers no pci_driver.remove(), no PCI bus notifier, and no pci_error_handlers — so nothing invalidates the cached pci_dev on removal. A bus notifier (or simply resolving the BDF at ioctl time instead of caching the pointer) would close this.

Environment: DGX Spark (aarch64), kernel 6.17.0-1021-nvidia, mstflint_access shipped inside linux-modules-6.17.0-1021-nvidia (not DKMS), force-loaded every boot by nvidia-mstflint-loader 23.03-1 via /etc/modules-load.d/nvidia-mstflint.conf. Userspace mstflint 4.26.0.

---

One more thing that makes this worse: the hotplug path that arms the UAF is enabled by default and undocumented.

mtk-hotplug-handler.sh gates itself on /etc/nvidia/cx7-hotplug-enabled, whose absence it calls the "safe default". But dgx-spark-mlnx-hotplug.postinst creates that file on install, so the feature is on out of the box (/sys/devices/platform/MTKP0001:00/pcie_hotplug/hotplug_enabled = 1 on both of our units). The DGX Spark User Guide's Spark Stacking page documents how to insert and remove the interconnect cable and says nothing about whether the systems may be powered on.

When it fires, remove_devices_in_domain() does echo 1 > /sys/bus/pci/devices/<bdf>/remove for every device in PCI domains 0000 and 0002 — root ports included — then plugin() rescans. That is what frees the struct pci_dev objects mstflint_access has cached, and it is why the AER log shows Multiple Uncorrectable (Fatal) on both root ports.

On our units the CX-7 is enumerated by the kernel at t=0.075 s and mlx5_core probes at t=1.23 s, while the cx7-pcie-hotplug driver only initialises at t=7.56 s. Boot-time enumeration therefore does not depend on the hotplug handler, and removing /etc/nvidia/cx7-hotplug-enabled (NVIDIA's own documented disable mechanism) is a safe interim mitigation that turns an accidental cable bump from "tear down the whole PCI domain" into "link down".