fix(talos): add pcie_port_pm=off for Arc B70 root-port race

[Aviator-Coding]

Intent

Close the PCIe runtime power-management race that made the Arc Pro B70 disappear from talos-3, by adding one kernel argument to the schematic; also record that the existing operational fix was confirmed to work, and write the runbook for the single attended reboot that will activate this.

SCOPE CONSTRAINT (explicit, from the requester): ship the CODE CHANGE ONLY. Do NOT upgrade, reboot, powercycle, drain, apply config to any node, or change any workload. That step is the captain's, timed by him. If the change appears unvalidatable without one, that is a decision to escalate, not to perform. No node was touched during this work: verified read-only that the card is still healthy, the live schematic is still b1a6b2ff, and uptime is unbroken.

MECHANISM (authoritative record: docs/hardware-incidents.md:9-64). At boot the link was down. pcie_failed_link_retrain() retrained at 2.5 GT/s and failed, and the scan of bus 01 found nothing. 218 ms in, with zero children, Linux runtime-suspended 00:01.0 to D3hot. The card then finished training properly (Gen4 x4, equalization complete on all three phases, real link partner). Nothing could ever notice, because SLTCAP HotPlugCapable=0 and SLTCTL=0x0000 disable every hotplug and link-state interrupt; a root port that cannot report a late-arriving device cannot self-heal. Separately, SLTCAP PowerController=0 is why cycling host power does not cycle the dock's PSU, which is why the 2026-08-24 host power cycle did not recover it.

THE CHANGE: add pcie_port_pm=off to extraKernelArgs in talos/schematic.yaml.j2, with a comment in the same style as its neighbours explaining what it does and why, pointing at docs/hardware-incidents.md.

VERIFICATION REQUIREMENT (explicitly demanded before writing it): confirm pcie_port_pm is a real parameter for the kernel Talos actually ships (6.18.x) and that 'off' is the correct value, from kernel documentation or source rather than from the request. Done: nodes run 6.18.44-talos; Linux v6.18 Documentation/admin-guide/kernel-parameters.txt:5082 documents pcie_port_pm= with 'off' = 'Disable power management of all PCIe ports'; drivers/pci/pci.c:173 pcie_port_pm_setup() sets pci_bridge_d3_disable, which makes pci_bridge_d3_possible() return false for PCI_EXP_TYPE_ROOT_PORT (pci.c:2998), so drivers/pci/pcie/portdrv.c:719 never reaches pm_runtime_allow() for the port. That chain is exactly the observed failure mode.

DELIBERATE COMMENT CONTENT: the comment must say that the schematic already carries pcie_aspm=off, which disables PCIe LINK power states but does NOT stop the PORT runtime-suspending itself to D3hot, because that distinction is the reason this failure survived an argument that looks like it should have caught it.

DO NOT TOUCH pcie_aspm=off, pci=realloc, or pci=assign-busses - they are load-bearing for this same GPU path. (Verified still intact.)

DOCUMENTATION WORK FOLDED IN, as requested:

1. Confirmation addendum: docs/hardware-incidents.md recorded the power-on-order fix but never recorded that it was confirmed to work. Added: talos-3 booted 2026-08-26T11:21:12Z with the B70 enumerated, and it has held stable under production load for three days since, with every failure signature in the incident table cleared.
2. The mitigation itself, stated honestly: it prevents the race on future boots, it is NOT retroactive, and it does not remove the need for the power-on order when a card has already been lost.
3. The runbook for the attended reboot, placed where whoever performs it will actually find it.

RUNBOOK REQUIREMENTS: kernel args are not activated by 'just talos apply-node', which only stages config; they need 'just talos upgrade-node talos-3', which is talosctl upgrade -m powercycle (confirmed against talos/mod.just rather than taken on trust). That reboot is the risk event and is also when talos-3 picks up e22cfb2 from PR #1479 dropping siderolabs/thunderbolt; the captain deliberately sequenced them into one maintenance window. The runbook states: capture the GPU's PCIe state before, so there is a baseline; the host powercycle does not cycle the dock PSU, so if the card does not return the recovery is the documented power-on order (dock PSU first, wait 5-10 seconds, confirm fans and LED, then the host; never together, never host-first); exactly how to verify success afterwards (8086:e2ff and 8086:e223 present, xe bound, 00:01.0 in D0, node advertising its GPU resource again); and that this reboot doubles as the test of whether the mitigation works, so the result is worth recording back into the incident doc. Every runbook command was verified read-only against the live node.

ACCEPTED TRADEOFF, to state plainly in the PR description so nobody is surprised: the schematic is cluster-wide, so this argument reaches every node. Only talos-3 has the affected card; it takes the argument in the attended maintenance, and the others pick it up on their next natural reboot. Cost is marginally higher idle power from PCIe ports staying in D0.

ADDITIONAL WORK I JUDGED IN SCOPE (not explicitly requested, flagged for review):

• talos/AGENTS.md previously said an operator must run apply-node '(or upgrade-node)' for any *.j2 change. For schematic.yaml.j2 that is wrong in a way that fails silently, since kernel args and extensions are baked into the Image Factory image. Documented the distinction so the next person does not rediscover it.
• Added pointers to the runbook from the root AGENTS.md pre-reboot checklist and talos/AGENTS.md, so an operator about to reboot talos-3 hits the warning.
• Recorded the schematic delta the reboot closes: talos-3 runs b1a6b2ff and still has thunderbolt loaded, main is 7f25ace8, this branch is a46161e7. IDs confirmed against factory.talos.dev and the live node ExtensionStatus.

VALIDATION ALREADY RUN LOCALLY: scripts/ci/talos-validate.sh and scripts/ci/version-consistency.sh (the only two gates that read the changed files) both pass. Four python regression tests fail on missing litellm/hcl2 modules; those test files are byte-identical to the base commit and unrelated to a Talos/docs change, so they are pre-existing environmental gaps, not regressions.

STYLE CONSTRAINT: this repo's owner requires plain dash '-' and never the em dash character in authored text; verified zero em dashes in all added lines.

What Changed

• Adds pcie_port_pm=off to talos/schematic.yaml.j2 extraKernelArgs (with comment clarifying why existing pcie_aspm=off does not cover port runtime D3hot) so late-training PCIe devices are not orphaned when a root port suspends before link-up.
• Documents recovery confirmation, mitigation limits, schematic ID delta, and the attended just talos upgrade-node talos-3 runbook in docs/hardware-incidents.md, with cross-links from AGENTS.md, talos/AGENTS.md, and docs/ai-gpu-changelog.md; also corrects apply-node vs upgrade-node guidance for schematic/kernel-arg changes.
• Adds scripts/ci/schematic-pcie-port-pm-test.py to assert the rendered schematic keeps pcie_port_pm=off plus the load-bearing GPU path args and round-trips that contract through factory.talos.dev.

Tradeoff: the schematic is cluster-wide, so all nodes get the arg; only talos-3 needs it for the B70. Cost is marginally higher idle power from PCIe ports staying in D0. Activation remains an attended captain-run upgrade-node (not part of this PR).

Risk Assessment

✅ Low: Single well-documented cluster-wide kernel arg with matching runbook and no live node operations; residual activation risk is explicitly deferred to the captain's attended upgrade-node window.

Testing

Exercised the offline operator path only: rendered the schematic, confirmed pcie_port_pm=off plus preserved pcie_aspm/pci=realloc/pci=assign-busses, posted it to factory.talos.dev (id a46161e7…, base 7f25ace8…), verified Linux v6.18 documents off as disabling PCIe port PM, and re-ran talos-validate plus version-consistency green. Added and ran schematic-pcie-port-pm-test.py with a failing negative control. No node upgrade/reboot was performed, matching the explicit scope constraint.

Evidence: schematic-pcie-port-pm-test.py pass

OK: schematic renders with pcie_port_pm=off and load-bearing GPU args; factory id a46161e7… stores the same contract

OK: schematic renders with pcie_port_pm=off and load-bearing GPU args; factory id a46161e7… stores the same contract

Evidence: negative control without pcie_port_pm=off

NEGATIVE_OK: extraKernelArgs missing required entries ['pcie_port_pm=off']

NEGATIVE_OK: extraKernelArgs missing required entries ['pcie_port_pm=off']; have=['-init_on_alloc', '-init_on_free', '-selinux', 'apparmor=0', 'init_on_alloc=0', 'init_on_free=0', 'intel_iommu=on', 'iommu=pt', 'mitigations=off', 'module_blacklist=igc', 'security=none', 'sysctl.kernel.kexec_load_disabled=1', 'nvme_core.default_ps_max_latency_us=0', 'pcie_aspm=off', 'pci=realloc', 'pci=assign-busses', 'xe.force_probe=a7a0', 'i915.force_probe=!a7a0', 'talos.auditd.disabled=1']

Evidence: rendered schematic YAML

---
customization:
  extraKernelArgs:
    - -init_on_alloc # Less security, faster puter
    - -init_on_free # Less security, faster puter
    - -selinux # Less security, faster puter
    - apparmor=0 # Less security, faster puter
    - init_on_alloc=0 # Less security, faster puter
    - init_on_free=0 # Less security, faster puter
    - intel_iommu=on # PCI Passthrough
    - iommu=pt # PCI Passthrough
    - mitigations=off # Less security, faster puter
    - module_blacklist=igc # Disable onboard NIC
    # - module_blacklist=thunderbolt # Prevent Thunderbolt NIC driver from loading during early boot
    - security=none # Less security, faster puter
    - sysctl.kernel.kexec_load_disabled=1 # Meteor Lake CPU & Intel iGPU
    - nvme_core.default_ps_max_latency_us=0 # Disable NVMe APST deep sleep to prevent I/O stalls
    - pcie_aspm=off # Disable PCIe Active State Power Management to prevent link state transitions
    # pcie_aspm=off above governs ASPM *link* power states only - it does not stop the kernel
    # runtime-suspending an idle root PORT to D3hot, which is a different mechanism one layer up.
    # That gap is why this failure survived it: on talos-3 the B70's OCuLink port (00:01.0) was
    # runtime-suspended to D3hot 218 ms into boot while bus 01 was still empty, and the card only
    # finished training afterwards. With SLTCAP HotPlugCapable=0 and SLTCTL=0x0000 nothing could
    # ever rediscover it. pcie_port_pm=off sets pci_bridge_d3_disable, so pci_bridge_d3_possible()
    # returns false for root ports and portdrv never calls pm_runtime_allow() on them.
    # See docs/hardware-incidents.md [2026-08-24].
    - pcie_port_pm=off # Keep PCIe root ports out of runtime D3hot so a late-training card is never orphaned
    - pci=realloc # Reallocate PCIe resource windows for the Arc B70 Pro OCuLink switch
    - pci=assign-busses # BIOS leaves the B70's e2ff switch at [bus 00-00] with no spare bus numbers behind 00:01.0; kernel must renumber all buses (NIC renames are safe: bonds select members via LinkAliasConfig)
    - xe.force_probe=a7a0 # Bind xe to the Raptor Lake i9-13900H iGPU (8086:a7a0), which xe skips by default
    - i915.force_probe=!a7a0 # Keep i915 off the iGPU so xe owns it (load-bearing: siderolabs/i915 ships i915.ko)
    - talos.auditd.disabled=1 # Less security, faster puter
    # Bond interface kernel parameters for early network initialization
    # - bond=bond0:enp2s0f0np0,enp2s0f1np1:mode=802.3ad:miimon=100:lacp_rate=1:xmit_hash_policy=layer3+4:updelay=200:downdelay=200
    # - ip=bond0:dhcp # Enable DHCP on bond0 during early boot
  systemExtensions:
    officialExtensions:
      - siderolabs/xe # Intel iGPU + Arc discrete GPU
      # Firmware only, in practice: xe drives the a7a0 iGPU but still loads its
      # blobs from i915/ (adlp_dmc.bin, adlp_guc_70.bin), which siderolabs/xe does
      # not ship - it carries only xe/ (bmg_*, lnl_*). Dropping this in 73c9e3da
      # left talos-1/2 probing -ENOENT. i915.ko rides along but is kept off the
      # iGPU by i915.force_probe=!a7a0 above, so xe still owns it.
      - siderolabs/i915 # Intel iGPU firmware (i915/adlp_*) for the xe driver
      - siderolabs/intel-ucode # Intel iGPU
      - siderolabs/mei # Intel iGPU
      - siderolabs/nfsrahead # NFS Performance
      # siderolabs/thunderbolt is deliberately excluded: it caused connection
      # trouble on one node (unwanted DHCP on the Thunderbolt NIC at early boot)
      # and is not in use. Do not re-add it.

Evidence: factory-stored schematic (includes pcie_port_pm=off)

customization:
    extraKernelArgs:
        - -init_on_alloc
        - -init_on_free
        - -selinux
        - apparmor=0
        - init_on_alloc=0
        - init_on_free=0
        - intel_iommu=on
        - iommu=pt
        - mitigations=off
        - module_blacklist=igc
        - security=none
        - sysctl.kernel.kexec_load_disabled=1
        - nvme_core.default_ps_max_latency_us=0
        - pcie_aspm=off
        - pcie_port_pm=off
        - pci=realloc
        - pci=assign-busses
        - xe.force_probe=a7a0
        - i915.force_probe=!a7a0
        - talos.auditd.disabled=1
    systemExtensions:
        officialExtensions:
            - siderolabs/xe
            - siderolabs/i915
            - siderolabs/intel-ucode
            - siderolabs/mei
            - siderolabs/nfsrahead

Evidence: branch schematic id a46161e7…

a46161e7a33fbde0589543ccc42a80403294ffd09868d4a09c10df0b64732021

a46161e7a33fbde0589543ccc42a80403294ffd09868d4a09c10df0b64732021

Evidence: base schematic id 7f25ace8…

7f25ace820d39f0048bfd5fc6cdf882c6af2410f229d495ed2ff384c4c6807b8

7f25ace820d39f0048bfd5fc6cdf882c6af2410f229d495ed2ff384c4c6807b8

Evidence: Linux v6.18 pcie_port_pm kernel parameter docs

pcie_port_pm= [PCIE] PCIe port power management handling: off Disable power management of all PCIe ports force Forcibly enable power management of all PCIe ports

pcie_port_pm=	[PCIE] PCIe port power management handling:
		off	Disable power management of all PCIe ports
		force	Forcibly enable power management of all PCIe ports

	pcie_pme=	[PCIE,PM] Native PCIe PME signaling options:
		nomsi	Do not use MSI for 

Evidence: Linux v6.18 pcie_port_pm_setup source

static int __init pcie_port_pm_setup(char *str)
{
	if (!strcmp(str, "off"))
		pci_bridge_d3_disable = true;
	else if (!strcmp(str, "force"))
		pci_bridge_d3_force = true;
	return 1;
}

Evidence: talos-validate.sh

OK: 3 node config(s) render and validate; schematic renders

/var/folders/yr/h20mxtv56yj1c1pt9tr1_kbc0000gn/T/tmp.ZYaSwnCaOL/talos-1.yaml is valid for metal mode
/var/folders/yr/h20mxtv56yj1c1pt9tr1_kbc0000gn/T/tmp.ZYaSwnCaOL/talos-2.yaml is valid for metal mode
/var/folders/yr/h20mxtv56yj1c1pt9tr1_kbc0000gn/T/tmp.ZYaSwnCaOL/talos-3.yaml is valid for metal mode
OK: 3 node config(s) render and validate; schematic renders

Evidence: version-consistency.sh

OK: 6 machineconfig version pin(s) match the tuppr CRs (Talos v1.13.9, Kubernetes v1.36.3)

OK: 6 machineconfig version pin(s) match the tuppr CRs (Talos v1.13.9, Kubernetes v1.36.3)

Evidence: evidence summary

# pcie_port_pm=off validation evidence

## Factory schematic (end-user/Image Factory consumer)

- base (main / pre-change) schematic id: `7f25ace820d39f0048bfd5fc6cdf882c6af2410f229d495ed2ff384c4c6807b8`
- this branch schematic id: `a46161e7a33fbde0589543ccc42a80403294ffd09868d4a09c10df0b64732021`
- docs claim prefix `a46161e7`: matches=True
- docs claim main prefix `7f25ace8`: matches=True

## Active extraKernelArgs (rendered + factory-stored)

`` `
-init_on_alloc
-init_on_free
-selinux
apparmor=0
init_on_alloc=0
init_on_free=0
intel_iommu=on
iommu=pt
mitigations=off
module_blacklist=igc
security=none
sysctl.kernel.kexec_load_disabled=1
nvme_core.default_ps_max_latency_us=0
pcie_aspm=off
pcie_port_pm=off
pci=realloc
pci=assign-busses
xe.force_probe=a7a0
i915.force_probe=!a7a0
talos.auditd.disabled=1
`` `

## Required contract

- pcie_port_pm=off present: True
- pcie_aspm=off preserved: True
- pci=realloc preserved: True
- pci=assign-busses preserved: True

## Kernel v6.18 documentation (pcie_port_pm)

`` `
pcie_port_pm=	[PCIE] PCIe port power management handling:
		off	Disable power management of all PCIe ports
		force	Forcibly enable power management of all PCIe ports

	pcie_pme=	[PCIE,PM] Native PCIe PME signaling options:
		nomsi	Do not use MSI for 
`` `

## Kernel v6.18 pcie_port_pm_setup

`` `c
static int __init pcie_port_pm_setup(char *str)
{
	if (!strcmp(str, "off"))
		pci_bridge_d3_disable = true;
	else if (!strcmp(str, "force"))
		pci_bridge_d3_force = true;
	return 1;
}
`` `

## upgrade-node activates schematic via powercycle

`` `
upgrade-node node *args:
    ip="$(just talos _ip {{ node }})"
    just talos render-config {{ node }} > /dev/null # validate render before touching the node
    talosctl -n "$ip" upgrade -i "$(just talos _machine-image {{ node }})" -m powercycle --timeout=10m {{ args }}


`` `

## Scope note

No node was upgraded, rebooted, or reconfigured during this test phase.
Activation remains the captain-attended `just talos upgrade-node talos-3`.

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

• MINIJINJA_CONFIG_FILE=.minijinja.toml scripts/ci/talos-validate.sh
• scripts/ci/version-consistency.sh
• python3 scripts/ci/schematic-pcie-port-pm-test.py (render + parse extraKernelArgs + factory.talos.dev POST/GET round-trip)
• negative control: same test asserts failure when pcie_port_pm=off is stripped from rendered args
• factory schematic id check: branch a46161e7… vs base 7f25ace8… matches docs table
• Linux v6.18 kernel-parameters + pcie_port_pm_setup source confirmation that off sets pci_bridge_d3_disable
• talos/mod.just upgrade-node recipe confirms -m powercycle and _schematic-id factory POST path
• docs/hardware-incidents.md structure check for recovery confirmation, mitigation limits, and attended reboot runbook

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.