Skip to content

fix(talos): drop siderolabs/thunderbolt from factory schematic - #1479

Merged
Aviator-Coding merged 1 commit into
mainfrom
fm/homeops-talos-thunderbolt-remove
Aug 29, 2026
Merged

fix(talos): drop siderolabs/thunderbolt from factory schematic#1479
Aviator-Coding merged 1 commit into
mainfrom
fm/homeops-talos-thunderbolt-remove

Conversation

@Aviator-Coding

Copy link
Copy Markdown
Owner

Intent

Captain decision 2026-08-29, in the captain's own words: "We are not actively using it we have 1 node where we had issues to create a connection thats why we removed it." So the intent was genuine removal, the inline comment is correct, and the code is wrong.

talos/schematic.yaml.j2 listed - siderolabs/thunderbolt #intentionally removed to avoid Thunderbolt NIC getting unwanted DHCP address at early boot. The item is NOT commented out, so the extension still ships in every built schematic while the comment claims it was removed. The contradiction dates to commit 49a742c (PR #877).

Remove siderolabs/thunderbolt from the schematic so the built image matches the stated intent. Prefer deleting the line outright over leaving a commented-out corpse; if you keep any note, make it a clear standalone comment recording that Thunderbolt is deliberately excluded because it caused connection trouble on one node and is not in use, so nobody re-adds it.

Check whether any other template or doc references the thunderbolt extension and would now be stale - talos/machineconfig.yaml.j2, talos/nodes/*.yaml.j2, talos/AGENTS.md, and docs/. Correct anything that says the extension is present. Do not go further than that.

HARD SAFETY LIMIT: this is a FILE CHANGE ONLY. Do not run talosctl against live nodes, do not apply any machine config, do not upgrade or reboot a node, and do not trigger an image build against live hardware. Rolling this out to nodes is a separate captain decision that has NOT been given. If a node rollout is needed for correctness, say so in the PR body and stop - do not do it.

Acceptance criteria: the built schematic no longer includes siderolabs/thunderbolt; no file still claims the extension is present; repo template validation still passes (render the schematic and show the extension list in the PR body as proof); nothing was applied to any live node.

Implementation notes for the reviewer: the live list item was replaced with a standalone exclusion comment, not a commented-out list item. The already-commented kernel arg module_blacklist=thunderbolt was left as-is because it does not claim the extension is present. No other files claimed the extension was installed. Offline scripts/ci/talos-validate.sh passed. Rendered officialExtensions: siderolabs/xe, siderolabs/i915, siderolabs/intel-ucode, siderolabs/mei, siderolabs/nfsrahead. PR body must include that rendered extension list and must state that taking this off running images still needs a separate captain decision (new factory schematic ID plus just talos upgrade-node per node).

What Changed

  • Removed the live siderolabs/thunderbolt entry from talos/schematic.yaml.j2 systemExtensions.officialExtensions and replaced it with a standalone exclusion comment (connection trouble on one node / unwanted early-boot DHCP; not in use; do not re-add).
  • Built schematic officialExtensions are now: siderolabs/xe, siderolabs/i915, siderolabs/intel-ucode, siderolabs/mei, siderolabs/nfsrahead.
  • No other templates or docs claimed the extension was present; the existing commented module_blacklist=thunderbolt kernel arg was left as-is. Applying this to running nodes still needs a separate captain decision (new factory schematic ID plus just talos upgrade-node per node)—nothing was applied to live hardware.

Risk Assessment

✅ Low: Bounded template-only fix that removes the contradictory live extension entry and matches captain intent without touching live nodes or other configs.

Testing

Offline schematic render and yq-parsed officialExtensions prove thunderbolt is gone (five expected extensions only), baseline still reproduced the bug, scripts/ci/talos-validate.sh passed for all three nodes, and no other file still claims the extension is installed—file-change-only, no live node touch.

Evidence: Rendered officialExtensions after fix (clean yq values)

siderolabs/xe siderolabs/i915 siderolabs/intel-ucode siderolabs/mei siderolabs/nfsrahead

siderolabs/xe
siderolabs/i915
siderolabs/intel-ucode
siderolabs/mei
siderolabs/nfsrahead
Evidence: Before/after officialExtensions (base still had thunderbolt)
=== BEFORE (base 30fdc87a) officialExtensions ===
- siderolabs/xe # Intel iGPU + Arc discrete GPU
# Firmware only, in practice: xe drives the a7a0 iGPU but still loads its
# blobs from i915/ (adlp_dmc.bin, adlp_guc_70.bin), which siderolabs/xe does
# not ship - it carries only xe/ (bmg_*, lnl_*). Dropping this in 73c9e3da
# left talos-1/2 probing -ENOENT. i915.ko rides along but is kept off the
# iGPU by i915.force_probe=!a7a0 above, so xe still owns it.
- siderolabs/i915 # Intel iGPU firmware (i915/adlp_*) for the xe driver
- siderolabs/intel-ucode # Intel iGPU
- siderolabs/mei # Intel iGPU
- siderolabs/nfsrahead # NFS Performance
- siderolabs/thunderbolt #intentionally removed to avoid Thunderbolt NIC getting unwanted DHCP address at early boot

=== AFTER (HEAD a812a3b2) officialExtensions ===
- siderolabs/xe # Intel iGPU + Arc discrete GPU
# Firmware only, in practice: xe drives the a7a0 iGPU but still loads its
# blobs from i915/ (adlp_dmc.bin, adlp_guc_70.bin), which siderolabs/xe does
# not ship - it carries only xe/ (bmg_*, lnl_*). Dropping this in 73c9e3da
# left talos-1/2 probing -ENOENT. i915.ko rides along but is kept off the
# iGPU by i915.force_probe=!a7a0 above, so xe still owns it.
- siderolabs/i915 # Intel iGPU firmware (i915/adlp_*) for the xe driver
- siderolabs/intel-ucode # Intel iGPU
- siderolabs/mei # Intel iGPU
- siderolabs/nfsrahead # NFS Performance
# siderolabs/thunderbolt is deliberately excluded: it caused connection
# trouble on one node (unwanted DHCP on the Thunderbolt NIC at early boot)
# and is not in use. Do not re-add it.

=== delta ===
removed: siderolabs/thunderbolt (was present in base render, absent in HEAD render)
Evidence: Semantic extension assert (PASS)

before: ['siderolabs/xe', 'siderolabs/i915', 'siderolabs/intel-ucode', 'siderolabs/mei', 'siderolabs/nfsrahead', 'siderolabs/thunderbolt'] after: ['siderolabs/xe', 'siderolabs/i915', 'siderolabs/intel-ucode', 'siderolabs/mei', 'siderolabs/nfsrahead'] PASS: baseline had thunderbolt; fixed render drops it; list matches expected five extensions

before: ['siderolabs/xe', 'siderolabs/i915', 'siderolabs/intel-ucode', 'siderolabs/mei', 'siderolabs/nfsrahead', 'siderolabs/thunderbolt']
after: ['siderolabs/xe', 'siderolabs/i915', 'siderolabs/intel-ucode', 'siderolabs/mei', 'siderolabs/nfsrahead']
PASS: baseline had thunderbolt; fixed render drops it; list matches expected five extensions
Evidence: scripts/ci/talos-validate.sh

OK: 3 node config(s) render and validate; schematic renders

/var/folders/yr/h20mxtv56yj1c1pt9tr1_kbc0000gn/T/tmp.iJZoowhG3G/talos-1.yaml is valid for metal mode
/var/folders/yr/h20mxtv56yj1c1pt9tr1_kbc0000gn/T/tmp.iJZoowhG3G/talos-2.yaml is valid for metal mode
/var/folders/yr/h20mxtv56yj1c1pt9tr1_kbc0000gn/T/tmp.iJZoowhG3G/talos-3.yaml is valid for metal mode
OK: 3 node config(s) render and validate; schematic renders
Evidence: Full rendered schematic.yaml
---
customization:
  extraKernelArgs:
    - -init_on_alloc # Less security, faster puter
    - -init_on_free # Less security, faster puter
    - -selinux # Less security, faster puter
    - apparmor=0 # Less security, faster puter
    - init_on_alloc=0 # Less security, faster puter
    - init_on_free=0 # Less security, faster puter
    - intel_iommu=on # PCI Passthrough
    - iommu=pt # PCI Passthrough
    - mitigations=off # Less security, faster puter
    - module_blacklist=igc # Disable onboard NIC
    # - module_blacklist=thunderbolt # Prevent Thunderbolt NIC driver from loading during early boot
    - security=none # Less security, faster puter
    - sysctl.kernel.kexec_load_disabled=1 # Meteor Lake CPU & Intel iGPU
    - nvme_core.default_ps_max_latency_us=0 # Disable NVMe APST deep sleep to prevent I/O stalls
    - pcie_aspm=off # Disable PCIe Active State Power Management to prevent link state transitions
    - pci=realloc # Reallocate PCIe resource windows for the Arc B70 Pro OCuLink switch
    - pci=assign-busses # BIOS leaves the B70's e2ff switch at [bus 00-00] with no spare bus numbers behind 00:01.0; kernel must renumber all buses (NIC renames are safe: bonds select members via LinkAliasConfig)
    - xe.force_probe=a7a0 # Bind xe to the Raptor Lake i9-13900H iGPU (8086:a7a0), which xe skips by default
    - i915.force_probe=!a7a0 # Keep i915 off the iGPU so xe owns it (load-bearing: siderolabs/i915 ships i915.ko)
    - talos.auditd.disabled=1 # Less security, faster puter
    # Bond interface kernel parameters for early network initialization
    # - bond=bond0:enp2s0f0np0,enp2s0f1np1:mode=802.3ad:miimon=100:lacp_rate=1:xmit_hash_policy=layer3+4:updelay=200:downdelay=200
    # - ip=bond0:dhcp # Enable DHCP on bond0 during early boot
  systemExtensions:
    officialExtensions:
      - siderolabs/xe # Intel iGPU + Arc discrete GPU
      # Firmware only, in practice: xe drives the a7a0 iGPU but still loads its
      # blobs from i915/ (adlp_dmc.bin, adlp_guc_70.bin), which siderolabs/xe does
      # not ship - it carries only xe/ (bmg_*, lnl_*). Dropping this in 73c9e3da
      # left talos-1/2 probing -ENOENT. i915.ko rides along but is kept off the
      # iGPU by i915.force_probe=!a7a0 above, so xe still owns it.
      - siderolabs/i915 # Intel iGPU firmware (i915/adlp_*) for the xe driver
      - siderolabs/intel-ucode # Intel iGPU
      - siderolabs/mei # Intel iGPU
      - siderolabs/nfsrahead # NFS Performance
      # siderolabs/thunderbolt is deliberately excluded: it caused connection
      # trouble on one node (unwanted DHCP on the Thunderbolt NIC at early boot)
      # and is not in use. Do not re-add it.
Evidence: Stale reference check (only exclusion comments remain)
=== all thunderbolt mentions in repo (excl .git) ===
./talos/schematic.yaml.j2:14:    # - module_blacklist=thunderbolt # Prevent Thunderbolt NIC driver from loading during early boot
./talos/schematic.yaml.j2:39:      # siderolabs/thunderbolt is deliberately excluded: it caused connection
./talos/schematic.yaml.j2:40:      # trouble on one node (unwanted DHCP on the Thunderbolt NIC at early boot)

=== live YAML list items '- siderolabs/thunderbolt' ===
PASS: none

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • bash scripts/ci/talos-validate.sh (3 node metal validates + schematic render)
  • minijinja-cli talos/schematic.yaml.j2 | yq '.customization.systemExtensions.officialExtensions[]' on HEAD vs base 30fdc87a
  • Semantic before/after assertion: baseline list still contained siderolabs/thunderbolt; HEAD list is exactly xe, i915, intel-ucode, mei, nfsrahead
  • Repo-wide check for live - siderolabs/thunderbolt list items and other docs/template presence claims
  • Confirmed no talosctl apply/upgrade/reboot or factory schematic POST against live hardware
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

The line was added in 49a742c with a comment claiming it was
intentionally removed, but it was never commented out, so every
built image still ships the extension.

Captain decision 2026-08-29: Thunderbolt is unused and caused
connection trouble on one node. Leave a standalone comment so
it is not re-added. File change only - no node apply or image
build.
@mortyops mortyops Bot added the area/talos label Aug 29, 2026
@Aviator-Coding
Aviator-Coding merged commit e22cfb2 into main Aug 29, 2026
7 checks passed
@Aviator-Coding
Aviator-Coding deleted the fm/homeops-talos-thunderbolt-remove branch August 29, 2026 03:58
Aviator-Coding added a commit that referenced this pull request Aug 29, 2026
* fix(talos): add pcie_port_pm=off to stop root ports runtime-suspending

The Arc Pro B70 disappeared from talos-3 because Linux runtime-suspended
the OCuLink root port 00:01.0 to D3hot 218 ms into boot, while bus 01 was
still empty. The card finished training afterwards (Gen4 x4, equalization
complete on all 3 phases, real link partner), but SLTCAP HotPlugCapable=0
and SLTCTL=0x0000 mean every hotplug and link-state interrupt is disabled,
so a port that sleeps before its device answers can never rediscover it.

pcie_aspm=off was already present and does not cover this: it governs ASPM
link power states, not runtime D-state suspension of the port itself.

Verified against Linux v6.18 (nodes run 6.18.44-talos):
- Documentation/admin-guide/kernel-parameters.txt:5082 documents
  pcie_port_pm= with "off - Disable power management of all PCIe ports".
- drivers/pci/pci.c:173 pcie_port_pm_setup() sets pci_bridge_d3_disable,
  which makes pci_bridge_d3_possible() return false for PCI_EXP_TYPE_ROOT_PORT
  (pci.c:2998), so drivers/pci/pcie/portdrv.c never reaches pm_runtime_allow()
  for the port.

The schematic is cluster-wide, so all three nodes get the argument. Only
talos-3 has the affected card; the others pick it up on their next reboot.
The tradeoff is marginally higher idle power from PCIe ports staying in D0.

File change only. Kernel args ride in the factory schematic, so this needs
`just talos upgrade-node talos-3` (talosctl upgrade -m powercycle) to take
effect - `apply-node` only stages machine config. That attended reboot is
the captain's to time; runbook lands in docs/hardware-incidents.md.

* docs(talos): confirm B70 recovery, record pcie_port_pm mitigation and reboot runbook

docs/hardware-incidents.md [2026-08-24] recorded the power-on-order fix but
never recorded that it worked. Three additions to that entry:

1. Confirmed recovered. talos-3 booted 2026-08-26T11:21:12Z with the B70
   enumerated and has held stable under production load for three days, with
   every failure signature in the Evidence table cleared (e2ff/e223 present,
   xe bound, 00:01.0 back in D0/active with runtime_suspended_time=0, bus
   01-04 populated, NICs back on enp6s0f*). 182 W live draw with vllm serving.

2. The pcie_port_pm=off mitigation, stated honestly: it prevents the race on
   future boots, it is not retroactive, and it does not remove the need for
   the dock-PSU-first power-on order once a card is already lost. Includes
   why pcie_aspm=off never covered this (link states vs port D-state) and the
   Linux v6.18 source references.

3. Runbook for the single attended reboot that activates it: Ceph safety
   gate, PCIe baseline capture, `just talos upgrade-node talos-3`, the
   dock-PSU-first recovery if the card does not return, and the verification
   commands. Notes the reboot also carries e22cfb2 (#1479, thunderbolt
   drop) by deliberate sequencing, and that it doubles as the test of the
   mitigation so the outcome is worth recording back.

Every runbook command was verified read-only against the live node.

Also adds pointers where an operator will actually hit them: the root
AGENTS.md pre-reboot checklist, and talos/AGENTS.md. The latter also gains
the distinction that schematic.yaml.j2 changes need `upgrade-node`, not
`apply-node` - kernel args are baked into the Image Factory image, so an
`apply-node`d schematic change silently does nothing.

* docs(talos): record the schematic delta the attended reboot closes

talos-3 runs schematic b1a6b2ff (#1444) and still has thunderbolt loaded.
main is 7f25ace8 (thunderbolt dropped, e22cfb2/#1479) and this change is
a46161e7 (thunderbolt dropped plus pcie_port_pm=off), so the single attended
reboot closes both gaps at once. IDs confirmed against factory.talos.dev and
the live node's ExtensionStatus; upgrade-node still derives the target itself,
so they are for confirmation only.

* no-mistakes(document): Fixed stale schematic apply vs upgrade docs

* no-mistakes: apply CI fixes

* no-mistakes: apply CI fixes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant