fix(talos): drop siderolabs/thunderbolt from factory schematic - #1479
Merged
Conversation
The line was added in 49a742c with a comment claiming it was intentionally removed, but it was never commented out, so every built image still ships the extension. Captain decision 2026-08-29: Thunderbolt is unused and caused connection trouble on one node. Leave a standalone comment so it is not re-added. File change only - no node apply or image build.
Aviator-Coding
added a commit
that referenced
this pull request
Aug 29, 2026
* fix(talos): add pcie_port_pm=off to stop root ports runtime-suspending The Arc Pro B70 disappeared from talos-3 because Linux runtime-suspended the OCuLink root port 00:01.0 to D3hot 218 ms into boot, while bus 01 was still empty. The card finished training afterwards (Gen4 x4, equalization complete on all 3 phases, real link partner), but SLTCAP HotPlugCapable=0 and SLTCTL=0x0000 mean every hotplug and link-state interrupt is disabled, so a port that sleeps before its device answers can never rediscover it. pcie_aspm=off was already present and does not cover this: it governs ASPM link power states, not runtime D-state suspension of the port itself. Verified against Linux v6.18 (nodes run 6.18.44-talos): - Documentation/admin-guide/kernel-parameters.txt:5082 documents pcie_port_pm= with "off - Disable power management of all PCIe ports". - drivers/pci/pci.c:173 pcie_port_pm_setup() sets pci_bridge_d3_disable, which makes pci_bridge_d3_possible() return false for PCI_EXP_TYPE_ROOT_PORT (pci.c:2998), so drivers/pci/pcie/portdrv.c never reaches pm_runtime_allow() for the port. The schematic is cluster-wide, so all three nodes get the argument. Only talos-3 has the affected card; the others pick it up on their next reboot. The tradeoff is marginally higher idle power from PCIe ports staying in D0. File change only. Kernel args ride in the factory schematic, so this needs `just talos upgrade-node talos-3` (talosctl upgrade -m powercycle) to take effect - `apply-node` only stages machine config. That attended reboot is the captain's to time; runbook lands in docs/hardware-incidents.md. * docs(talos): confirm B70 recovery, record pcie_port_pm mitigation and reboot runbook docs/hardware-incidents.md [2026-08-24] recorded the power-on-order fix but never recorded that it worked. Three additions to that entry: 1. Confirmed recovered. talos-3 booted 2026-08-26T11:21:12Z with the B70 enumerated and has held stable under production load for three days, with every failure signature in the Evidence table cleared (e2ff/e223 present, xe bound, 00:01.0 back in D0/active with runtime_suspended_time=0, bus 01-04 populated, NICs back on enp6s0f*). 182 W live draw with vllm serving. 2. The pcie_port_pm=off mitigation, stated honestly: it prevents the race on future boots, it is not retroactive, and it does not remove the need for the dock-PSU-first power-on order once a card is already lost. Includes why pcie_aspm=off never covered this (link states vs port D-state) and the Linux v6.18 source references. 3. Runbook for the single attended reboot that activates it: Ceph safety gate, PCIe baseline capture, `just talos upgrade-node talos-3`, the dock-PSU-first recovery if the card does not return, and the verification commands. Notes the reboot also carries e22cfb2 (#1479, thunderbolt drop) by deliberate sequencing, and that it doubles as the test of the mitigation so the outcome is worth recording back. Every runbook command was verified read-only against the live node. Also adds pointers where an operator will actually hit them: the root AGENTS.md pre-reboot checklist, and talos/AGENTS.md. The latter also gains the distinction that schematic.yaml.j2 changes need `upgrade-node`, not `apply-node` - kernel args are baked into the Image Factory image, so an `apply-node`d schematic change silently does nothing. * docs(talos): record the schematic delta the attended reboot closes talos-3 runs schematic b1a6b2ff (#1444) and still has thunderbolt loaded. main is 7f25ace8 (thunderbolt dropped, e22cfb2/#1479) and this change is a46161e7 (thunderbolt dropped plus pcie_port_pm=off), so the single attended reboot closes both gaps at once. IDs confirmed against factory.talos.dev and the live node's ExtensionStatus; upgrade-node still derives the target itself, so they are for confirmation only. * no-mistakes(document): Fixed stale schematic apply vs upgrade docs * no-mistakes: apply CI fixes * no-mistakes: apply CI fixes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Captain decision 2026-08-29, in the captain's own words: "We are not actively using it we have 1 node where we had issues to create a connection thats why we removed it." So the intent was genuine removal, the inline comment is correct, and the code is wrong.
talos/schematic.yaml.j2 listed
- siderolabs/thunderbolt #intentionally removed to avoid Thunderbolt NIC getting unwanted DHCP address at early boot. The item is NOT commented out, so the extension still ships in every built schematic while the comment claims it was removed. The contradiction dates to commit 49a742c (PR #877).Remove siderolabs/thunderbolt from the schematic so the built image matches the stated intent. Prefer deleting the line outright over leaving a commented-out corpse; if you keep any note, make it a clear standalone comment recording that Thunderbolt is deliberately excluded because it caused connection trouble on one node and is not in use, so nobody re-adds it.
Check whether any other template or doc references the thunderbolt extension and would now be stale - talos/machineconfig.yaml.j2, talos/nodes/*.yaml.j2, talos/AGENTS.md, and docs/. Correct anything that says the extension is present. Do not go further than that.
HARD SAFETY LIMIT: this is a FILE CHANGE ONLY. Do not run talosctl against live nodes, do not apply any machine config, do not upgrade or reboot a node, and do not trigger an image build against live hardware. Rolling this out to nodes is a separate captain decision that has NOT been given. If a node rollout is needed for correctness, say so in the PR body and stop - do not do it.
Acceptance criteria: the built schematic no longer includes siderolabs/thunderbolt; no file still claims the extension is present; repo template validation still passes (render the schematic and show the extension list in the PR body as proof); nothing was applied to any live node.
Implementation notes for the reviewer: the live list item was replaced with a standalone exclusion comment, not a commented-out list item. The already-commented kernel arg module_blacklist=thunderbolt was left as-is because it does not claim the extension is present. No other files claimed the extension was installed. Offline scripts/ci/talos-validate.sh passed. Rendered officialExtensions: siderolabs/xe, siderolabs/i915, siderolabs/intel-ucode, siderolabs/mei, siderolabs/nfsrahead. PR body must include that rendered extension list and must state that taking this off running images still needs a separate captain decision (new factory schematic ID plus just talos upgrade-node per node).
What Changed
siderolabs/thunderboltentry fromtalos/schematic.yaml.j2systemExtensions.officialExtensionsand replaced it with a standalone exclusion comment (connection trouble on one node / unwanted early-boot DHCP; not in use; do not re-add).officialExtensionsare now:siderolabs/xe,siderolabs/i915,siderolabs/intel-ucode,siderolabs/mei,siderolabs/nfsrahead.module_blacklist=thunderboltkernel arg was left as-is. Applying this to running nodes still needs a separate captain decision (new factory schematic ID plusjust talos upgrade-nodeper node)—nothing was applied to live hardware.Risk Assessment
✅ Low: Bounded template-only fix that removes the contradictory live extension entry and matches captain intent without touching live nodes or other configs.
Testing
Offline schematic render and yq-parsed officialExtensions prove thunderbolt is gone (five expected extensions only), baseline still reproduced the bug, scripts/ci/talos-validate.sh passed for all three nodes, and no other file still claims the extension is installed—file-change-only, no live node touch.
Evidence: Rendered officialExtensions after fix (clean yq values)
siderolabs/xe siderolabs/i915 siderolabs/intel-ucode siderolabs/mei siderolabs/nfsraheadEvidence: Before/after officialExtensions (base still had thunderbolt)
Evidence: Semantic extension assert (PASS)
before: ['siderolabs/xe', 'siderolabs/i915', 'siderolabs/intel-ucode', 'siderolabs/mei', 'siderolabs/nfsrahead', 'siderolabs/thunderbolt'] after: ['siderolabs/xe', 'siderolabs/i915', 'siderolabs/intel-ucode', 'siderolabs/mei', 'siderolabs/nfsrahead'] PASS: baseline had thunderbolt; fixed render drops it; list matches expected five extensionsEvidence: scripts/ci/talos-validate.sh
OK: 3 node config(s) render and validate; schematic rendersEvidence: Full rendered schematic.yaml
Evidence: Stale reference check (only exclusion comments remain)
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
bash scripts/ci/talos-validate.sh(3 node metal validates + schematic render)minijinja-cli talos/schematic.yaml.j2 | yq '.customization.systemExtensions.officialExtensions[]'on HEAD vs base30fdc87aSemantic before/after assertion: baseline list still containedsiderolabs/thunderbolt; HEAD list is exactly xe, i915, intel-ucode, mei, nfsraheadRepo-wide check for live- siderolabs/thunderboltlist items and other docs/template presence claimsConfirmed no talosctl apply/upgrade/reboot or factory schematic POST against live hardware✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.