fix(windows): write TLS bootstrap-config in NodePrep, not BasePrep by benjamin-brady · Pull Request #8795 · Azure/AgentBaker

benjamin-brady · 2026-06-30T01:38:15Z

Work item: AB#38630884

Summary

Fixes AB#38630884: PIS / VHD-caching Windows nodes fail to register with cannot create certificate signing request: Unauthorized.

Root cause

On PIS/VHD-cached clusters the Windows VHD is baked from a temporary VMSS with TLS bootstrapping enabled. During the bake, BasePrep writes c:\k\bootstrap-config with the bake-time TLS bootstrap token and drops a C:\AzureData\base_prep.complete marker — both captured into the cached VHD.

On the real node the CSE skips BasePrep because that marker exists, so bootstrap-config keeps the stale baked token instead of the fresh $global:TLSBootstrapToken delivered via CustomData. The apiserver then rejects the kubelet's first CSR as Unauthorized.

The bootstrap-config write was simply left in BasePrep when the script was split into BasePrep/NodePrep (commit 63b9d9b9, #6601).

Fix

Move Write-BootstrapKubeConfig from BasePrep into NodePrep, before Install-KubernetesServices. kubelet (installed/started by Install-KubernetesServices) is the only consumer of bootstrap-config, so writing it in NodePrep guarantees the live token is used on every real provision.

No VHD rebake required — the Windows CSE script is rendered fresh into CustomData by the RP on every provision; only the on-disk bootstrap-config file is baked. The corrected NodePrep write takes effect on new nodes from existing cached VHDs.
A move (not keep-and-refresh) is correct — BasePrep never starts kubelet nor connects to the API server, so nothing there consumes the token-based bootstrap-config. Calico in BasePrep uses the separate cert-based config from Write-KubeConfig, which is unchanged.

Validation

go test ./pkg/agent/... passes; make generate-testdata produced no snapshot drift.
Suggested manual validation (from the work item):
- Provision a Windows node from a PIS cached VHD; confirm c:\k\bootstrap-config token == $global:TLSBootstrapToken and the node reaches Ready.
- Rotate the agent pool TLS bootstrap token, provision again from the same cached VHD, and confirm the node still joins.

Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com

PIS/VHD-cached Windows nodes fail to register because the on-disk bootstrap kubeconfig (c:\k\bootstrap-config) holds the stale bake-time TLS bootstrap token. BasePrep wrote bootstrap-config, but it is skipped on cached nodes (base_prep.complete marker), so the live token from $global:TLSBootstrapToken never overwrote the baked one and the apiserver rejected the kubelet CSR as Unauthorized. kubelet, installed/started by Install-KubernetesServices in NodePrep, is the only consumer of bootstrap-config. Move Write-BootstrapKubeConfig from BasePrep into NodePrep (before Install-KubernetesServices) so it is written from the live token on every real provision. No VHD rebake is required since the CSE script is rendered fresh into CustomData each boot. The Calico setup in BasePrep uses the separate cert-based 'config' from Write-KubeConfig, which is unchanged, so BasePrep keeps working. Work item: 38630884 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR fixes a Windows node registration failure for PIS / VHD-cached clusters where a stale, bake-time TLS bootstrap token in c:\k\bootstrap-config causes kubelet CSR requests to be rejected as Unauthorized. The fix aligns bootstrap-config generation with the actual provisioning phase so the live $global:TLSBootstrapToken from CustomData is always used when kubelet starts.

Changes:

Move Write-BootstrapKubeConfig from BasePrep to NodePrep.
Ensure bootstrap-config is written before Install-KubernetesServices (i.e., before kubelet is started) during real node provisioning.

Same PIS/VHD-cache class of bug as the bootstrap-config move. BasePrep wrote c:\k\config (the cert-based kubeconfig from Write-KubeConfig) with bake-time cluster material (CA, server, AgentCertificate). BasePrep is skipped on cached nodes, so on a PIS node config keeps the bake cluster's values and is never refreshed. config is only consumed in NodePrep: - Calico's GetCalicoKubeConfig (Start-InstallCalico) uses it to reach the API server, then it is removed at the end of NodePrep; and - in legacy cert-auth mode (no token, no secure bootstrap) it is kubelet's own kubeconfig. Move Write-KubeConfig (and its temporary/permanent log block) into NodePrep, before Install-KubernetesServices, so it is written from the live values on every real provision. No consumer of config exists in BasePrep, so the move is safe; the existing conditional removal at the end of NodePrep is unchanged. Work item: 38630884 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Trim the verbose per-kubeconfig comment to a short rationale so it is less likely to drift from the code as consumers change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Add a teeth-y regression guard for the BasePrep->NodePrep kubeconfig fix. The VHDCaching legacy-TLS-bootstrap scenario now injects a stale sentinel bootstrap token at the BAKE (pre-provision) stage only, while the real cluster token is used at provision time. If bootstrap-config were written in BasePrep (the bug), the cached VHD would carry the stale token and the node would fail to register; because it is written in NodePrep, the live token wins. The validator asserts bootstrap-config contains the live token and never the sentinel. Adds Config.PreProvisionBootstrapConfigMutator, a bake-stage-only hook, wired into runScenarioWithPreProvision. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Move remaining cluster-specific Windows config writes to NodePrep so PIS cached VHDs do not bake stale kubeconfig, cert, azurestack, or azure.json data. NodePrep regenerates the files from live CustomData before runtime consumers start. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

…compat The CSE scripts package cached on the VHD (currently v0.0.52, pinned via windowsCSEScriptsPackageURL) configures containerd during BasePrep and reads the pause image from C:\k\kubeclusterconfig.json. Moving Write-KubeClusterConfig entirely to NodePrep meant the file did not exist when the cached containerd logic ran, failing CSE with 'Cannot find path C:\k\kubeclusterconfig.json' on both cached and non-cached Windows nodes (a new-CSE / old-VHD-scripts skew). Write kubeclusterconfig.json in BasePrep again so the cached scripts find it, while still rewriting it from live CustomData in NodePrep so PIS/VHD-cached nodes (which skip BasePrep) refresh the cluster-specific values. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

benjamin-brady · 2026-07-01T00:49:53Z

The E2E tests that cover windows and windows VHD caching passed.

Copilot AI review requested due to automatic review settings June 30, 2026 01:38

benjamin-brady requested review from AbelHu, Devinwong, SriHarsha001, awesomenix, calvin197, cameronmeissner, djsly, ganeshkumarashok, lilypan26, mxj220, pdamianov-dev, phealy, r2k1, runzhen, sulixu, surajssd, timmy-wright, xuexu6666 and zachary-bailey as code owners June 30, 2026 01:38

Copilot started reviewing on behalf of benjamin-brady June 30, 2026 01:38 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

benjamin-brady and others added 2 commits June 30, 2026 14:21

docs(windows): condense NodePrep kubeconfig comment

e0c6105

Trim the verbose per-kubeconfig comment to a short rationale so it is less likely to drift from the code as consumers change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 30, 2026 02:53

Copilot started reviewing on behalf of benjamin-brady June 30, 2026 02:53 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

Comment thread parts/windows/kuberneteswindowssetup.ps1 Outdated

benjamin-brady and others added 2 commits June 30, 2026 15:34

Apply suggestions from code review

176e1ad

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 30, 2026 07:12

Copilot started reviewing on behalf of benjamin-brady June 30, 2026 07:13 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

Comment thread e2e/scenario_win_test.go

timmy-wright approved these changes Jun 30, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings June 30, 2026 23:15

Copilot started reviewing on behalf of benjamin-brady June 30, 2026 23:16 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

benjamin-brady force-pushed the bbrady/windows-pis-bootstrap-token branch from ff17e16 to 46181f2 Compare June 30, 2026 23:39

timmy-wright approved these changes Jul 1, 2026

View reviewed changes

benjamin-brady enabled auto-merge (squash) July 1, 2026 00:48

r2k1 approved these changes Jul 1, 2026

View reviewed changes

benjamin-brady merged commit 1a0c644 into main Jul 1, 2026
31 of 32 checks passed

benjamin-brady deleted the bbrady/windows-pis-bootstrap-token branch July 1, 2026 02:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(windows): write TLS bootstrap-config in NodePrep, not BasePrep#8795

fix(windows): write TLS bootstrap-config in NodePrep, not BasePrep#8795
benjamin-brady merged 7 commits into
mainfrom
bbrady/windows-pis-bootstrap-token

benjamin-brady commented Jun 30, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

benjamin-brady commented Jul 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

benjamin-brady commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root cause

Fix

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

benjamin-brady commented Jul 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

benjamin-brady commented Jun 30, 2026 •

edited

Loading