docs(kubelet): clarify config file generation precedence with tests and variable naming

[abigailliang-aks-sig-node]

Summary

Document and test the existing kubelet config file generation precedence model in the legacy CSE path. Adds variable rename (kubeletConfig → flagConfig) for clarity, error handling for json.MarshalIndent, and unit tests explicitly validating the three-layer precedence: CustomKubeletConfig wins > flags backfill > v1beta1 defaults. No behavioral changes. Scriptless path flag→config file sync to follow in a separate PR.

Why this matters

When kubelet config file mode is enabled, translated flags (e.g. --max-pods, --cluster-dns, --event-qps) are removed from CLI and only exist in the config file. The config file needs ~40 fields populated, but CustomKubeletConfig only covers ~12 user-configurable fields. The remaining ~28 fields must come from the RP-provided KubeletConfig flag map — otherwise kubelet falls back to v1beta1 defaults which are often not what AKS intends.

Precedence model

┌─────────────────────────────────────────────────────────────────┐
│              kubelet config file generation                       │
│                                                                   │
│  Priority 1 (highest): CustomKubeletConfig                        │
│  ┌─────────────────────────────────────────────────────────┐      │
│  │ User-specified via AKS API (~12 fields)                 │      │
│  │ e.g. ImageGcHighThreshold=90, SeccompDefault=true       │      │
│  │                                                         │      │
│  │ → wins when both sources set the same field             │      │
│  └─────────────────────────────────────────────────────────┘      │
│                          │                                        │
│                          ▼                                        │
│  Priority 2: KubeletConfig flags                                  │
│  ┌─────────────────────────────────────────────────────────┐      │
│  │ RP-provided defaults from CLI flag map (~40 fields)     │      │
│  │ e.g. --max-pods=110, --cluster-dns=172.16.0.10          │      │
│  │                                                         │      │
│  │ → backfills the ~28 fields not set by CustomKubeletConfig│     │
│  └─────────────────────────────────────────────────────────┘      │
│                          │                                        │
│                          ▼                                        │
│  Priority 3 (lowest): kubelet v1beta1 defaults                    │
│  ┌─────────────────────────────────────────────────────────┐      │
│  │ Built-in kubelet defaults                               │      │
│  │ e.g. eventRecordQPS=5, imageGCHighThreshold=85          │      │
│  │                                                         │      │
│  │ → used when neither source sets the field               │      │
│  └─────────────────────────────────────────────────────────┘      │
└─────────────────────────────────────────────────────────────────┘

Example:

Field	CustomKC	Flags	Config file gets
imageGCHighThresholdPercent	✅ user set 90	--image-gc-high-threshold=85	90 (CustomKC wins)
maxPods	❌ not set	--max-pods=110	110 (backfilled from flags)
clusterDNS	❌ not set	--cluster-dns=172.16.0.10	["172.16.0.10"] (backfilled)
evictionHard	❌ not set	--eviction-hard=memory.available<750Mi,...	{...} (backfilled)
eventRecordQPS	❌ not set	--event-qps=0	0 (backfilled, not v1beta1 default of 5)

This precedence is consistent with existing behavior — setCustomKubeletConfig() has always overwritten flag-derived values. The pre-existing test TestGetKubeletConfigFileCustomKCShouldOverrideValuesPassedInKc explicitly validates this.

Flag classification: config file vs CLI-only

When config file mode is enabled, flags are split into two categories:

Translated to config file (TranslatedKubeletConfigFlags, 37 flags) — removed from CLI, written to /etc/default/kubeletconfig.json:

Flag	Config file field
--address	address
--anonymous-auth	authentication.anonymous.enabled
--authentication-token-webhook	authentication.webhook.enabled
--authorization-mode	authorization.mode
--client-ca-file	authentication.x509.clientCAFile
--cgroups-per-qos	cgroupsPerQOS
--cluster-dns	clusterDNS
--cluster-domain	clusterDomain
--container-log-max-files	containerLogMaxFiles
--container-log-max-size	containerLogMaxSize
--cpu-cfs-quota	cpuCFSQuota
--cpu-cfs-quota-period	cpuCFSQuotaPeriod
--cpu-manager-policy	cpuManagerPolicy
--enforce-node-allocatable	enforceNodeAllocatable
--event-qps	eventRecordQPS
--eviction-hard	evictionHard
--eviction-max-pod-grace-period	evictionMaxPodGracePeriod
--eviction-soft	evictionSoft
--eviction-soft-grace-period	evictionSoftGracePeriod
--fail-swap-on	failSwapOn
--feature-gates	featureGates
--image-gc-high-threshold	imageGCHighThresholdPercent
--image-gc-low-threshold	imageGCLowThresholdPercent
--kube-reserved	kubeReserved
--kube-reserved-cgroup	kubeReservedCgroup
--max-pods	maxPods
--node-status-update-frequency	nodeStatusUpdateFrequency
--node-status-report-frequency	nodeStatusReportFrequency (omitted from CLI)
--pod-manifest-path	staticPodPath
--pod-max-pids	podPidsLimit
--protect-kernel-defaults	protectKernelDefaults
--read-only-port	readOnlyPort
--resolv-conf	resolvConf
--rotate-certificates	rotateCertificates
--rotate-server-certificates	serverTLSBootstrap
--serialize-image-pulls	serializeImagePulls
--streaming-connection-idle-timeout	streamingConnectionIdleTimeout
--system-reserved	systemReserved
--system-reserved-cgroup	systemReservedCgroup
--tls-cert-file	tlsCertFile
--tls-cipher-suites	tlsCipherSuites
--tls-private-key-file	tlsPrivateKeyFile
--topology-manager-policy	topologyManagerPolicy
--allowed-unsafe-sysctls	allowedUnsafeSysctls

CLI-only — no corresponding field in KubeletConfiguration schema, must stay on command line:

Flag	Reason
--cloud-provider=external	Deprecated/removed from KubeletConfiguration in K8s 1.29+
--kubeconfig	Bootstrap prerequisite — kubelet needs this to connect to API server before it can read config
--bootstrap-kubeconfig	Same as above — TLS bootstrap path
--node-ip	Not a KubeletConfiguration field
--container-runtime-endpoint	Set via systemd drop-in, not in KubeletConfiguration
--runtime-cgroups	Set via systemd drop-in
--runtime-request-timeout	Set via systemd drop-in
--cgroup-driver	Set via systemd drop-in
--volume-plugin-dir	Set in kubelet.service unit
--enable-server	Set in kubelet.service unit
--node-labels	Set in kubelet.service unit from KUBELET_NODE_LABELS
--v (verbosity)	Set in kubelet.service unit

Changes

• pkg/agent/utils.go: Rename kubeletConfig → flagConfig to clarify its role in the three-way flow (flagConfig → setCustomKubeletConfig → output). Add error handling for json.MarshalIndent.
• pkg/agent/utils_test.go: Add TestGetKubeletConfigFileContent_PrecedenceRules and TestGetKubeletConfigFileContent_MergesFlagsWithoutOverwritingContent documenting the precedence model.

Future work (follow-up PR)

• Scriptless path (aks-node-controller): flag→config file sync with conflict detection for all translated flags

---

Test Plan

Unit Tests

go test ./pkg/agent/ -run "KubeletConfigFile|OrderedKubeletConfigFlag" -v

All existing tests pass without modification, including:

• TestGetKubeletConfigFileContent_PrecedenceRules — validates three-layer precedence (CustomKC wins > flags backfill > v1beta1 defaults)
• TestGetKubeletConfigFileContent_MergesFlagsWithoutOverwritingContent — CustomKC value preserved, flags fill missing fields
• TestGetKubeletConfigFileCustomKCShouldOverrideValuesPassedInKc — pre-existing test confirming CustomKC overrides flags

E2E Tests

#	Scenario	Command	Purpose
1	CSE path + CustomKubeletConfig	go test -run '^Test_Ubuntu2204_KubeletCustomConfig/default$'	Validates the change: config file has "eventRecordQPS": 0 (backfilled from flags) and "failSwapOn": true (CustomKC); CLI flags only have --cloud-provider, --kubeconfig, --node-ip
2	No CustomKubeletConfig	go test -run '^Test_Ubuntu2404Gen2/default$'	Regression: without CustomKC the merge logic is not triggered; no config file exists, all ~30 flags remain on CLI

E2E Manual Verification (SSH into VM)

# Verify config file content (Test #1)
sudo cat /etc/default/kubeletconfig.json | python3 -m json.tool

# Verify CLI flags (should only have non-translated flags)
sudo cat /etc/default/kubelet

# Verify kubelet process args
ps aux | grep kubelet | grep -v grep | head -1

Scope Note

No changes to aks-node-controller/ in this PR. Scriptless path flag→config file sync will be addressed in a follow-up PR.

[Copilot]

Pull request overview

This PR updates how pkg/agent generates kubelet config-file JSON by merging sources: start from customKubeletConfig content and only backfill missing fields from flag-derived values, while keeping the existing kubelet CLI flag filtering behavior unchanged.

Changes:

• Refactors GetKubeletConfigFileContent to build a content-derived config first, then merge in missing fields from the flag-derived config.
• Introduces helper functions to merge config content and flags via a map-based merge.
• Adds a unit test to verify that content values are not overwritten while missing fields are filled from flags.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
pkg/agent/utils.go	Refactors kubelet config-file generation and adds helper-based merge logic; keeps CLI filtering semantics while preparing for a future “config-file authoritative” shift.
pkg/agent/utils_test.go	Adds a unit test validating that content wins on conflicts and flags backfill missing fields.

[abigailliang-aks-sig-node]

@abigailliang-aks-sig-node please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree company="Microsoft"

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[awesomenix]

what was wrong with previous name here? kubeletConfig seems accurate isnt it?

[abigailliang-aks-sig-node]

changed the name to flagConfig to emphasize that this struct is derived from flags and will subsequently be overlaid by CustomKC—aligning with the "step 2a: flag-derived → step 2b: CustomKC overlay" flow described in the PR summary.

overall Im fine with both name, I can revert the rename if needed.

[lilypan26]

i also prefer the old naming, i think kubeletConfig is more representative of what it is

[abigailliang-aks-sig-node]

done