feat(cse): emit kubelet active settings as a structured JSON Kusto event

[abigailliang-aks-sig-node]

Summary

Emit kubelet's effective runtime configuration (CLI flags + config file settings) as a structured JSON event in GuestAgentGenericLogs, queryable via parse_json() from Kusto.

Motivation

There is currently no easy way to query kubelet's active settings across nodes in Kusto. Debugging requires SSH-ing into individual nodes and running journalctl -u kubelet. This event enables a single Kusto query to surface kubelet configuration for all nodes in a cluster.

Design: one event per boot is sufficient

• Kubelet's startup flags/config are constant within a single boot — unlike memory usage, they don't drift, so there is no meaningful timechart to draw over a constant.
• Configuration only changes on restart/reconfig — and every restart re-runs CSE, which re-emits the event. So "one event per boot" naturally covers every real configuration change point.

Why shell-only (no Go-side changes needed)

In both traditional and scriptless phase 2 modes, the shell scripts (cse_main.sh → ensureKubelet()) are always executed:

• Traditional mode: CSE extension directly executes the shell scripts
• Scriptless phase 2 (production default): aks-node-controller provision → BuildCSECmd() → generates and executes the same shell scripts

Since emitKubeletActiveFlagsEvent() lives inside ensureKubelet() in cse_config.sh, it runs in both modes. No Go-side implementation is needed — the shell path covers all provisioning paths.

The event runs after kubelet is started (systemctl start --no-block kubelet), so journalctl -u kubelet contains the FLAG lines by the time the poll runs.

Why this runs in CSE, not during VHD bake

Kubelet's actual configuration (flags, config file) is node-specific — it depends on CustomData parameters (cluster DNS, max-pods, feature gates, etc.) and kubelet must be running before its active flags can be read from journalctl.

Implementation

1. getKubeletActiveFlagLines() — Polls journalctl -u kubelet -b --no-pager -r (with retries, since kubelet is started --no-block) to extract the FLAG: --name="value" lines kubelet prints at startup.
2. getKubeletActiveFlagsJSON() — Assembles a compact JSON payload containing:
   • flags: A curated set of CLI flag values (KUBELET_TRACKED_FLAGS)
   • config_file: The full content of the KubeletConfiguration file pointed to by --config (this file is already a compact subset generated by AgentBaker containing only RP-specified overrides)
3. Size guard — GuestAgentGenericLogs.Context1 is hard-truncated at 3072 bytes (3 KiB), verified empirically. We cap at 3000 bytes to leave margin. If exceeded, config_file is dropped and replaced with "truncated:exceeded_size_cap", with a log warning.
4. emitKubeletActiveFlagsEvent() — Writes the Guest Agent event file directly (bypasses logs_to_events wrapper to avoid the "Completed: " prefix that would break parse_json()). Called as (emitKubeletActiveFlagsEvent || true) & — background, best-effort, never blocks provisioning.

Kusto query example

GuestAgentGenericLogs
| where TaskName == "AKS.CSE.ensureKubelet.kubeletActiveFlags"
| extend payload = parse_json(Context1)
| project PreciseTimeStamp, RoleInstanceName, Cluster,
    maxPods = payload.config_file.maxPods,
    clusterDNS = payload.config_file.clusterDNS,
    featureGates = payload.config_file.featureGates,
    cpuManagerPolicy = payload.config_file.cpuManagerPolicy,
    flags = payload.flags

Test plan

• ShellSpec tests pass (819 examples, 0 failures)
• make generate produces no diff
• E2E via [Unofficial][Orchestration] AgentBaker + Linux VHD Full E2E Tests pipeline (bake + apply with current branch)

[abigailliang-aks-sig-node]

Tracking work item: AB#38936519

[github-actions]

Windows Unit Test Results

  3 files   11 suites   50s ⏱️
381 tests 381 ✅ 0 💤 0 ❌
384 runs  384 ✅ 0 💤 0 ❌

Results for commit 6de1e0b.

♻️ This comment has been updated with latest results.

[aks-node-assistant]

AgentBaker Linux gate detective

Run: 173691175
Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E
First failing step: Test_AzureLinuxV3_MANA validating traffic through MANA VF via debug pod exec

Detective summary: VMSS/node provisioning, SSH, validation pod, MANA PCI, driver, and VF bonding all passed; the failure occurred when Kubernetes exec from the debug pod returned apiserver-to-kubelet proxy 502 Bad Gateway before the curl traffic check could prove a datapath issue.

Likely cause / signature: Transient kubelet streaming/exec transport failure; reusing existing wiki signature wireserver-exec-context-deadline-kubelet-streaming.
Wiki signature: wireserver-exec-context-deadline-kubelet-streaming

Confidence: High

Recommended owner/action: E2E/test-infra owner should track under the kubelet-streaming flake and consider retrying exec on 502/500/context-deadline validation failures; no PR-author action recommended from this signal alone.

Strongest alternative: Real MANA datapath regression, but less likely because MANA setup validations passed and the observed error is the Kubernetes exec proxy transport failing before curl results were available.

Evidence: timeline failed task exit 1; failed test run 545377067; task log 562 around line 799; build metadata and PR #9034 metadata.

[aks-node-assistant]

AgentBaker Linux gate detective

Run: 173784179
Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E
First failing step: Test_Ubuntu2204_ArtifactStreaming_ARM64 kernel-log validation

Detective summary: Ubuntu 22.04 ArtifactStreaming ARM64 provisioned successfully, reached VM validation, then kernel-log validation flagged an early boot Call trace/PANIC_CRASH after node Ready, SSH, validation pod, KSCR/bootstrap, and NIC/rx checks passed.

Likely cause / signature: Likely transient/platform kernel call-trace validation noise; the kubelet-active-flags JSON PR is unlikely causal because the failure is in kernel log scan after unrelated provisioning/validation succeeded.
Wiki signature: ubuntu2204-artifactstreaming-arm64-kernel-calltrace-validation

Confidence: Medium

Recommended owner/action: Node Lifecycle/E2E owners should track as a new Ubuntu2204 ArtifactStreaming ARM64 kernel-calltrace signature and inspect scenario kernel logs if it recurs.

Strongest alternative: A PR-induced kubelet/config output instability, but less likely because the PR surface is telemetry/config formatting and the node was otherwise healthy through validation.

Evidence: timeline failed task exit 1; failed test run 546207590; task log 562 around line 772; build/PR metadata.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

aks-node-controller/app.go:192

• PR description says the scriptless implementation runs as a goroutine (go a.eventLogger.EmitKubeletActiveFlagsEvent()), but this code calls it synchronously. Since it can sleep/retry while waiting for kubelet FLAG lines, it can delay the provision command completion path unexpectedly. Either update the description, or make the call non-blocking with an explicit tradeoff (possible loss of the event if the process exits before the goroutine finishes).

		// Emit kubelet active flags as a structured event for Kusto querying.
		// Best-effort: telemetry failures do not fail provisioning.
		a.eventLogger.EmitKubeletActiveFlagsEvent()

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

aks-node-controller/app.go:192

• This call runs EmitKubeletActiveFlagsEvent() synchronously on the provisioning success path, which can block for up to ~20s due to the journalctl polling loop. The PR description says the Go implementation runs as a goroutine to avoid blocking; consider making this a goroutine here as well so provisioning completion timing is unaffected.

		// Emit kubelet active flags as a structured event for Kusto querying.
		// Best-effort: telemetry failures do not fail provisioning.
		a.eventLogger.EmitKubeletActiveFlagsEvent()

aks-node-controller/helpers/kubelet_flags.go:182

• pollKubeletFlags() sleeps between attempts even when journalctl fails (readKubeletFlagsFromJournal returns nil) and also sleeps after the final attempt. That makes EmitKubeletActiveFlagsEvent() take ~20s in environments without systemd/journal (and can slow unit tests unnecessarily). Consider breaking immediately on journalctl failure and avoiding unconditional sleep on the last iteration (ideally making the sleep context-aware).

// journalctl itself is unavailable (not installed, permission denied, etc.) — retrying won't
// help, so we bail immediately.
func pollKubeletFlags() map[string]string {
	ctx, cancel := context.WithTimeout(context.Background(), time.Duration(kubeletFlagsPollMaxAttempts)*kubeletFlagsPollInterval)
	defer cancel()

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

aks-node-controller/helpers/kubelet_flags.go:18

• The TaskName used for the scriptless kubelet-active-flags event ("AKS.AKSNodeController.ensureKubelet.kubeletActiveFlags") does not match the TaskName referenced in the PR description/query example ("ensureKubelet.kubeletActiveFlags"), so the documented Kusto query will miss events emitted by aks-node-controller. Either update the query/docs everywhere this is consumed, or align the TaskName here.

	// kubeletActiveFlagsTaskName is the event TaskName for the kubelet active flags telemetry.
	// Scriptless (aks-node-controller) and shell CSE use different TaskName values; keep this in sync
	// with the query patterns documented in the PR description.
	kubeletActiveFlagsTaskName = "AKS.AKSNodeController.ensureKubelet.kubeletActiveFlags"
	// kubeletFlagsPollMaxAttempts is how many times to poll journalctl for FLAG lines.

aks-node-controller/app.go:193

• EmitKubeletActiveFlagsEvent() can block for up to ~20s (10 attempts × 2s interval) while polling journalctl, but it is called synchronously on the success path. This contradicts the PR description’s claim that the scriptless implementation runs as a goroutine and can also delay the provision command/service completion unnecessarily. Consider moving this to an asynchronous execution path (or otherwise shortening/bounding the poll) while ensuring the process lifetime still allows the event to be written.

		a.eventLogger.LogEvent("Provision", "Completed", helpers.EventLevelInformational, startTime, endTime)
		slog.Info("aks-node-controller finished successfully.")
		// Emit kubelet active flags as a structured event for Kusto querying.
		// Best-effort: telemetry failures do not fail provisioning.
		a.eventLogger.EmitKubeletActiveFlagsEvent()
	}

[djsly]

lets ask the team if there are other flags they think we might need to log. Also, it might be important to have all the flag that RP can changes logged

[djsly]

this means we will see a single flag, is there a chance that we will have a race conditiion where not all flags are printed ?, should we wait for a minimum of flags to show ? I also recall that kubelet will print the list of flags a few time, form arg, overwrite from kubeconfig., etc

[djsly]

we don't have a helper for journalctl with proper retry backoff ?

[awesomenix]

This will delay our CSE completion, are there any other alternative ways?

[abigailliang-aks-sig-node]

i dont think so, unless I'm missing something

In cse_config.sh, we run the emission in the background, so it does not block CSE:

AgentBaker/parts/linux/cloud-init/artifacts/cse_config.sh

Lines 1016 to 1017 in f33d277

	(emitKubeletActiveFlagsEvent || true) &
	}

[awesomenix]

Recommend creating a async oneshot service which does this parsing and logging

[djsly]

should we use a different string ? normally AKS.CSE is used for timing ? I might be wrong and we might not have an official nomenclature for the log event naming.

AKS.Runtime ?

Actually, since we are waiting for all active flags to show up, we might be best to track the timing.

It might be best to not block the next steps on waiting for kubelet to log the FLAGS, it should run in the background, to not affect the CSE provsionning time. so AKS.Runtime might make more sense, (similar to how we log STLS data

curious what others thinks

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

parts/linux/cloud-init/artifacts/cse_config.sh:1016

• The PR description states the Go/aks-node-controller path is the “primary” implementation (with TaskName AKS.AKSNodeController.ensureKubelet.kubeletActiveFlags and helper code under aks-node-controller/helpers/kubelet_flags.go), but the current changeset only adds the shell fallback (AKS.CSE.ensureKubelet.kubeletActiveFlags). This mismatch makes it unclear whether the scriptless/phase-2 path is actually covered by this PR or whether the description needs updating.

    # Emit kubelet's effective startup configuration as a structured JSON event so it is queryable
    # per node from Kusto (rollout verification for the flags-to-config-file migration, and config/
    # version tracking dashboards). The JSON lands verbatim in GuestAgentGenericLogs.Context1 and is
    # consumable with parse_json(). Best-effort: never fail node provisioning on this.
(emitKubeletActiveFlagsEvent || true) &