fix: restore DNS after unexpected LocalDNS exit - #9360
fix: restore DNS after unexpected LocalDNS exit#9360Saewon Kwak (saewoni) wants to merge 3 commits into
Conversation
Windows Unit Test Results 3 files 13 suites 40s ⏱️ Results for commit 94d56c0. ♻️ This comment has been updated with latest results. |
There was a problem hiding this comment.
Pull request overview
Removes unnecessary LocalDNS cgroup delegation to prevent teardown failures after unclean supervisor termination.
Changes:
- Sets
Delegate=noforlocaldns.service. - Adds VHD-content validation for the setting.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
parts/linux/cloud-init/artifacts/localdns-delegate.conf |
Disables cgroup delegation. |
vhdbuilder/packer/test/linux-vhd-content-test.sh |
Verifies delegation remains disabled. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
🟡 Changes recommended
The new E2E test does not compile and invokes an unsupported systemctl verb.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
e2e/scenario_localdns_lifecycle_test.go:43
- 🔴 High Risk — 🔧 Script Logic:
Config.Validatoris declared asfunc(context.Context, *Scenario) error(e2e/types.go:233-234), so this no-result function cannot be assigned at line 35 and the E2E package will not compile. Return the error fromexecScriptOnVMForScenarioValidateExitCode(discarding only its result value).
func validateLocalDNSLifecycle(ctx context.Context, s *Scenario) {
execScriptOnVMForScenarioValidateExitCode(ctx, s, `
- Files reviewed: 6/6 changed files
- Comments generated: 1
- Review effort level: Balanced
|
|
||
| # Verify the normal stop path completes before the service is started again. | ||
| systemctl stop localdns.service | ||
| systemctl is-inactive --quiet localdns.service |
There was a problem hiding this comment.
Addressed in commit c3778a344. The E2E now uses sudo for mutating systemctl and kill operations, and polls boundedly for active/running instead of relying on a fixed two-second delay.
| Delegate=cpu No newline at end of file | ||
| # LocalDNS does not create or manage child cgroups. Keeping the service | ||
| # non-delegated lets systemd reliably tear down all CoreDNS descendants. | ||
| Delegate=no |
There was a problem hiding this comment.
I think this is the correct fix. But let's verify the metrics exporter is still working after the change.
There was a problem hiding this comment.
This comment was on the earlier Delegate=no change, which has been removed from PR #9360. The current PR does not change cgroup layout, LOCALDNS_CGROUP_DIR, cpu.stat, memory.current, resources.prom, or the exporter. The existing LocalDNS exporter E2E validation remains applicable and checks non-zero CPU and memory metrics. No exporter-specific regression is introduced by the current Fix 1 changes.
Ye Wang [msft] (yewmsft)
left a comment
Ye Wang [msft] (yewmsft)
left a comment
There was a problem hiding this comment.
Review — LocalDNS delegated cgroup teardown
good root cause work and the live repro is genuinely useful. but I think the framing is off, and there are two blockers.
this is not a cgroup bug, it is a missing teardown guarantee. nothing reverts the node DNS config when the supervisor dies without cooperating. SIGKILL is not trappable, so no trap and no ExecStop= can ever cover the kill -9 path you reproduced. ExecStopPost= is the hook that runs on every exit path — details inline on localdns.service.
blockers:
ExecStop=does not run when the main process dies unexpectedly, so it does nothing for the bug in the description — and it regresses ordinary stop intofailed. the copilot comment on that line is correct, verified:ERR_LOCALDNS_FAIL=216atlocaldns.sh:10, trap at:1056, noSuccessExitStatus=in the unit.e2e/scenario_localdns_lifecycle_test.go:43drops both return values, so the new test cannot fail.
also: ${DEFAULT_ROUTE_INTERFACE} is unbound under set -u on the cleanup path (localdns.sh:719-722).
please split this. Delegate=cpu → no and the networkctl reconfigure addition are two unrelated changes riding along. both stand on their own merits, and the delegate one can merge immediately.
one question before any of this: on the AKS#5930 nodes, was 70-localdns.conf still on disk while the service was dead? if yes that confirms missing-teardown over the cgroup story, and reshapes the fix.
| KillMode=control-group | ||
| # Ask localdns.sh to run its graceful cleanup first. The stop helper waits for | ||
| # the parent to exit before systemd applies KillMode=control-group to leftovers. | ||
| ExecStop=/bin/sh -c 'kill -SIGINT "$MAINPID" 2>/dev/null || true; while kill -0 "$MAINPID" 2>/dev/null; do sleep 0.1; done' |
There was a problem hiding this comment.
ExecStop only runs for an explicit stop job. when the main process dies unexpectedly — your kill -9 $MAINPID repro — systemd skips ExecStop and goes straight to ExecStopPost. man systemd.service under ExecStopPost=: "this includes cases where the commands configured in ExecStop= were used, or where the service exited unexpectedly."
so this line does nothing for the bug in the description. it only changes systemctl stop/restart.
and the copilot comment above is right. SIGINT lands on the error trap at localdns.sh:1056 which exits $ERR_LOCALDNS_FAIL (216, localdns.sh:10), and there is no SuccessExitStatus=. so every ordinary stop now leaves the unit in failed. today SIGTERM is untrapped, bash dies from the signal systemd itself sent, and stop is clean — so this is a regression, not a fix.
drop ExecStop.
There was a problem hiding this comment.
Addressed in commit c3778a344. Removed ExecStop; added cleanup-only ExecStopPost=/opt/azure/containers/localdns/localdns.sh cleanup, which runs after unexpected exits and always returns 0. Added a dedicated SIGTERM trap that runs graceful cleanup and exits 0. The existing 216 failure status remains for genuine error paths.
| KillMode=mixed | ||
| # CoreDNS is launched as a child of localdns.sh. Kill the complete service | ||
| # cgroup together so no descendant survives an unclean supervisor exit. | ||
| KillMode=control-group |
There was a problem hiding this comment.
this is the wrong layer. the real problem is that nothing reverts the node DNS config when the supervisor dies without cooperating — SIGKILL is not trappable, so no trap and no ExecStop can ever cover it.
use ExecStopPost, which runs on every exit path including kill -9:
ExecStopPost=/opt/azure/containers/localdns/localdns.sh cleanuplocaldns.sh already defines all its functions above the main flow, and cleanup_iptables_and_dns self-initializes its network vars (:665), so this is small:
# after function definitions, before the main flow
if [ "${1:-}" = "cleanup" ]; then
cleanup_localdns_configs || echo "Best-effort cleanup reported errors."
exit 0
fiit must exit 0 unconditionally — a failing ExecStopPost marks the unit failed, on exactly the teardown path we are trying to keep unwedged.
now walk your repro with that in place: kill -9 → ExecStopPost removes the drop-in → restart → ExecStart re-adds it → ... → StartLimit trips, unit stays dead, but the last ExecStopPost already ran, so resolv.conf is back on upstream DNS. no blackhole. "unit is dead" becomes degraded-but-working instead of an outage.
and add a TERM trap in the script so ordinary stop is graceful and exits 0:
trap 'echo "Received SIGTERM, shutting down."; cleanup_localdns_configs; exit 0' TERMkeep 216 on ABRT ERR INT PIPE for genuine failures. note cleanup can then run more than once (TERM trap + EXIT trap + ExecStopPost), so it has to be idempotent.
KillMode=control-group is fine to keep, but it is not the fix — mixed already SIGKILLs the whole cgroup at TimeoutStopSec.
There was a problem hiding this comment.
Addressed in commit c3778a344. ExecStopPost now invokes the script in cleanup mode, so it handles both explicit stops and unexpected exits. Cleanup-only mode performs DNS/network cleanup and exits 0; systemd remains responsible for process/cgroup cleanup. The normal SIGTERM trap also exits 0 after graceful cleanup.
There was a problem hiding this comment.
Applied Fix 1 in commit c9862f958. The PR is now focused on unexpected-exit DNS teardown: ExecStopPost invokes cleanup mode after any service exit, normal SIGTERM cleanup exits 0, and process cleanup remains systemd-owned. The unrelated delegation and network reconfigure changes were removed from the branch.
| Delegate=cpu No newline at end of file | ||
| # LocalDNS does not create or manage child cgroups. Keeping the service | ||
| # non-delegated lets systemd reliably tear down all CoreDNS descendants. | ||
| Delegate=no |
There was a problem hiding this comment.
this one is right and stands on its own. localdns.sh forks coredns as a plain child and never writes cgroup.procs or cgroup.subtree_control — it only reads cpu.stat and memory.current at :816-817. delegation for a payload that never builds a subtree is dead config. and if the original intent was cpu accounting, the directive is CPUAccounting=yes, not Delegate=.
nit: Delegate=no is the systemd default, so this file is now a no-op. keeping it for explicitness is fine — say that in the comment.
please split this into its own PR. it is unrelated cleanup and can merge immediately, while the teardown fix needs an isolated repro.
one check before it goes: the exporter reads $LOCALDNS_CGROUP_DIR/cpu.stat and memory.current with || echo "0", so it would break silently. ValidateLocalDNSExporterMetrics already asserts non-zero and runs for every localdns scenario (validation.go:111) — but e2e runs main's VHD, so it will not exercise this change. run it against a VHD built from the branch, or just eyeball cat /sys/fs/cgroup/localdns.slice/cgroup.subtree_control on the repro node (expect cpu memory).
There was a problem hiding this comment.
The delegation change is no longer part of this PR. localdns-delegate.conf remains unchanged at Delegate=cpu; the PR now focuses on unexpected-exit DNS teardown and the service lifecycle. The exporter remains covered by the existing LocalDNS exporter validation; this change does not alter the cgroup path or resource accounting.
There was a problem hiding this comment.
I traced the history and current code for Delegate=cpu.
It was introduced in the original LocalDNS VHD packaging change, 9b65e12735, Add localdns related files to VHDs (#6110), on April 7, 2025. The commit introduced localdns.service, localdns.sh, and localdns-delegate.conf, but does not document why CPU delegation was added.
The current code shows that LocalDNS:
- Launches CoreDNS as a normal child of
localdns.sh. - Reads
cpu.statandmemory.currentfor metrics. - Does not write
cgroup.procsorcgroup.subtree_control. - Does not create or manage child cgroups.
The resource limits are configured separately on localdns.slice with CPUQuota and MemoryMax. Reading cgroup metric files does not require Delegate=cpu; if the original intent was CPU accounting, the relevant systemd setting would be CPUAccounting=yes, not Delegate=.
So Delegate=cpu appears to be legacy or unnecessary configuration, but the history does not prove the original intent. I agree it should be treated as a separate cleanup PR rather than part of the unexpected-exit DNS teardown fix. Any follow-up should validate the exporter and cgroup metrics across Ubuntu, Azure Linux, and Mariner using VHDs built with and without delegation.
| echo "Reloading network configuration succeeded." | ||
|
|
||
| echo "Reconfiguring network interface ${DEFAULT_ROUTE_INTERFACE}." | ||
| "${NETWORKCTL_RECONFIGURE_CMD[@]}" "${DEFAULT_ROUTE_INTERFACE}" |
There was a problem hiding this comment.
three things here:
-
${DEFAULT_ROUTE_INTERFACE}with no:-.set -euo pipefailat :2, and every other reference in this file uses${DEFAULT_ROUTE_INTERFACE:-}. the guard at :666 only initializes when NETWORK_DROPIN_FILE/DIR is empty, so this can be unbound here → bash aborts mid-cleanup with iptables and resolv.conf half reverted.set +ein cleanup_localdns_configs does not coverset -u. -
if [ "$?" -ne 0 ]→if ! "${NETWORKCTL_RECONFIGURE_CMD[@]}" "${DEFAULT_ROUTE_INTERFACE:-}"; then -
should a reconfigure failure really fail cleanup? you are adding a second hard-fail to the teardown path in a PR whose whole point is that teardown must not get stuck. log and continue.
also :712 right above is still eval "$NETWORKCTL_RELOAD_CMD" — same injection shape, same function. you converted the new one to an array, convert that one too.
| # Reapply the loaded network configuration to the default-route link after removing | ||
| # the LocalDNS drop-in. Reload only refreshes networkd's configuration; reconfigure | ||
| # causes the live link state, including DNS settings, to be reapplied. | ||
| NETWORKCTL_RECONFIGURE_CMD=(networkctl reconfigure) |
There was a problem hiding this comment.
this whole reconfigure change is a separate fix — different root cause (stale resolver entry), different failure mode — appended to a cgroup-teardown PR. please split it out. it makes the teardown fix reviewable and shippable on its own.
There was a problem hiding this comment.
Addressed in commit c3778a344. Removed the unrelated networkctl reconfigure addition and its tests from this PR. The existing network reload path is unchanged.
| systemctl start localdns.service | ||
| systemctl is-active --quiet localdns.service | ||
|
|
||
| for i in 1 2 3 4 5 6 7 8 9 10; do |
There was a problem hiding this comment.
StartLimitBurst=5 in 10s, and you fire 10 kills over 2.5s. whether you trip the limit depends on how fast the unit comes back up between kills, so this is timing dependent — it will flake.
There was a problem hiding this comment.
Addressed in commit c3778a344. The E2E now waits for each restart to reach active/running with a bounded polling loop rather than depending on a fixed delay.
| return 1 | ||
| fi | ||
|
|
||
| if ! grep -Fq 'ExecStop=/bin/sh -c' /etc/systemd/system/localdns.service || \ |
There was a problem hiding this comment.
this asserts on the exact shell text of ExecStop. reformat the line and the test breaks with zero behavior change.
assert the merged unit instead:
systemctl show localdns.service -p KillMode -p Delegate --valuethat also proves the drop-in actually merged, which grepping two separate files does not.
There was a problem hiding this comment.
Addressed in commit c3778a344. The VHD content test now checks for the ExecStopPost cleanup hook rather than asserting the exact old ExecStop shell text. Runtime merged-unit behavior is covered by the lifecycle E2E.
There was a problem hiding this comment.
🟡 Changes recommended
The lifecycle test rejects the correct resolver state and does not reproduce the reported rapid-restart race.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
e2e/scenario_localdns_lifecycle_test.go:85
- 🟡 Medium Risk — 🔧 Script Logic: This assertion is inverted for the final state: the service has been started and confirmed active, and
disable_dhcp_use_clusterlistenerconfigures the live node resolver to use169.254.10.10(localdns.sh:635-648). The existingValidateLocalDNSResolutionlikewise requires defaultdigoutput to report that server (e2e/validators.go:2108-2120). As written, a correctly configured LocalDNS node fails this new lifecycle test; poll until the listener appears instead of asserting that it is absent.
- Files reviewed: 4/5 changed files
- Comments generated: 1
- Review effort level: Balanced
| for i in 1 2 3; do | ||
| main=$(sudo systemctl show -p MainPID --value localdns.service) | ||
| sudo kill -9 "$main" | ||
| recovered=false | ||
| for attempt in 1 2 3 4 5 6 7 8 9 10 11 12; do | ||
| state=$(sudo systemctl show localdns.service -p ActiveState -p SubState --value) | ||
| if [ "$state" = $'active\nrunning' ]; then | ||
| recovered=true | ||
| break | ||
| fi | ||
| sleep 1 | ||
| done | ||
| test "$recovered" = true | ||
| done |
There was a problem hiding this comment.
🟡 Changes recommended
Three critical issues leave the intended cgroup and DNS recovery fixes incomplete.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 4/5 changed files
- Comments generated: 3
- Review effort level: Balanced
| printf '%s\n' "$state" | grep -q '^ActiveState=active$' | ||
| printf '%s\n' "$state" | grep -q '^SubState=running$' | ||
| printf '%s\n' "$state" | grep -q '^Result=success$' | ||
| ! sudo awk '/^nameserver/ {print $2}' /run/systemd/resolve/resolv.conf | grep -qx '169.254.10.10' |
| @@ -1,2 +1,2 @@ | |||
| [Service] | |||
| Delegate=cpu No newline at end of file | |||
| Delegate=cpu | |||
| echo "Attempt to reload network configuration." | ||
| eval "$NETWORKCTL_RELOAD_CMD" | ||
| if [ "$?" -ne 0 ]; then | ||
| if ! "${NETWORKCTL_RELOAD_CMD}" reload; then | ||
| echo "Failed to reload network after removing the DNS configuration." | ||
| return 1 | ||
| fi |
c3778a3 to
c9862f9
Compare
Saewon Kwak (saewoni)
changed the title
There was a problem hiding this comment.
🟡 Changes recommended
DNS restoration can be skipped after iptables failures, and the automated tests have correctness and coverage gaps.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 5
- Review effort level: Balanced
| # Only restore node DNS configuration here; systemd owns process cleanup. | ||
| # Always exit successfully so a cleanup error cannot wedge systemd recovery. | ||
| if [ "${1:-}" = "cleanup" ]; then | ||
| cleanup_iptables_and_dns || echo "Best-effort LocalDNS DNS cleanup reported errors." |
| AKSNodeConfigMutator: func(_ *Cluster, cfg *aksnodeconfigv1.Configuration) { | ||
| cfg.LocalDnsProfile.EnableLocalDns = true | ||
| }, | ||
| Validator: validateLocalDNSLifecycle, |
| # Repeatedly kill the supervisor and wait for Restart=on-failure recovery. | ||
| for i in 1 2 3; do | ||
| main=$(sudo systemctl show -p MainPID --value localdns.service) | ||
| test "$main" -gt 0 | ||
| sudo kill -9 "$main" |
| recovered=false | ||
| for attempt in 1 2 3 4 5 6 7 8 9 10 11 12; do | ||
| state=$(sudo systemctl show localdns.service -p ActiveState -p SubState --value) | ||
| if [ "$state" = $'active\nrunning' ]; then | ||
| recovered=true | ||
| break | ||
| fi | ||
| sleep 1 | ||
| done |
| if [ "${1:-}" = "cleanup" ]; then | ||
| cleanup_iptables_and_dns || echo "Best-effort LocalDNS DNS cleanup reported errors." | ||
| exit 0 |
Saewon Kwak (saewoni)
changed the title
| AKSNodeConfigMutator: func(_ *Cluster, cfg *aksnodeconfigv1.Configuration) { | ||
| cfg.LocalDnsProfile.EnableLocalDns = true | ||
| }, | ||
| Validator: validateLocalDNSLifecycle, |
There was a problem hiding this comment.
If possible, could you attach validateLocalDNSLifecycle as the Validator on the existing Test_LocalDNSHostsPlugin distros instead of 3 new scenarios, gated to the 3 target distros and running last, to reuse already-provisioned LocalDNS nodes. This can save some test resources and reduce test time.
There was a problem hiding this comment.
Addressed in commit c1b9b1209c. The standalone lifecycle scenarios were removed, and validateLocalDNSLifecycle is now attached to the existing Test_LocalDNSHostsPlugin scenarios for Ubuntu 22.04, Ubuntu 24.04, and Azure Linux V3. It runs after the existing validation and reuses the already-provisioned LocalDNS VM for each distro. Both BootstrapConfigMutator and AKSNodeConfigMutator remain enabled. The E2E module compiles successfully.
AgentBaker Linux gate detectiveRun: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=179413719 TL;DR: LocalDNS lifecycle validation failed after stop/restart/kill recovery because journalctl still contained Likely cause / signature: Strongest alternative: systemd/cgroup VM-local flake; less likely because the same validation failed across two OS scenarios in the same PR run and the changed files directly target LocalDNS lifecycle cleanup. Recommended owner/action: PR author / LocalDNS owner: inspect service cleanup behavior around the LocalDNS cgroup and ensure the new lifecycle test tolerates/handles systemd Evidence: build summary shows Wiki signature: e2e-localdns-lifecycle-controlgroup-not-cleaned |
There was a problem hiding this comment.
🟡 Changes recommended
Cleanup errors can leave DNS unrestored, and the E2E test has recovery-detection and terminal-state coverage gaps.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
parts/linux/cloud-init/artifacts/localdns.sh:990
- 🔴 High Risk — 🔧 Script Logic: This post-stop path does not reliably restore DNS on a cleanup error.
cleanup_iptables_and_dnsreturns immediately when any iptables deletion fails (lines 690-692), before it removes70-localdns.conf(lines 697-704); this branch then logs the error and exits 0, leaving the node pointed at the dead resolver—the outage this change is intended to prevent. Make DNS drop-in removal/reload unconditional and aggregate any iptables error afterward.
cleanup_iptables_and_dns || echo "Best-effort LocalDNS DNS cleanup reported errors."
- Files reviewed: 4/4 changed files
- Comments generated: 2
- Review effort level: Balanced
| # Repeatedly kill the supervisor and wait for Restart=on-failure recovery. | ||
| for i in 1 2 3; do | ||
| main=$(sudo systemctl show -p MainPID --value localdns.service) | ||
| test "$main" -gt 0 | ||
| sudo kill -9 "$main" |
| state=$(sudo systemctl show localdns.service -p ActiveState -p SubState --value) | ||
| if [ "$state" = $'active\nrunning' ]; then | ||
| recovered=true | ||
| break | ||
| fi |
|
LGTM in terms of the integration with overall provisioning process and AgentBaker's e2e infra. |
There was a problem hiding this comment.
🟡 Changes recommended
DNS restoration is not guaranteed, and the regression tests can false-pass or overlook expected failure signals.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (3)
e2e/scenario_localdns_hosts_test.go:93
- 🟡 Medium Risk — 🔧 Script Logic: This poll can accept the pre-kill service instance: immediately after
kill -9, systemd may still reportactive/runningbefore processing the supervisor's death. The loop then breaks without proving thatRestart=on-failurecreated a new supervisor, so the regression can false-pass. Require a positiveMainPIDdifferent from the killed PID before declaring recovery.
state=$(sudo systemctl show localdns.service -p ActiveState -p SubState --value)
if [ "$state" = $'active\nrunning' ]; then
recovered=true
break
fi
parts/linux/cloud-init/artifacts/localdns.sh:990
- 🔴 High Risk — 🔧 Script Logic:
cleanup_iptables_and_dnsreturns immediately when any iptables deletion fails (lines 690–692), before it removes the DNS drop-in and reloads the network (lines 697–711). This new post-exit path then masks that failure and exits successfully, so a failed service can still leave node DNS pointing at the dead LocalDNS listener. Make DNS restoration an unconditional/finally-style step, and report any iptables error only after the DNS step has run.
cleanup_iptables_and_dns || echo "Best-effort LocalDNS DNS cleanup reported errors."
parts/linux/cloud-init/artifacts/localdns.sh:991
- 🟡 Medium Risk — 🧪 Test Coverage: The repository requires ShellSpec coverage for shell-script changes, but the existing
localdns_spec.shsources this file and returns at line 984, so it never exercises this new executablecleanupdispatch or its always-success contract. Add ShellSpec cases that invoke cleanup mode with successful and failing cleanup implementations; the VHD-content assertion only verifies the unit text, not this behavior.
if [ "${1:-}" = "cleanup" ]; then
cleanup_iptables_and_dns || echo "Best-effort LocalDNS DNS cleanup reported errors."
exit 0
- Files reviewed: 4/4 changed files
- Comments generated: 1
- Review effort level: Balanced
| if sudo journalctl -u localdns.service --since "@$test_start" --no-pager | grep -q 'Failed to kill control group'; then | ||
| echo "WARNING: LocalDNS cgroup teardown warning observed" | ||
| fi |
Problem
This PR addresses the LocalDNS failure reported by Azure/AKS#5930.
localdns.servicesuperviseslocaldns.sh, which starts CoreDNS as a background process. During an unexpected supervisor exit, such asSIGKILL, the script cannot run its cleanup traps. The node can therefore retain the network drop-in that points DNS at LocalDNS while the resolver is no longer available. Restart attempts can also encounter leftover processes and fail, eventually leaving the service dead and causing a node-level DNS outage.Fix
ExecStopPost=/opt/azure/containers/localdns/localdns.sh cleanupso DNS cleanup runs after both normal and unexpected service exits.cleanupmode tolocaldns.shthat restores node DNS configuration and always exits successfully; systemd remains responsible for process cleanup.SIGTERMwith graceful cleanup and exit status 0; genuine error paths retain the existing failure status.The cgroup delegation configuration and unrelated network reconfiguration change are intentionally not part of this PR.
Live reproduction
The failure was reproduced twice on the disposable cluster
sakwa-localdns-repro-0710using the affected Ubuntu 24.04 imageAKSUbuntu-2404gen2containerd-202608.06.1. Reproduction targeted the single system-pool nodeaks-sysnp-14424852-vmss000000with LocalDNS enabled.Healthy baseline:
Fault injection:
Observed failure:
The first run reached
NRestarts=13; the second reachedNRestarts=6. Each run was recovered withsystemctl reset-failed localdns.servicefollowed bysystemctl restart localdns.service.Validation
The E2E scenario uses isolated, scenario-created single-VM VMSS instances. It validates normal stop/start, then repeats supervisor termination and polls for service recovery. It asserts the final service state and rejects the cgroup teardown and StartLimit errors.
The E2E package compiles locally:
The actual cloud E2E requires VHDs built from this branch because the LocalDNS unit files are baked into the VHD. The prior cloud E2E run reached the test pipeline but failed on test-harness issues; those issues were corrected in this update. A fresh cloud E2E result for the current commit is still required.