Skip to content

v1.8.1 — resume.sh bounces dnsmasq + CoreDNS + stuck pods

Latest

Choose a tag to compare

@bogdandragosvasile bogdandragosvasile released this 21 Apr 08:17
· 2 commits to main since this release

Fix

After v1.8.0, ./scripts/resume.sh could leave the cluster with broken cluster DNS on some resumes — most visibly argocd-repo-server stuck in CrashLoopBackOff because it couldn't helm pull from argoproj.github.io (lookup failed via CoreDNS → dnsmasq → upstream).

Three issues after a Colima daemon reload:

  1. dnsmasq comes back listening on :53 but unresponsive. CoreDNS forwards to it for *.local and the forward chain stalls, cascading into every pod that needs external DNS (repo-server helm charts, cert-manager webhooks, etc.).
  2. CoreDNS holds a stale upstream cache even after dnsmasq recovers.
  3. Ghost pods from the pre-stop state land stuck Terminating or CrashLoopBackOff on the broken DNS.

resume.sh now:

  • docker restart dnsmasq
  • kubectl -n kube-system rollout restart deploy coredns + wait
  • Force-remove Terminating ghosts (--force --grace-period=0)
  • Delete CrashLoopBackOff pods so they reschedule against the now-healthy DNS

Observed end-to-end today: these three fixes take argocd-repo-server from 5 restarts + CrashLoopBackOff to 1/1 Running in ~40 seconds.

Full changelog

See CHANGELOG.md.