“Safely Drain A Node” does not explain how to handle DaemonSet pods

[liuwh08]

The draining guide of draining explicitly ignores Daemonsets pods by

kubectl drain --ignore-daemonsets <node name>

Wonder why Daemonsets are safer to skip draining comparing to other pods from Deployment or Statefulsets (or maybe regular pods). There should be no difference compatability-wise about those pods to survive a drop-in kubelet MINOR upgrade.

[liuwh08]

cc @liggitt

[utkarsh-singh1]

Related Doc - https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/#use-kubectl-drain-to-remove-a-node-from-service
/language en
/sig docs

[sftim]

If there are daemon set-managed pods, drain will not proceed without --ignore-daemonsets, and regardless it will not delete any daemon set-managed pods, because those pods would be immediately replaced by the daemon set controller, which ignores unschedulable markings.

from linked page https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#drain

Does that explain it?

[liggitt]

DaemonSet pods are not safer to skip draining, they are just more difficult to drain effectively.

In the skew/upgrade doc, it indicates pods must be removed from a node before upgrading, because kubelet does not test or guarantee compatibility of on-disk structures between minor versions. That guidance applies to all types of pods, including daemonset pods. From an upgrade perspective, it doesn't especially care whether you use the kubectl drain command or not, just that the end result is there are no pods assigned to the node prior to upgrade.

In the page that describes running the drain command, it walks through steps to remove pods, but apparently does not provide instructions for effectively removing daemonset pods. That's the issue to fix (either by documenting effective steps or by improving the drain command or daemonset controller or both)

[sftim]

I think the kubectl drain docs imply that the subcommand doesn't drain daemonsets. Part of the fix could include an improvement to how we explain the subcommand.

The key question that Safely Drain A Node must answer is: How do we stop DaemonSets from replacing Pods after we remove them, and before we have finished maintenance?

We can document this, but it might be a bit of a pain: I think we'd need to look at labelling nodes and changing the scheduling rules for each DaemonSet.

[sftim]

/triage accepted
/sig cluster-lifecycle

[sftim]

/priority backlog

IMO: not quite important-longterm, but close

/lifecycle frozen

/retitle “Safely Drain A Node” does not explain how to handle DaemonSet pods

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted