Mark nodes as "NotReady" until critical pods from daemonsets are Ready

[lbernail]

What would you like to be added:
An extension to node conditions that would mark them NotReady until critical pods from daemonsets are running (maybe a CriticalPodsReady or DaemonsetsReady similar to the NetworkReady condition).

Why is this needed:
When using cluster-autoscaler, application pods are immediately scheduled when a node starts and sometimes they start before local pods from daemonsets are Ready, which can be an issue. A few examples:

• kube-proxy, which prevents application from connecting to their dependencies (DNS is the most common problem)
• local dns pod: we deploy a local dns cache as a daemonset and configure pods to use it as a resolver, so resolution fail until this pod is up
• monitoring agent: some applications send metrics very early when they start and require the monitoring agent to be ready
• kube2iam: required by some applications to get aws credentials

/sig scheduling
@kubernetes/sig-scheduling-feature-requests

[Joseph-Irving]

We very recently made a controller to deal with this exact issue. https://github.com/uswitch/nidhogg tainting the node if certain daemonset pods haven't been scheduled and are ready yet.

[lbernail]

We very recently made a controller to deal with this exact issue. https://github.com/uswitch/nidhogg tainting the node if certain daemonset pods haven't been scheduled and are ready yet.

@Joseph-Irving Very nice. I'll definitely have a look. It would be great to have it natively in kubernetes but this would work in the meantime (and if the feature is not accepted of course)

[yastij]

cc @kubernetes/sig-scheduling-feature-requests @kubernetes/sig-node-feature-requests

I think this is something worth exploring. This can be tied to priorityClass: mark nodes with a specific condition until pods of a specific priorityClass are running on the node.

[Huang-Wei]

I like the idea. We have pod ready++, why not node ready++?

[yastij]

/assign

I’ll brainstorm with @lbernail and see if can come up with a KEP to kickoff discussions.

[cizixs]

We have the same feature requirement, and have come with up a design and implementation running in our kubernetes cluster.

And we would love to share and contribute to this.

[yastij]

@cizixs - can you share some thoughts and use cases here ?

[kevin-wangzefeng]

We very recently made a controller to deal with this exact issue. https://github.com/uswitch/nidhogg tainting the node if certain daemonset pods haven't been scheduled and are ready yet.

Having a controller to add taints to block node readiness during initialization may not work well under race condition. To complete this solution, you can make nodes born with that taint.

[cizixs]

@cizixs - can you share some thoughts and use cases here ?

User Case

• CNI service: we implemented CNI with a running service, pods can only be scheduled to the node when CNI service is up and running
• monitor agent: monitoring data is very important for us, running application with no metrics is considered serious problem
• ...

Design

Add a APIServer admission, add a taint to newly added node, so nodes are not schedule-able even if kubelet is running.

Define a CRD, and each node has a corresponding CR(with readinessGates field) indicates if the node is ready++. We use a CRD instead of adding a new node field, so node spec is keep unchanged, and it's easy to keep update with community.

Daemonset pods on the node is responsible for updating node CR readiness status that should concern them.

Add a new controller that creates node CR when a node is created(readinessGate list is read from a configmap), and also listens to node readiness CR to add or delete node taint based on if all readinessGate is ready.