Node taints should go back to baseline on machine restart

[philpearl]

We apply a taint to nodes, and remove the taint once our daemon is up and running on the node. This then allows other pods to run on the node. We don't want other pods to run on the node until the daemon is up & running.

This works well until the underlying machine is restarted. If the kubernetes node is destroyed and we get a fresh new node then everything is OK. If not then our taint is not reapplied, and pods are started on the node even though the daemon is not fully active.

I believe the changes made under #322 (Remove taints re-application from node annotator) have caused or contributed to this issue.

• what would be the view on reverting 322 or making the behaviour configurable in some way? Perhaps having a set of taints that are always re-applied when the controller starts?
• is it possible to ensure the node is destroyed and recreated at the kubernetes level if the underlying VM is replaced or restarted? We're using auto-scaling node pools.

[philpearl]

@achandrasekar @mikedanese I wonder if you considered this regression when you were looking at #322?

[aojea]

We apply a taint to nodes, and remove the taint once our daemon is up and running on the node. This then allows other pods to run on the node. We don't want other pods to run on the node until the daemon is up & running.

can you expand on this?

do you have a controller that watches nodes and taint based on the events?

[philpearl]

Yes, there's a controller that watches for the daemon and it removes the taint when it sees the daemon has started.

[aojea]

Yes, there's a controller that watches for the daemon and it removes the taint when it sees the daemon has started.

shouldn't you be watching the Node and acting on the Node object?

you can detect a node has changed based on the UUID

[philpearl]

We tried that too. But there's nothing that means we get to make our change to the node before the scheduler starts putting pods on it. That's what the taint would guarantee, but it's not added back even though the rest of the node state has returned to baseline

[achandrasekar]

@philpearl #322 was not a regression. In fact it was a fix to address the regression introduced with #285. We introduced the behavior to reapply taints. But there are a lot of people relying on the older behavior of not reapplying taints. They use these taints similar to the way you mentioned. Except for them it is only needed on new node startup and not on node restarts. Since this was a regression, we had to revert the change.

But ideally, like you mentioned, we need a way to configure if the taints should be reapplied and when.

is it possible to ensure the node is destroyed and recreated at the kubernetes level if the underlying VM is replaced or restarted? We're using auto-scaling node pools.
I believe on VM replacement like during preemption, this should already happen. Let me know if this is not the case. Restart behavior should be addressed separately.

[philpearl]

@achandrasekar thanks for the update - it's good to understand the full history! Is finding a way to configure if the taints should be reapplied and when something that you're working on / might happen soon? If not would you be open to a PR (although it looks like I'd have a huge amount of ramp up to do to work out how to build / test / deploy!) ??

[achandrasekar]

@philpearl There is work planned to address this. But we don't have an ETA yet. I can provide an update once we do. The tainting behavior has a lot of implications to Cluster Autoscaler's scale up/down behavior. So, accepting a PR on this is difficult.

[achandrasekar]

Are you facing the issue only with preemptible nodes like the one mentioned here - cilium/cilium#21594? There is ongoing work to address this specific case.

[thejosephstevens]

Hey all, I think I might be running into the same issue here (also running Cilium with NOEXECUTE taints). The specific symptoms I'm seeing are that there's a node reboot event (like Node nodename-c9lq has been rebooted, boot id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx), but then I see some pods that actually continue running through the reboot (they have an age that dates back to before the reboot).

The effective outcome is the same as what phil is dealing with. I was expecting the NOEXECUTE taint to kick in and blow those pods away and keep them from coming back until the daemon starts, but instead they persisted, which resulted in application failures.

Let me know if there's anything I can provide that would be helpful to debugging this, this is impacting our customers with some regularity.

[owais]

We are facing the same issue with underlying VM restarts in GKE. It is pretty easy to replicate by restarting the underlying VM instance. In some cases, some pods (both deployments and daemonsets) run before cilium and enter a bad state requiring restarts.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.