kube-dns should support custom tolerations

[JeanMertz]

/kind feature

(note: we're not using the latest Kubernetes version yet, but I couldn't find any changes related to this in the latest updates, so please close this if I simply updating will fix this for us)

Currently, when you edit a kube-dns deployment, some changes are persisted (f.e. adding a nodeAffinity config block), while others are reverted (f.e. adding a second toleration to the already existing CriticalAddonsOnly toleration).

The last example is problematic for us (running on GKE).

We have two node pools:

• a regular node pool with a taint applied to the nodes to only allow pods to run on them that we manually approve
• a preemptible node pool without any taints

We want kube-dns (and any other critical pods) to run on the regular nodes for better uptime (see also: #41125), so we update its deployment with the following blocks:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: type
          operator: In
          values:
          - regular

tolerations:
- key: CriticalAddonsOnly # this one is already present by default
  operator: Exists
- key: dedicated # we add this one to allow scheduling on the tainted regular nodes
  operator: Equal
  value: regular
  effect: NoSchedule

After saving this change, for a brief moment, the pods are scheduled as expected, but moments after, the change is reverted, and specifically the added toleration is removed, the nodeAffinity is still there however.

Can we make toleration changes persistent and not be reverted by the API itself?

Environment:

• Kubernetes version (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T21:08:42Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.3-gke.0", GitCommit:"86d3ac5eaf57223302c95e7d9fc1aeff55fb0c15", GitTreeState:"clean", BuildDate:"2017-11-08T21:42:58Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"}
  

• Cloud provider or hardware configuration: GKE

[JeanMertz]

/sig network

[thockin]

Unfortunately, there's not really a way to have a field that takes values from two different sources. In this case, the addon's base YAML (managed by GKE) sets one toleration and you are trying to set another. The patch logic will throw your changes out periodically, when the addon is re-asserted, as you are experiencing. Once we are using priority and preemption, we won't need this toleration. In fact, I can not convince myself it is being used at all.

@bsalamat or @vishh do you have any context?

[bsalamat]

I agree with @thockin that GKE and your deployment race against one another in setting tolerations for kube-dns. One solution that comes to my mind is to set specific labels to the nodes in each of the node pools and create all your pods with node affinity to one of the two labels depending on where you want them to run. For example, all the nodes in your first node pool have "pool:groupA" label and all the nodes in the second pool have "pool:groupB". Then all the pods created in your cluster will have node affinity to one of the two labels depending on where you want them to run. Your kube-dns will have node affinity to "pool:groupA". By doing this, you will get rid of the node taints and the pod tolerations.

As @thockin also mentioned, priority and preemption remove the need to have critical pods and CriticalAddonsOnly taint, but the feature is in alpha and is not enabled in GKE. Our plan is to move it to Beta in the next release.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[JeanMertz]

/lifecycle frozen
/remove-lifecycle stale

[fotinakis]

Any updates on this issue? We'd love to apply tolerations/affinity to kube-dns in a proper and maintainable way.

[Glennvd]

+1 for this, at the moment it's not possible to add pre-emptible pools to your GKE cluster if you want to run any other service reliably on it. Since you can give your essential pods a nodeAffinity but not the kube-system pods it relies on (like kube-dns).

[bgrant0607]

Ref #23233

[thockin]

@Glennvd You can create a node pool with a taint - any pod that doesn't explicitly tolerate that taint will avoid those nodes.

[matti]

@thockin but doing that causes "Does not have minimum availability" for kube-dns, because apparently once GKE sees more nodes it changes replicas: 2

[pajel]

Related issue #kubernetes/kops#6063

[bgrant0607]

cc @bowei @justinsb

[ahsan-storm]

Any update on adding tolerations on kube-dns via kops?

[cgsimmons]

Also would like an update for this issue. We currently experience dns latency because we have a large tainted node-pool (relative to untainted) that we cannot run dns pods on. Thus traffic is constantly routed for the dns service to nodes with no kube-dns pods.

[jagadishg]

Any update on this issue? Our node pool has all nodes tainted. But coz of this issue, it has become necessary for us to have node pools with untainted nodes. We can't have untainted node just to run this pod. Ideally, we would like to be able to add custom tolerations to kube-dns, kube-dns-autoscaler, heapster, so we can run them on tainted nodes

[lutierigb]

priority and preemption is GA since k8s 1.14 and in GKE(checked in 1.15) we even have two PriorityClass resources for critical workloads in the cluster and one of them already set in the kube-dns deployment.

isn't time to get rid of the toleration in in the kube-dns addon ? @thockin @bsalamat

[BouweCeunen]

would love to see this implemented so I can set some tolerations on those pods.

[VengefulAncient]

Please look into this issue, it's been years. kube-dns is already quite problematic in smaller clusters due to its high CPU requests and pod anti-affinity not really working, resulting in it choking out workloads from smaller nodes, and not having custom tolerations means it can't even be pinned to a node where it won't affect anything.

[aojea]

As of Kubernetes v1.12, CoreDNS is the recommended DNS Server, replacing kube-dns.

[VengefulAncient]

kube-dns is still used by GKE even on the latest Kubernetes version. This is not a solution.

[aojea]

point is that is a GKE issue, not a kubernetes or sig-network issue 😅
/remove sig-network