DaemonSet controller doesn't handle NoSchedule taints correctly

[mikedanese]

Details in #44445 (comment)

@dchen1107 consider adding this to the 1.7 release although it appears to have been broken in 1.6. I'll add 1.7 milestone for you to remove if this doesn't make the cut.

Fix is here #48189

cc @kubernetes/sig-apps-bugs

[mikedanese]

Summary: DaemonSet controller evicts running pods when a NoSchedule taint is added to the node. In this case the pod should instead continue to run.

[k82cn]

/cc

[dchen1107]

I am still processing all the information being linked, and trying to summarize what I understand so far:

• There is a regression in daemonset controller (DSC) in 1.6. Basically there is a simulation introduced in DSC in 1.6, it checks if a given daemonset can be place on a node. If it fails, DSC would not schedule the newly coming daemonset or kill the running one on the node.
• Fluentd is a daemonset which currently tolerates both NoSchedule and NoExecute taints. But DSC doesn't respect NoSchedule and still kill the running fluentd on the node.
• The proposed solution is fixing DSC simulation to take NoSchedule taint into the consideration.

@gmarek @mikedanese Am I understanding this correctly?

If my understanding above is correct, can someone help me to understand the following issues, so that we can decide how urgent it is and should we include this in v1.7.0:

• Why we never catch the problem in 1.6 release process? Lack of testing coverage? Shouldn't by default GCE / GKE master is marked with NoSchedule taint, and all fluentd should fail there?
• We are running 1.6.7 release in production now, why the problem is emerging 3 month later? How come we didn't catch the issue on GKE master nodes?

I am inclined to punt this to v1.7.1 so that we can have a better understanding on the issue and proper fix with more test coverage. But I can be convinced on this by the severity in the production today.

cc/ @davidopp @erictune @kubernetes/kubernetes-release-managers

[mikedanese]

The problem is DS treats NoSchedule like it is NoExecute. The minute difference between the effects is NoSchedule taint should allow already running pods that are intolerant to continue running. Today it deletes them.

Shouldn't by default GCE / GKE master is marked with NoSchedule taint, and all fluentd should fail there?

That was reverted in GCE before 1.6 due to backwards compatibility. The master in GKE doesn't even register.

We are running 1.6.7 release in production now, why the problem is emerging 3 month later?

Poor test coverage for a new feature (NoSchedule in DS introduced in 1.6, NoExecute in 1.7). This came from a few separate user reports.

How come we didn't catch the issue on GKE master nodes?

GKE masters don't register. Taints are only relevant to nodes that register.

[mikedanese]

The change I link includes the missing test coverage.

[davidopp]

This definitely needs to be fixed in 1.7. It's a serious bug.

[davidopp]

And we should cherrypick it into the next 1.6 patch release.

[yastij]

So this one is making it to 1.7 ?

[dchen1107]

I agreed this is a serious bug, but I am not still clear on why this should have such high severity in production, and we even want to delay the new release v1.7.0 for this last-minute-found regression introduced in v1.6.

Basically the change / fix might have deep impact we might ignore for now under the release pressure. I would rather we speed time on a clear solution with a better test coverage and longer soaking time.

[kow3ns]

Would it be possible to mark the Pods in question as tolerating the NoSchedule taint to prevent the eviction of Running critical Pods when a Node is tainted with NoSchedule?
If so, could we use this as a valid workaround and defer the implementation of a fix to the next point release?
My concern is the same as @dchen1107, we could potentially introduce instability into 1.7 directly prior to release.

[erictune]

Given that this is in 1.6, IIUC, then deferring the fix to 1.7.1 seems reasonable to me.

[davidopp]

sig-cluster-lifecycle owns the DaemonSet controller and should make the decision (together with the release team).

@kow3ns We could change the fluentd config to toleration NoSchedule taint, but that would only affect new pods. Not sure if it is worth it if we are going to fix it in 1.7.1.

[0xmichalis]

sig-cluster-lifecycle owns the DaemonSet controller and should make the decision (together with the release team).

I thought that was sig-apps (or at least a joint effort between the two sigs).

[davidopp]

We need to have every controller owned by a single SIG (of course other SIGs can be "consultants"). I'm going to create a spreadsheet right now and send it out to kubernetes-dev for discussion. But that is for the long-term. For right now, if sig-apps owns the DaemonSet controller, then they should make the decision.

[davidopp]

(Actually I think we may have had that in the community repo somewhere... I'll look)

[davidopp]

Anyway it seems that most people seem to think we should not put the fix in 1.7, but put it in 1.7.1 and cherrypick to the next 1.6 release. I guess that is the right thing to do, but let's make sure the release notes are very clear about this problem.

[enisoc]

As for 1.6, I have been asked to cut 1.6.7 potentially as early as next week to fix another issue (#48108). Keeping in mind the holidays, do you think the fix for this DS issue will be ready to cherrypick by the end of this week? If not, please let me know if you consider this issue severe enough that I should hold the 1.6.7 train for it.

[davidopp]

Well, we need to find a reviewer for #48189.

But since we are releasing 1.7 without this fix, I guess there's no need to hold 1.6.7 for it either.

[gmarek]

I think we should put it into 1.6.7 - this is an issue people are actually complaining about (I heard about it from multiple sources)

[dchen1107]

xref: #46733

@mikedanese @gmarek can you put together a paragraph to describe this known issue for 1.7.0 and pre-1.6.7 release. Thanks!

[enisoc]

@gmarek Do you think it's enough for 1.6.7 to cherrypick #48182?

cc @crassirostris

[gmarek]

That's the most painful part, so I think that should be suffice. Assuming that we'll cut 1.6.8 in not too distant future.

[mikedanese]

Will close after cherrypicks are in

[kow3ns]

@mikedanese are we good to close this?

[mikedanese]

Do we want to cherrypick to 1.6? I'd say no...