Use a Deployment for kube-dns

[MrHohn]

Attempt to fix #31554

Switching kube-dns from using Replication Controller to Deployment.

The outdated kube-dns YAML file in coreos and juju dir is also updated. Most of the specific memory limit in the files remain unchanged because it seems like people were modifying it explicitly(c8d82fc). Only the memory limit for healthz is increased due to this pending investigation(#29688).

YAML files stay in *-rc.yaml format considering there are a lots of scripts in cluster and hack dirs are using this format. But it may be fine to changed them all.

@bprashanth @girishkalele

---

This change is

[MrHohn]

cc @thockin

[roberthbailey]

Have we tested upgrades from a version where kube-dns is running as an RC (e.g. 1.3.6) to a cluster where it's running as a deployment (with this PR) to ensure that it works as expected? And then tested upgrading to a version with a newer deployment to ensure that it works properly? The advantage of using a deployment is the update semantics (you could tell the deployment to update instead of deleting & recreating it), but I don't believe that the addon manager will actually do that during an update, which negates the advantages of switching.

While it would be nice if all cluster addons ran in deployments, this seems to introduce churn unnecessarily. @kubernetes/sig-cluster-lifecycle is working on replacing the current addon management system with something nicer (as part of the kubeadm work) so it might be better to just defer trying to clean this up in the mean time (adding extra review & testing time) and wait for the overhaul to the addon management system.

[MrHohn]

It is correct that the current addon manager would not take the advantage of deployment during upgrade. As below codes indicate:

function update-object() {
    local -r obj_type=$1
    local -r namespace=$2
    local -r obj_name=$3
    local -r file_path=$4
    log INFO "updating the ${obj_type} ${namespace}/${obj_name} with the new definition ${file_path}"
    delete-object ${obj_type} ${namespace} ${obj_name}
    create-object ${obj_type} ${file_path}
}

And switching kube-dns to using deployment has not been completely tested. Should hold this off with regard to testing and addon manager's upgrade.

[thockin]

Can someone speak to what the lifecycle work is, if not a deployment?

On Wed, Sep 7, 2016 at 8:40 AM, Robert Bailey notifications@github.com
wrote:

Have we tested upgrades from a version where kube-dns is running as an RC
(e.g. 1.3.6) to a cluster where it's running as a deployment (with this PR)
to ensure that it works as expected? And then tested upgrading to a version
with a newer deployment to ensure that it works properly? The advantage of
using a deployment is the update semantics (you could tell the deployment
to update instead of deleting & recreating it), but I don't believe that
the addon manager will actually do that during an update, which negates the
advantages of switching.

While it would be nice if all cluster addons ran in deployments, this
seems to introduce churn unnecessarily. @kubernetes/sig-cluster-lifecycle
https://github.com/orgs/kubernetes/teams/sig-cluster-lifecycle is
working on replacing the current addon management system with something
nicer (as part of the kubeadm work) so it might be better to just defer
trying to clean this up in the mean time (adding extra review & testing
time) and wait for the overhaul to the addon management system.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#32018 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVOSzTCTYDN5E6vRqSeAbbUwA6eVeks5qntrlgaJpZM4J0KFt
.

[bprashanth]

I'm guessing a rewrite of this shell script: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/addon-manager/kube-addon-update.sh

Recreating a deployment is useless, by applying the update should work.

[bprashanth]

Eitherways I guess we'd be looking at 1.5 rollout, so planning for all addons in deployments and using apply instead of create/delet in the new addon controller SGTM (if that's the proposed plan of action)

[thockin]

I think apply is correct, and we will need to be able to change from RC
to Deployment in a number of places.

On Wed, Sep 7, 2016 at 10:11 AM, Prashanth B notifications@github.com
wrote:

Eitherways I guess we'd be looking at 1.5 rollout, so planning for all
addons in deployments and using apply instead of create/delet in the new
addon controller SGTM (if that's the proposed plan of action)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#32018 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVPxYNV2QnH0fEep5yTPRR2rpZ1pSks5qnvBTgaJpZM4J0KFt
.

[mikedanese]

#29551 is the last feature I was hoping for in kubectl apply before using it for addons. Eventual goal for addon management is while true; do kubectl apply -f addons/ and move all RCs to deployments.

[thockin]

do we have a plan for how to delete an addon that used to be an RC and
replace it with a deployment? That is this case.

On Wed, Sep 7, 2016 at 10:32 AM, Mike Danese notifications@github.com
wrote:

#29551 #29551 is the last
feature I was hoping for in kubectl apply before using it for addons.
Eventual goal for addon management is while true; do kubectl apply -f
addons/ and move all RCs to deployments.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#32018 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVFHMbEde9DJsA6qplAIsMQwvRRWwks5qnvU1gaJpZM4J0KFt
.

[bprashanth]

we can orphan the pods (update rc selectos and scale to 0 in one update) then create the deployment so the rs adopts them

[thockin]

We just need a way to describe the transition - dropping files into a dir
is insufficient.

On Wed, Sep 7, 2016 at 10:47 AM, Prashanth B notifications@github.com
wrote:

we can orphan the pods (update rc selectos and scale to 0 in one update)
then create the deployment so the rs adopts them

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#32018 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVATOuYbFR56xjHPEJctVWNzHOcoLks5qnviagaJpZM4J0KFt
.

[mikedanese]

Isn't that what #29551 does?

[bprashanth]

From a quick scan, I don't want to delete the dns pods i just want to delete the rc with --cascade=false, then deployment upgrade the dns pods to the new version

[mikedanese]

I think that can be covered with the use of a label selector with kubectl apply --prune. We would need a query that matches only the top level addon-managed resources and not the resources that they create. kubernetes.io/cluster-service: "true" would be a good candidate but it looks like the dns rc is (probably incorrectly) setting that label on pods it creates:

https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/skydns-rc.yaml.in#L40

Regardless, the primary motivation of #29551 is exactly this so if you see something that is broken about the proposal it would be helpful if you pointed it out.

[k8s-bot]

GCE e2e build/test passed for commit b5c17fa.

• Test Results
• Build Log
• Test Artifacts
• Internal Jenkins Results

[k8s-github-robot]

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

[k8s-bot]

GCE e2e build/test passed for commit b5c17fa.

• Test Results
• Build Log
• Test Artifacts
• Internal Jenkins Results

[k8s-github-robot]

Automatic merge from submit-queue

[bprashanth]

Wait, we didn't actually want this to merge correct?
@MrHohn @mikedanese
or was the plan to get it in 1.5 and since we've done 1.4 cherrypicks it's ok?

[MrHohn]

Oh, I do think it should not be merged too. Sorry

[roberthbailey]

@thockin added an lgtm label 8 days ago and i guess we never removed it...

[thockin]

I think it's fine since were in cherrypick mode. Do not cherrypick this.
but WE MUST FINISH THE UPGRADE STORY, ASAP.

Fair?

On Sat, Sep 10, 2016 at 11:48 PM, Robert Bailey notifications@github.com
wrote:

@thockin https://github.com/thockin added an lgtm label 8 days ago and
i guess we never removed it...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#32018 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVFwjj2CXJXoJuOSnmO8BzopOwKFyks5qo6QvgaJpZM4J0KFt
.

[gmarek]

I merged the revert of this PR - it completely broke Rescheduler tests.

[piosz]

Once re-reverting please ensure that you update also Rescheduler e2e.

[thockin]

@Mrhohh we should talk about this one. I am WFH this morning, but let's
catch up later.

On Mon, Sep 12, 2016 at 1:01 AM, Piotr Szczesniak notifications@github.com
wrote:

Once re-reverting please ensure that you update also Rescheduler
https://github.com/kubernetes/kubernetes/blob/master/test/e2e/rescheduler.go
e2e.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#32018 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVBjUNjJEF9NKgReQZ_IBNREXw5IVks5qpQbLgaJpZM4J0KFt
.

[MrHohn]

@piosz Sorry about the mistake. I was not aware of this e2e test before.

@thockin Yeah, no problem.

[piosz]

@MrHohn no worries. It's not obvious.

[ApsOps]

@thockin Was this merged in v1.4.x or not?

I see the change listed in CHANGELOG, but don't see them when I checkout the v1.4.6 tag. 🤔

[MrHohn]

@ApsOps Sorry for the confusion. This is not in v1.4.x. This PR got reverted by #32452 a day after it was merged.

kube-dns are now using a Deployment (introduced by #36008). And it is part of v1.5.

[ApsOps]

Got it. Thanks!

…

On Thu, 1 Dec 2016 at 11:15 PM, Zihong Zheng ***@***.***> wrote: @ApsOps <https://github.com/ApsOps> Sorry for the confusion. This is not in v1.4.x. This PR got reverted by #32452 <#32452> a day after it was merged. kube-dns are now using a Deployment (introduced by #36008 <#36008>). And it is part of v1.5. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#32018 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACQOwpJ_sVS8i5Y6XD9EVmpRaJ3qSnJHks5rDwepgaJpZM4J0KFt> .