Add the ability to migrate CoreDNS configmap in kubeadm #78033

rajansandeep · 2019-05-17T14:27:57Z

What type of PR is this?

/kind feature

What this PR does / why we need it:
Currently in kubeadm, when upgrading from Kubernetes (Eg. k8s 1.13 -> 1.14), the CoreDNS ConfigMap is retained as-is and applied to the upgraded version. This is done due to the possibility that the user could have customized the Corefile for their use-case.

This approach is disadvantageous because this can lead to an incompatible Corefile which can lead to CoreDNS entering a CrashLoopBackOff state, and hence DNS not working in the cluster.

This PR adds the ability for users to seamlessly migrate their CoreDNS Configuration when they upgrade their clusters using kubeadm. The CoreDNS Migration library helps to handle migrations of CoreDNS Corefiles to be compatible with new versions of CoreDNS.

The PR includes 3 preflight-checks while upgrading to check:

If the CoreDNS image is an official release
If the Migration of CoreDNS is possible (Checks if there are any errors while migrating the CoreDNS Config from the current version to the next version)
If there are any CoreDNS plugins present that are unsupported by the migration library.

The user will be provided Logs and can override these checks.

During Upgrade:

There is a check to see if the CoreDNS Config is the default (not edited). In that case, the user will be upgraded to the new default CoreDNS Config.
If the CoreDNS Config has been modified, then the user will be migrated to the equivalent compatible Config for the latest version of CoreDNS. This will include handling all deprecations that may be present.

The user will be informed appropriately through logs.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:
The first commit includes the code change required for this PR.
The second commit includes the vendor and go mod changes.

This PR also updates the CoreDNS version to 1.5.0
Does this PR introduce a user-facing change?:

Kubeadm now seamlessly migrates the CoreDNS Configuration when upgrading CoreDNS.

rajansandeep · 2019-05-17T14:28:47Z

/cc @chrisohaver @neolit123

neolit123

thanks for the PR @rajansandeep
i understand the motivation of this change thus i've spent time doing a detailed review.

we might need at least some unit tests and possibly extend the release note that preflight checks are run now.

ideally more eyes from @kubernetes/sig-cluster-lifecycle-pr-reviews are needed here.

neolit123 · 2019-05-17T15:00:14Z

cmd/kubeadm/app/cmd/upgrade/common.go

@@ -92,6 +93,10 @@ func enforceRequirements(flags *applyPlanFlags, dryRun bool, newK8sVersion strin
 		return nil, nil, nil, errors.Wrapf(err, "couldn't create a Kubernetes client from file %q", flags.kubeConfigPath)
 	}

+	if err := runCoreDNSpreflightChecks(client, ignorePreflightErrorsSet); err != nil {


the upgrade checks should have a single entry point: runPreflightChecks
the call to runCoreDNSpreflightChecks() should be moved there.
client needs to be passed to runPreflightChecks.

neolit123 · 2019-05-17T15:02:36Z

cmd/kubeadm/app/cmd/upgrade/common.go

+		return err
+	}
+	if currentDNSType == kubeadmapi.CoreDNS {
+		fmt.Println("[preflight] Running CoreDNS migration pre-flight checks.")


we can omit this notification, the preflight package will show check names.

neolit123 · 2019-05-17T15:05:01Z

cmd/kubeadm/app/cmd/upgrade/common.go

+	}
+	if currentDNSType == kubeadmapi.CoreDNS {
+		fmt.Println("[preflight] Running CoreDNS migration pre-flight checks.")
+		return upgrade.RunCoreDNSMigrationCheck(client, ignorePreflightErrors)


currentDNSType, _, err := dns.DeployedDNSAddon(client) if err != nil { return err } if currentDNSType == kubeadmapi.CoreDNS { ...

can be inside upgrade.RunCoreDNSMigrationCheck, and optionally make the function a no-op if kube-dns is used.
(i.e. exit early and don't run preflight.RunChecks(migrationChecks, os.Stderr, ignorePreflightErrors))

neolit123 · 2019-05-17T15:08:21Z

cmd/kubeadm/app/phases/addons/dns/dns.go

+		}
+	}
+	// Get the Corefile and installed CoreDNS version.
+	corefile, currentInstalledCoreDNSVersion, err := GetCoreDNSInfo(client)


GetCoreDNSInfo() can accept a *ConfigMap as an argument.
if nil it should fetch the CM, if non-nil it should use the given CM.
this avoids fetching the same CM twice.

neolit123 · 2019-05-17T15:08:54Z

cmd/kubeadm/app/phases/addons/dns/dns.go

+	if !migration.Default("", corefile) {
+		updatedCorefile, err := migration.Migrate(currentInstalledCoreDNSVersion, kubeadmconstants.CoreDNSVersion, corefile, false)
+		if err != nil {
+			return errors.Wrap(err, "unable to migrate CoreDNS Configmap")


Configmap -> ConfigMap

neolit123 · 2019-05-17T15:40:19Z

cmd/kubeadm/app/phases/addons/dns/dns.go

@@ -297,6 +298,83 @@ func createDNSService(dnsService *v1.Service, serviceBytes []byte, client client
 	return nil
 }

+func createCoreDNSConfigMap(client clientset.Interface, cm *v1.ConfigMap) error {


can we get an unit test for this function?

neolit123 · 2019-05-17T15:40:59Z

cmd/kubeadm/app/phases/upgrade/migrate.go

+// checkUnsupportedPlugins checks if there are any plugins included in the current configuration
+// that is unsupported for migration.
+func checkUnsupportedPlugins(client clientset.Interface) error {
+	klog.V(1).Infoln("validating if there are any unsupported CoreDNS plugins in the Config")


Config->Corefile?

neolit123 · 2019-05-17T15:41:29Z

cmd/kubeadm/app/phases/upgrade/migrate.go

+
+		}
+		if UnsupportedPlugins != "" || UnsupportedVersion != "" {
+			return errors.Errorf("there are unsupported plugins in the CoreDNS Configuration")


Configuration->Corefile?

neolit123 · 2019-05-17T15:42:40Z

cmd/kubeadm/app/phases/upgrade/migrate.go

+
+// Checks if the CoreDNS image currently installed is an official released version.
+func checkRelease(client clientset.Interface) error {
+	klog.V(1).Infoln("validating if the image installed is the  official release of CoreDNS.")


extra space before official.

neolit123 · 2019-05-17T15:51:53Z

cmd/kubeadm/app/phases/upgrade/migrate.go

+
+	isValid := migration.Released(coreDNSImageSha[1])
+	if !isValid {
+		return errors.Errorf("the CoreDNS image currently installed is not an official image. If you still wish to proceed, do so at your own risk.")


this is not a check we have for the control plane or etcd images.
to my understanding, if someone has built coredns from source at any SHA, migration.Released would return false. while this is still not recommended, blocking on it with a preflight check might be a bit too strict.

but possibly needs discussion / feedback from more people.

+1 for not having this. Users in China may be implicated.

I think it is generally safe to remove this check.

The main reason for adding this check is to prevent external plugin users from inadvertently replacing their custom CoreDNS with a stock one. A stock CoreDNS will crash when it encounters an unknown plugin in the Corefile at start up.

But most users just use a stock version of CoreDNS. And IIUC, those who use CoreDNS compiled with external plugins will upgrade using "phases", and manually upgrade CoreDNS with an updated version compiled with the required external plugins if desired.

cblecker · 2019-05-29T00:54:12Z

Is this needed/intended to merge before 1.15 code freeze?

rajansandeep · 2019-05-29T01:00:16Z

@cblecker Yes! This is needed for 1.15

cblecker · 2019-05-29T01:19:25Z

@rajansandeep Looks like the caddy dependency needs to be updated.

Can you please run:

hack/pin-dependency.sh github.com/mholt/caddy v1.0.0
hack/pin-dependency.sh github.com/BurntSushi/toml v0.3.1
hack/pin-dependency.sh github.com/golang/mock v1.2.0
hack/pin-dependency.sh github.com/google/uuid v1.1.1
hack/pin-dependency.sh github.com/gorilla/websocket v1.4.0
hack/pin-dependency.sh github.com/miekg/dns v1.1.3
hack/pin-dependency.sh github.com/onsi/ginkgo v1.8.0
hack/pin-dependency.sh github.com/onsi/gomega v1.5.0
hack/pin-dependency.sh github.com/russross/blackfriday v0.0.0-20170610170232-067529f716f4
hack/pin-dependency.sh github.com/stretchr/testify v1.3.0
hack/pin-dependency.sh gopkg.in/natefinch/lumberjack.v2 v2.0.0
hack/pin-dependency.sh gopkg.in/square/go-jose.v2 v2.2.2
hack/pin-dependency.sh gopkg.in/yaml.v2 v2.2.2
GO111MODULE=on go mod edit -dropreplace github.com/natefinch/lumberjack
GO111MODULE=on go mod edit -dropreplace github.com/shurcooL/sanitized_anchor_name
GO111MODULE=on go mod edit -dropreplace gopkg.in/yaml.v1
hack/update-vendor.sh

And commit the changes? Then when complete, include a comment with the output of hack/lint-dependencies.sh

neolit123 · 2019-05-29T01:43:44Z

Can you please run:

looks simple enough 🙄

/priority important-soon
/milestone v1.15

neolit123 · 2019-08-13T22:39:07Z

/remove-area apiserver cloudprovider

neolit123 · 2019-08-13T22:40:53Z

/approve
/assign @rosti

@rosti PTAL and LGTM if you think this is good.
at this point i prefer if we merge the PR, run it trough e2e and iterate if needed before 1.16.

neolit123 · 2019-08-13T23:38:32Z

@rajansandeep

if you are changing the version of coredns-kubeadm, make sure all of the following files are updated with the newest version including /home/prow/go/src/k8s.io/kubernetes/build/dependencies.yaml
then run ./hack/verify-external-dependencies-version.sh

https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/78033/pull-kubernetes-verify/1161403539239473152

build/dependencies.yaml needs an update at this point.

rajansandeep · 2019-08-15T20:59:03Z

This still needs root approval for the changes in vendor/deps...

neolit123 · 2019-08-15T21:11:38Z

/assign @liggitt

PTAL WRT approval of the /vendor changes.
we can amend on the configmap/upgrade logic if needed in follow ups.

thanks!

BenTheElder · 2019-08-15T21:15:37Z

cmd/kubeadm/app/phases/addons/dns/manifests.go

@@ -314,9 +314,9 @@ data:
    .:53 {
        errors
        health
+        ready


FYI @aojea the ipv6 config may need tweaking.

BenTheElder · 2019-08-15T21:19:02Z

vendor/github.com/coredns/corefile-migration/migration/caddy/alltokens.go

@@ -0,0 +1,33 @@
+// Copyright 2015 Light Code Labs, LLC


is this package a copy from a different upstream? do we have accurate licensing info for this (in Godeps/) ?
is there a reason this isn't imported from the original authors? was it modified?

^ @chrisohaver

Originally I just imported the caddy package. But at the suggestion of reviewers concerned for the number of imported packages, I copied the required pieces locally. The licensing was copied verbatim from the original project. IIRC, I had to modify it slightly (a function moved from one package to another), to reduce the amount of copied code.

do we have accurate licensing info for this (in Godeps/) ?

Sorry, I'm not familiar with the licensing process ... Please let me know what I need to do here to make everyone happy.

Originally I just imported the caddy package. But at the suggestion of reviewers concerned for the number of imported packages, I copied the required pieces locally. The licensing was copied verbatim from the original project.

I see, that makes sense I suppose.

IIRC, I had to modify it slightly (a function moved from one package to another), to reduce the amount of copied code.

So you modified one of the source files then? Do we need to denote this somehow?

Sorry, I'm not familiar with the licensing process ... Please let me know what I need to do here to make everyone happy.

I'm not actually sure either, I haven't seen this case before.

So you modified one of the source files then? Do we need to denote this somehow?

I recall I had to do that to stop the "bleeding" ... otherwise i'd be duplicating most caddy. I don't know if we need to or how to properly denote this (comments in the files? readme in same directory?). IMHO, copying files like this feels a hack to me, and I'd prefer just importing the packages normally. OTOH, I don't have a good understanding of the concern for importing the package in the first place, other than it created a lot of dependencies (which I infer is bad).

I propose we just import the packages normally, and remove the copied files. Any objections?

@chrisohaver -- yeah, I'd feel better about that route.

/hold
(until that's resolved.)

I fixed the part. @justaugustus PTAL and cancel the hold if you feel it's good. Thanks!

Thanks for fixing the licensing piece, @rajansandeep!
/hold cancel

liggitt · 2019-08-16T03:46:22Z

I had the same question w.r.t. that package license

…up data key

neolit123 · 2019-08-20T20:43:30Z

/lgtm
/approve
/retest

rajansandeep · 2019-08-20T20:44:57Z

Looks like all hurdles have been crossed and requires one final approval tag.
/assign @liggitt

liggitt · 2019-08-21T01:59:18Z

/approve

k8s-ci-robot · 2019-08-21T01:59:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fabriziopandini, liggitt, neolit123, rajansandeep

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Add the ability to migrate CoreDNS configmap in kubeadm Kubernetes-commit: e1c2c67

k8s-ci-robot requested review from lavalamp, yagonobre and a team May 17, 2019 14:28

k8s-ci-robot requested review from chrisohaver and neolit123 May 17, 2019 14:28

chrisohaver approved these changes May 17, 2019

View reviewed changes

neolit123 reviewed May 17, 2019

View reviewed changes

rajansandeep mentioned this pull request May 18, 2019

Bump CoreDNS version to 1.5.0 and update manifest #78030

Merged

chrisohaver mentioned this pull request May 20, 2019

[Failing tests] Multiple job failures on the BeforeSuite step while waiting for CoreDNS to be ready #78123

Closed

alejandrox1 mentioned this pull request May 20, 2019

[Failing Tests] DNS configmap nameserver tests failing in master-blocking #78129

Closed

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels May 28, 2019

k8s-ci-robot added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label May 29, 2019

k8s-ci-robot added this to the v1.15 milestone May 29, 2019

k8s-ci-robot removed the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label May 29, 2019

k8s-ci-robot removed sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Aug 13, 2019

k8s-ci-robot removed area/apiserver area/cloudprovider labels Aug 13, 2019

k8s-ci-robot assigned rosti Aug 13, 2019

add the ability to migrate coredns configmap

6821d21

BenTheElder reviewed Aug 15, 2019

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 20, 2019

rajansandeep added 3 commits August 20, 2019 13:20

move the corefile backup into a single configmap with a corefile-back…

6bb9eee

…up data key

update build dep

e617f37

update dependencies

b6da3e4

k8s-ci-robot added area/kubectl sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/cli Categorizes an issue or PR as relevant to SIG CLI. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Aug 20, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 20, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 21, 2019

k8s-ci-robot merged commit e1c2c67 into kubernetes:master Aug 21, 2019

wking pushed a commit to wking/kubernetes that referenced this pull request Jul 21, 2020

Merge pull request kubernetes#78033 from rajansandeep/corednsmigration

09d93bf

Add the ability to migrate CoreDNS configmap in kubeadm Kubernetes-commit: e1c2c67

@@ @@ -314,9 +314,9 @@ data: @@
                   .:53 {
                       errors
                       health
+                      ready

Add the ability to migrate CoreDNS configmap in kubeadm #78033

Add the ability to migrate CoreDNS configmap in kubeadm #78033

Conversation

rajansandeep commented May 17, 2019 • edited

rajansandeep commented May 17, 2019

neolit123 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cblecker commented May 29, 2019

rajansandeep commented May 29, 2019

cblecker commented May 29, 2019

neolit123 commented May 29, 2019

neolit123 commented Aug 13, 2019

neolit123 commented Aug 13, 2019

neolit123 commented Aug 13, 2019 • edited

rajansandeep commented Aug 15, 2019

neolit123 commented Aug 15, 2019

Choose a reason for hiding this comment

BenTheElder Aug 15, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt commented Aug 16, 2019

neolit123 commented Aug 20, 2019

rajansandeep commented Aug 20, 2019

liggitt commented Aug 21, 2019

k8s-ci-robot commented Aug 21, 2019

rajansandeep commented May 17, 2019 •

edited

neolit123 commented Aug 13, 2019 •

edited

BenTheElder Aug 15, 2019 •

edited