Upgrade to 1.16.x failed if CoreDNS was previously deployed with self-built image

[vryzhenkin]

What happened:
Upgrade using kubeadm failed because of CoreDNS migration enchancement

What you expected to happen:
Upgrade passed correctly

How to reproduce it (as minimally and precisely as possible):

1. Use any 1.15.x Kubernetes with CoreDNS that deployed with any unusual image (self built for example). We using commits counter as part of image tag, e.g - 1.6.2-100.
2. Try to upgrade to 1.16.x with kubeadm.

Anything else we need to know?:
Logs part without ignoring preflight:

[preflight] Some fatal errors occurred:
    [ERROR CoreDNSUnsupportedPlugins]: start version 'v1.6.2-100' not supported
    [ERROR CoreDNSMigration]: start version 'v1.6.2-100' not supported

Logs part with ignoring preflight:

unable to migrate CoreDNS ConfigMap: start version 'v1.6.2-100' not supported
[upgrade/postupgrade] FATAL post-upgrade error
k8s.io/kubernetes/cmd/kubeadm/app/cmd/upgrade.runApply
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/upgrade/apply.go:182
k8s.io/kubernetes/cmd/kubeadm/app/cmd/upgrade.NewCmdApply.func1
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/upgrade/apply.go:80
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:830
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864
k8s.io/kubernetes/cmd/kubeadm/app.Run
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
    _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
    /usr/local/go/src/runtime/proc.go:200
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1337

Environment:

• Kubernetes version (use kubectl version):

root@kaas-node-20262e33-f6a9-11e9-a537-02420a0f0f01:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.3-8+d06f4e3032941e", GitCommit:"d06f4e3032941e9e8099845fdde660d482f571fb", GitTreeState:"clean", BuildDate:"2019-09-13T14:22:14Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.3-8+d06f4e3032941e", GitCommit:"d06f4e3032941e9e8099845fdde660d482f571fb", GitTreeState:"clean", BuildDate:"2019-09-13T14:24:40Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  OpenStack
• OS (e.g: cat /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

• Kernel (e.g. uname -a):
  Linux kube-node-01 4.15.0-51-generic #55-Ubuntu SMP Wed May 15 14:27:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
  Kubeadm

[vryzhenkin]

/sig cluster-lifecycle

[tedyu]

The error came from vendor/github.com/coredns/corefile-migration/migration/migrate.go

It seems you should mention this issue on coredns project

[vryzhenkin]

@tedyu True. Will notify them also.

[chrisohaver]

/cc @rajansandeep

[rajansandeep]

/assign
In such cases where the migration isn't supported and the user chooses to ignore the warnings from the preflight check, the migration should be skipped.

wdyt? /cc @chrisohaver @neolit123

[chrisohaver]

Yes. IMO, the warnings should be phrased like "CoreDNS configuration migration will not occur: reason"... so that a user expects the config migration will not be done.

[neolit123]

/area kubeadm

[neolit123]

@rajansandeep

In such cases where the migration isn't supported and the user chooses to ignore the warnings from the preflight check, the migration should be skipped.

SGTM

[chrisohaver]

What you expected to happen:
Upgrade passed correctly

I think the best we can do here is to allow the upgrade to pass, but this will, in most practical cases, result in a cluster without a functioning DNS. The most common practical use-case for using a self built image of CoreDNS is to use external plugins. During the kubeadm upgrade, CoreDNS will be replaced with a stock release version of CoreDNS. When that new version of CoreDNS starts, it will not be able to load the existing Corefile's external plugins, and exit.

... or, we could finish the upgrade, leaving the existing CoreDNS deployment untouched, which I think works better.

[vryzhenkin]

@chrisohaver Yea, it's kinda strong words that "Upgrade passed correctly". Probably, I think the best solution will be to make possible skip Corefile migration and make kubeadm finish the upgrade without Corefile changes. Guys that using custom images of CoreDNS should care about Corefile for new version by themselves, IMO.

[chrisohaver]

I think the best solution will be to make possible skip Corefile migration and make kubeadm finish the upgrade without Corefile changes. Guys that using custom images of CoreDNS should care about Corefile for new version by themselves, IMO.

Yes, I think that would work best...