-
Notifications
You must be signed in to change notification settings - Fork 6.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CoreDNS pods left in crashloopbackoff state after running upgrade #6596
Comments
Yes we got a lot of issue with that, that's why we are now checking (when PR are created) and ensuring that CoreDNS version is supported by Corefile-migration bundled with Kubeadm |
Also the bug with configmap being left erroneous was also fixed on kubernetes end kubernetes/kubernetes#88811 I suggest to close this issue as it should not happen with recent version of either spray or kubernetes |
Sounds good.
I am happy to see there is a fix to the root cause.
Do you know in which Kubernetes version this is fixed?
Should I still bother creating a PR with my proposed work around with a
TODO to remove it once older versions are not supported or you think it's
not worth it?
I'm just wondering if we're the only ones bothered by the issue or not :-)
Cheers,
…On Thu, Aug 27, 2020, 13:39 Florian Ruynat ***@***.***> wrote:
Also the bug with configmap being left erroneous was also fixed on
kubernetes end kubernetes/kubernetes#88811
<kubernetes/kubernetes#88811>
I suggest to close this issue as it should not happen with recent version
of either spray or kubernetes
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#6596 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB2IFBOTZMVGTAEU5WI6E3TSCZAYJANCNFSM4QM2QFBQ>
.
|
Looks like the fix in Kubernetes landed in 1.19, so pretty recent. That's nice of you to have a patch and might be useful for some people (if they land on this issue while searching) so you may paste it here. |
/close |
@floryut: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Thanks @floryut . |
Environment:
Kubespray version (commit): 39fa950
Network plugin used: calico and flannel
Full inventory with variables:
Relevant bits from k8s-cluster.ym group vars:
kube_version: v1.16.10
kube_network_plugin: calico
dns_mode: coredns
Command used to invoke ansible:
ansible-playbook -b -i $inventory kubespray/upgrade-cluster.yml -vv
Output of ansible run:
See gist
The interesting bit is:
Anything else do we need to know:
CoreDNS version deployed is 1.6.5
Analysis
It seems running kubeadm upgrade, causes it to attempt migrating CoreDNS (whether or not it needs to).
But a specific kubeadm version seems to only support migration of some versions of CoreDNS (which would make sense).
In some scenario, kubeadm will not be able to migrate CoreDNS and will:
This results in the following invalid deployment:
The path to config file as specified to the container command and the path where the config is actually mounted are different.
I'm not sure what solution there is as upgrading kubeadm/kubernetes is not always easy in prod environments.
The best I can think of, is a workaround to detect this migration failure after kubeadm upgrade was run and to fix it the deployment then.
I did create such a patch and it is working. But perhaps someone has a more elegant solution in mind?
Cheers,
The text was updated successfully, but these errors were encountered: