runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized #1031
Labels
area/ecosystem
priority/awaiting-more-evidence
Lowest priority. Possibly useful, but not yet enough support to actually get it done.
Comments
|
@lukaszgryglicki Name: cncftest.io
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linuxand Name: devstats.team.io
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linuxI'm not expert of flannel, but I think you should check product documentation for how to make it work in environment with mixed platforms |
|
That's a good point, but how about error message - it seems really unrelated. It looks like some CNI config files are missing in Labels with arch name don't match indeed. But next label |
|
The same happens with Docker 18.03ce. I don't see any difference, this is not looking like a docker issue. This looks like some CNI config issue. |
|
@lukaszgryglicki
these are some nice specs. the way i test things is the following - if something doesn't work with weavenet, i try flannel and the other way around. so please try weave and if your CNI setup works with it, then this is CNI plugin related. while the kubeadm team supports plugins and addons, we usually delegate issues to their respective maintainers because we don't have the bandwidth to handle everything. |
neolit123
added
area/ecosystem
priority/awaiting-more-evidence
|
Sure, I've tried weave a few iterations ago. It ended in a container restart loop. |
|
So this is not a docker issue. On 17.03 the same: I'll try weave now and post results here. |
|
So, I've tried
|
|
So, summing up. It is impossible to install Kubernetes cluster with just single master and single worker node on Ubuntu 18.04. |
|
i think 18.04 broke both in terms of the Docker it bundles and for also while kubeadm is the frontend here, the issue really could be unrelated to kubeadm itself. some questions:
|
node (cncftest.io, amd64): node (devstats.cncf.io, amd64):
|
|
So, I've installed master
|
|
There is a small progress.
Any suggestions what should I do? Where should I report this? I already have a 2 nodes cluster (master and node amd64) but I want to help resolve this issue so one can use any arch master with any arch nodes just OOTB. |
|
@lukaszgryglicki I never tried by myself, but I guess you can deploy two hacked flannel manifest with different taints (ands names) for avoiding to mix up things, but again my suggestions is to ask to flannel people if there are already instructions about how to do this. |
|
But I've tweaked manifest on arm64 as suggested in the tutorial. Replaced And now why |
|
@lukaszgryglicki And now that I think of, might be you should fix the same issue with kube-proxy daemon set as well, but I can't test this now, sorry For the problem you have with weave I don't have enough info. One problem might be that weave doesn't works with |
|
So I should deploy two different manifests for flannel on a master, right? No matter if master happens to be arm64 or amd64, right? Should Master handle spawning correct arch deployment on itself and nodes? I didn't use |
|
cc @luxas - I've seen you've created some docs about multi-arch k8s deployments, maybe you can have some feedback? |
|
yeah, so this is an arch issue after all, thanks for confirming. have a look this by @luxas for context, if not seen it already:
it should but the manifest you are downloading is not a "fat" one: as far as i understand it the arch taints are propagated and you need to fix that with looks like a "fat" manifest is in master and was added here: related issue/pr: my assumption is that this is bleeding edge and you have to use: so bring the cluster down and give that a try and hope it works. |
|
OK, will try after the weekend and post my results here. Thanks. |
|
@lukaszgryglicki hi, did you get this working using the new flannel manifest? |
|
Not yet, I'll try today. |
|
OK finally worked: Fat mainifest from flannel |
|
Hi guys, I'm in the same situation. any idea? |
|
The error seems different, but did you use the fat manifest: https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml ? |
|
Yes I am: The problem now it the flannel container that doesn't stay up on arm. :( |
|
it does on |
|
I'm on arm64 but thanks, I'll continue to investigate... |
|
Ohh then sorry, I thought you're on arm. |
Reinstall docker on the notready node. |
That just did it! |
|
I had a similar case where I was creating the network plugin before linking the workers, which kept the /etc/cni/net.d missing. Hope that helps anyone with the same issue. |
Ran that Command on Master Machine and everything is in Ready state now. Thank you @saddique164 . |
|
The quickest way is to add Flannel into the Kubernetes on either AMD64 Architecture. 1. Update kube-flannel.yaml2. Complete Flannel network setup
|
|
I am using the kubernetes 1.18 version. I used this: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml There was no file created under /etc/cni/net.d |
NOTE: This seems to be kubelet issue. |
|
|
try this on master: sed -i 's/cgroup-driver=systemd/cgroup-driver=cgroupfs/g' /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf |
|
It starts and then fails again. |
|
If you see it is saying no network is found. Run this command and share the result. Kubectl get pods -n kube-system |
|
|
run this |
|
if nothing comes then run this one: |
|
Error from server: Get https://10.75.214.124:10250/containerLogs/kube-system/kube-flannel-ds-amd64-5tmhp/kube-flannel: dial tcp 10.75.214.124:10250: connect: connection refused |
|
How many nodes are running for you? IN the cluster? one node is doing this problem. This is called daemonset. They run on every node. Your control plan is not accepting the request from it. So i will suggest you to follow the following steps.
This process will work. |
|
kubectl get nodes: |
|
I tried the steps you mentioned: This is what I get. |
|
drain all the nodes except master and concentrate on that. When it will be ready then go to add others. |
|
Draining nodes and then kubeadm reset and init doesn't help. The cluster is not initialized afterward. |
|
My problem was that I was updating the hostname after the cluster was created. By doing that, it's like the master didn't know it was the master. I am still running:
but now I run it before the cluster initialization |
Add pod network add-on helped me out. |
My symptoms were 101% like @jotrobin reported and this finally fixed it. Thanks! |
1 task
did not work for me |
|
did not work for me |
|
If using the Weave CNI Plugin from a prior full install of Weave Net with your cluster, check their is no hard valued IPAM and other conf described . Remove any hard constraint under /etc/cni/net.d/weave.config and restart the kublet. Link for Weave net kubernetes : ### kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" Hope this helps anyone !! |
|
I am facing issues on "containerGCFailed" Node not ready state |
|
Ok, I have the same error, running coredns instead of Flannel. I need to check the logs to see what is wrong. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
apt-get update/upgrade.kubeadm init --pod-network-cidr=10.244.0.0/16. And then executed suggested commands.sysctl net.bridge.bridge-nf-call-iptables=1.wget https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml.vim kube-flannel.yml, replaceamd64witharm64kubectl apply -f kube-flannel.yml.kubectl get pods --all-namespaces:Then joined two AMD64 nodes using
kubeadm initoutput:1st node:
2nd node:
But on the master
kubectl get nodes:And then:
kubectl describe nodes(master isdevstats.team.io, nodes are:cncftest.ioanddevstats.cncf.io):Versions
kubeadm version (use
kubeadm version):Environment:
kubectl version):uname -a: Linux devstats.team.io 4.15.0-20-generic firewalld breaks kubeadm #21-Ubuntu SMP Tue Apr 24 06:16:20 UTC 2018 aarch64 aarch64 aarch64 GNU/Linuxlsb_release -a:uname -a):Linux devstats.team.io 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:20 UTC 2018 aarch64 aarch64 aarch64 GNU/Linuxdocker version:What happened?
The exact error seems to be:
On the node:
cat /etc/systemd/system/kubelet.service.d/10-kubeadm.confFrom this thread (no
KUBELET_NETWORK_ARGSthere).journalctl -xeon the node:Directory
/etc/cni/net.dexists, but is empty.What you expected to happen?
All nodes in
Readystate.How to reproduce it (as minimally and precisely as possible)?
Just follow steps from the tutorial. Tried 3 times and it happens all the time.
Anything else we need to know?
Master is ARM64, 2 nodes are AMD64.
Master and one node is in Amsterdam and 2nd node is in the USA.
I can use
kubectl taint nodes --all node-role.kubernetes.io/master-to run pods on master, but this is not a solution. I want to have a real multi-node cluster to work with.The text was updated successfully, but these errors were encountered: