Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

coredns stuck at ContainerCreating #87

Closed
drpaneas opened this issue Aug 13, 2018 · 21 comments
Closed

coredns stuck at ContainerCreating #87

drpaneas opened this issue Aug 13, 2018 · 21 comments

Comments

@drpaneas
Copy link

@drpaneas drpaneas commented Aug 13, 2018

I am running the latest kubeadm 1.11 where CoreDNS is the default. I've also installed weave-net plugin. However, I still cannot get coredns pod running ....

linux-uwkw:~ # kubectl get pods --namespace=kube-system
NAME                                           READY     STATUS              RESTARTS   AGE
coredns-78fcdf6894-fv74n                       0/1       ContainerCreating   0          6m
coredns-78fcdf6894-pnq22                       0/1       ContainerCreating   0          6m
etcd-linux-uwkw.fritz.box                      1/1       Running             0          5m
kube-apiserver-linux-uwkw.fritz.box            1/1       Running             0          5m
kube-controller-manager-linux-uwkw.fritz.box   1/1       Running             0          5m
kube-proxy-785pk                               1/1       Running             0          6m
kube-scheduler-linux-uwkw.fritz.box            1/1       Running             0          5m
weave-net-8tqdh                                2/2       Running             0          6m

Any debug tips?

@drpaneas

This comment has been minimized.

Copy link
Author

@drpaneas drpaneas commented Aug 14, 2018

I was missing the looback binary of CNI. Now everything is fine,

@drpaneas drpaneas closed this Aug 14, 2018
@doctorCC

This comment has been minimized.

Copy link

@doctorCC doctorCC commented Sep 11, 2018

Could you elaborate? I am having a similar issue, where I was trying to use the flannel plugin (which is Running) but there are 2 coredns which seem to be stuck in ContainerCreating
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-78fcdf6894-sdvhr 0/1 ContainerCreating 0 6m
kube-system coredns-78fcdf6894-t94cp 0/1 ContainerCreating 0 6m
kube-system etcd-ubuntu 1/1 Running 0 5m
kube-system kube-apiserver-ubuntu 1/1 Running 0 5m
kube-system kube-controller-manager-ubuntu 1/1 Running 0 5m
kube-system kube-flannel-ds-s390x-7z2zx 1/1 Running 0 2m
kube-system kube-proxy-bk62x 1/1 Running 0 6m
kube-system kube-scheduler-ubuntu 1/1 Running 0 5m

@chrisohaver

This comment has been minimized.

Copy link
Member

@chrisohaver chrisohaver commented Sep 11, 2018

@doctorCC Any errors/warnings in kubelet logs?

@johnbelamaric

This comment has been minimized.

Copy link
Member

@johnbelamaric johnbelamaric commented Sep 11, 2018

@doctorCC

This comment has been minimized.

Copy link

@doctorCC doctorCC commented Sep 25, 2018

I tore down the sysem and rebuilt it, allowing the emptyCluster to sit there overnight. Again there were two coredns which were never built. I tried to deploy a simple nginx, which also went into a ContainerCreating state. The logs from that were available and are below.
It appears to me that there is an issue with the Flannel network

I am running:
s390 architecture
kubectl version 1.11.3
attempting to follow directions from https://blog.alexellis.io/kubernetes-in-10-minutes/

kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default my-nginx-7b6fc8965-24skw 0/1 ContainerCreating 0 11m
kube-system coredns-78fcdf6894-ddjnv 0/1 ContainerCreating 0 22h
kube-system coredns-78fcdf6894-kvwvw 0/1 ContainerCreating 0 22h
kube-system etcd-ubuntu 1/1 Running 0 22h
kube-system kube-apiserver-ubuntu 1/1 Running 0 22h
kube-system kube-controller-manager-ubuntu 1/1 Running 0 22h
kube-system kube-flannel-ds-s390x-djgcz 1/1 Running 0 22m
kube-system kube-proxy-j7lg4 1/1 Running 0 22h
kube-system kube-scheduler-ubuntu 1/1 Running 0 22h
ubuntu@ubuntu:$ kubectl describe pod coredns-78fcdf6894-ddjnv
Error from server (NotFound): pods "coredns-78fcdf6894-ddjnv" not found
ubuntu@ubuntu:
$ kubectl describe pod coredns-78fcdf6894-kvwvw
Error from server (NotFound): pods "coredns-78fcdf6894-kvwvw" not found
ubuntu@ubuntu:~$ kubectl describe pod my-nginx-7b6fc8965-24skw
Name: my-nginx-7b6fc8965-24skw
Namespace: default
Priority: 0
PriorityClassName:
Node: ubuntu/9.12.41.193
Start Time: Tue, 25 Sep 2018 16:30:17 +0000
Labels: pod-template-hash=362974521
run=my-nginx
Annotations:
Status: Pending
IP:
Controlled By: ReplicaSet/my-nginx-7b6fc8965
Containers:
my-nginx:
Container ID:
Image: nginx:latest
Image ID:
Port: 9000/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-bkv6n (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-bkv6n:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-bkv6n
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message


Normal Scheduled 12m default-scheduler Successfully assigned default/my-nginx-7b6fc8965-24skw to ubuntu
Warning FailedCreatePodSandBox 12m kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "4e15faf72b2677c8ed5ffd06213b08e4597c8fe62c4a343926a08ee54dc93e2f" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to set up pod "my-nginx-7b6fc8965-24skw_default" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "4e15faf72b2677c8ed5ffd06213b08e4597c8fe62c4a343926a08ee54dc93e2f" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to teardown pod "my-nginx-7b6fc8965-24skw_default" network: failed to get IP addresses for "eth0": ]
Warning FailedCreatePodSandBox 12m kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "13bba308436c129bea5817f85a4df37631a44fcdb580e57405e6c3884c545a10" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to set up pod "my-nginx-7b6fc8965-24skw_default" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "13bba308436c129bea5817f85a4df37631a44fcdb580e57405e6c3884c545a10" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to teardown pod "my-nginx-7b6fc8965-24skw_default" network: failed to get IP addresses for "eth0": ]
Warning FailedCreatePodSandBox 12m kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "e36596ee6792d20609a851f1421a5dd808106be4a74bc64ff3329ae20a09583f" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to set up pod "my-nginx-7b6fc8965-24skw_default" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "e36596ee6792d20609a851f1421a5dd808106be4a74bc64ff3329ae20a09583f" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to teardown pod "my-nginx-7b6fc8965-24skw_default" network: failed to get IP addresses for "eth0": ]
Warning FailedCreatePodSandBox 12m kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "6a3a5c9badde3da9e514d0671688712bf93ba06fbee6450f4e3c8a020d1eab76" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to set up pod "my-nginx-7b6fc8965-24skw_default" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "6a3a5c9badde3da9e514d0671688712bf93ba06fbee6450f4e3c8a020d1eab76" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to teardown pod "my-nginx-7b6fc8965-24skw_default" network: failed to get IP addresses for "eth0": ]
Warning FailedCreatePodSandBox 12m kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "ea9c900d04d01cf41edf182c94612c7fb2a5db46d74d05bda035586e458642d4" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to set up pod "my-nginx-7b6fc8965-24skw_default" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "ea9c900d04d01cf41edf182c94612c7fb2a5db46d74d05bda035586e458642d4" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to teardown pod "my-nginx-7b6fc8965-24skw_default" network: failed to get IP addresses for "eth0": ]
Warning FailedCreatePodSandBox 11m kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "94000515ffeca2689a8c509921fad57cc239a18960fb53f758e75cc25d8ed217" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to set up pod "my-nginx-7b6fc8965-24skw_default" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "94000515ffeca2689a8c509921fad57cc239a18960fb53f758e75cc25d8ed217" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to teardown pod "my-nginx-7b6fc8965-24skw_default" network: failed to get IP addresses for "eth0": ]
Warning FailedCreatePodSandBox 11m kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "8e7322d37dfe96801fd3e731f5304431ff9f9848e04c5f01408c4fe00a0cea35" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to set up pod "my-nginx-7b6fc8965-24skw_default" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "8e7322d37dfe96801fd3e731f5304431ff9f9848e04c5f01408c4fe00a0cea35" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to teardown pod "my-nginx-7b6fc8965-24skw_default" network: failed to get IP addresses for "eth0": ]
Warning FailedCreatePodSandBox 11m kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "f83fdb6fc70c05e71f2515089f0f3c55142a05cecb9354be4e0be321658c35fc" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to set up pod "my-nginx-7b6fc8965-24skw_default" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "f83fdb6fc70c05e71f2515089f0f3c55142a05cecb9354be4e0be321658c35fc" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to teardown pod "my-nginx-7b6fc8965-24skw_default" network: failed to get IP addresses for "eth0": ]
Warning FailedCreatePodSandBox 11m kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "41d825f96630b1ba16dc1ed04e893d6157facd65024c63080ddb03e066592de4" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to set up pod "my-nginx-7b6fc8965-24skw_default" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "41d825f96630b1ba16dc1ed04e893d6157facd65024c63080ddb03e066592de4" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to teardown pod "my-nginx-7b6fc8965-24skw_default" network: failed to get IP addresses for "eth0": ]
Normal SandboxChanged 11m (x12 over 12m) kubelet, ubuntu Pod sandbox changed, it will be killed and re-created.
Warning FailedCreatePodSandBox 2m (x182 over 11m) kubelet, ubuntu (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "a3da6c0ffa892f34f9f3e3d9cd3a1f46590a551e895879c44448bb58fe43cb7a" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to set up pod "my-nginx-7b6fc8965-24skw_default" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "a3da6c0ffa892f34f9f3e3d9cd3a1f46590a551e895879c44448bb58fe43cb7a" network for pod "my-nginx-7b6fc8965-24skw": NetworkPlugin cni failed to teardown pod "my-nginx-7b6fc8965-24skw_default" network: failed to get IP addresses for "eth0": ]

@chrisohaver

This comment has been minimized.

Copy link
Member

@chrisohaver chrisohaver commented Sep 25, 2018

@smorrisfv

This comment has been minimized.

Copy link

@smorrisfv smorrisfv commented Oct 5, 2018

I'm running into the same issue, and I have IPv6 enabled. I'm using a fresh setup using kubeadm on Ubuntu Server 18.04.

NAMESPACE     NAME                               READY   STATUS              RESTARTS   AGE
kube-system   coredns-576cbf47c7-k2xxd           0/1     ContainerCreating   0          91m
kube-system   coredns-576cbf47c7-pqk9k           0/1     ContainerCreating   0          91m

When I run describe I get the following error message,

$ kubectl describe pod coredns-576cbf47c7-pqk9k
Error from server (NotFound): pods "coredns-576cbf47c7-pqk9k" not found

Which isn't very useful.

Any luck @doctorCC? Also, @drpaneas, what do you mean by loopback for CNI?

@johnbelamaric

This comment has been minimized.

Copy link
Member

@johnbelamaric johnbelamaric commented Oct 5, 2018

@kalginshah

This comment has been minimized.

Copy link

@kalginshah kalginshah commented Oct 9, 2018

I was missing the looback binary of CNI. Now everything is fine,

how exactly did u set the 'loopback'?

@smorrisfv

This comment has been minimized.

Copy link

@smorrisfv smorrisfv commented Oct 9, 2018

Thanks @johnbelamaric for your help. I found my issue, it was a bug in Flannel (coreos/flannel#1044) when running on K8s 1.12. What fixed it for me was the suggestion from this comment (kubernetes/kubernetes#48798).

@spitfire88

This comment has been minimized.

Copy link

@spitfire88 spitfire88 commented Oct 16, 2018

I was missing the looback binary of CNI. Now everything is fine,

how exactly did u set the 'loopback'?

  • go get -d github.com/containernetworking/plugins
  • cd ~/go/src/github.com/containernetworking/plugins
  • ./build.sh
  • sudo cp bin/* /opt/cni/bin/
@ababushk

This comment has been minimized.

Copy link

@ababushk ababushk commented Oct 20, 2018

Thank you @spitfire88! This works for me. Although I have now idea why I need to download and compile something when I'm trying to use kubeadm

@ola07

This comment has been minimized.

Copy link

@ola07 ola07 commented Nov 9, 2018

This is does not fix my issue

@mathieuduperre

This comment has been minimized.

Copy link

@mathieuduperre mathieuduperre commented Nov 28, 2018

executing spifire's recommended steps did not work. running this work:

kubectl -n kube-system apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml

(kubernetes/kubernetes#48798)
as recommended by smorrisfv

@msq6323013

This comment has been minimized.

Copy link

@msq6323013 msq6323013 commented Dec 29, 2018

docker 18.09.0,kube1.13.1
上面都不太好用,我CentOS7.3,是这样解决的:
编辑:/etc/default/grub中,ipv6.disable=0
运行:grub2-mkconfig -o /boot/grub2/grub.cfg
重启服务器

Tried above comments are not available for me, In my CentOS 7.3 server, here is the solution:
vi / etc/default/grub ==edit===> ipv6. disable = 0

grub2-mkconfig -o /boot/grub2/grub.cfg

Restart the server

上記の方法は私に対してあまり使いにくいですが、私のCenoOS 7.3は、このように解決しました。
編集:/etc/default/grub =====> ipv6. disable = 0
実行:grub2-mkconfig -o /boot/grub2/grub.cfg

再起動サーバー

@wilsonwu

This comment has been minimized.

Copy link

@wilsonwu wilsonwu commented Jan 17, 2019

I'm running into the same issue, and I have IPv6 enabled. I'm using a fresh setup using kubeadm on Ubuntu Server 18.04.

NAMESPACE     NAME                               READY   STATUS              RESTARTS   AGE
kube-system   coredns-576cbf47c7-k2xxd           0/1     ContainerCreating   0          91m
kube-system   coredns-576cbf47c7-pqk9k           0/1     ContainerCreating   0          91m

When I run describe I get the following error message,

$ kubectl describe pod coredns-576cbf47c7-pqk9k
Error from server (NotFound): pods "coredns-576cbf47c7-pqk9k" not found

Which isn't very useful.

Any luck @doctorCC? Also, @drpaneas, what do you mean by loopback for CNI?

you should add --namespace=kube-system when you run describe pod command.

@ravi134

This comment has been minimized.

Copy link

@ravi134 ravi134 commented May 20, 2019

I was missing the looback binary of CNI. Now everything is fine,

how exactly did u set the 'loopback'?

  • go get -d github.com/containernetworking/plugins
  • cd ~/go/src/github.com/containernetworking/plugins
  • ./build.sh
  • sudo cp bin/* /opt/cni/bin/

worked for me

@fabstao

This comment has been minimized.

Copy link

@fabstao fabstao commented Jul 25, 2019

Hi! I applied the deployment and the go get steps, now I get:

network: failed to create bridge "cni0": could not add "cni0": operation not supported

Please help

Host: ClearLinux
Kubernetes:

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"archive", BuildDate:"2019-07-04T23:54:15Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:09:21Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

Bridges:

# brctl show
bridge name	bridge id		STP enabled	interfaces
docker0		8000.02422aea25b1	no	

Docker version:

# docker version
Client:
 Version:           18.06.3
 API version:       1.38
 Go version:        go1.12.5
 Git commit:        d7080c17a580919f5340a15a8e5e013133089680
 Built:             Mon Jun  3 17:41:01 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.3
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       d7080c17a580919f5340a15a8e5e013133089680
  Built:            Mon Jun  3 17:41:13 2019
  OS/Arch:          linux/amd64
  Experimental:     false
@itsecforu

This comment has been minimized.

Copy link

@itsecforu itsecforu commented Sep 23, 2019

@msq6323013 on master node or worker?

@fabstao

This comment has been minimized.

Copy link

@fabstao fabstao commented Nov 29, 2019

Master node

@selimnl

This comment has been minimized.

Copy link

@selimnl selimnl commented Apr 2, 2020

run this command to active flannel mode
it works. core dns gets running.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.