failed to create a sandbox for pod: context deadline exceeded (etcdserver: read-only range request deadline exceeded)

[MoeHamdan]

Hello

I have minikube installed on Ubuntu 20.04 VM. Lately I started having issues with the pods, when I deploy a new pod or delete a pod it gets stuck at creating container and the terminated one gets stuck at terminating container.
This is the error that I get:

Failed create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "*******": operation timeout: context deadline exceeded

I have rabbitmq running as a docker container, which I noticed when I stop then everything starts working properly and I get no errors. I thought it was rabbitmq, however it was not as I tried enabling the dashboard addon; once I did that I started to have the issues again. I am not sure what is happening and why the apiserver is being blocked.

This error I found in logs for the pod etcd-ubuntu:
2021-03-22 08:56:19.520545 W | etcdserver/api/etcdhttp: /health error; QGET failed etcdserver: request timed out (status code 503) 2021-03-22 08:56:20.703489 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "error:context canceled" took too long (2.000067723s) to execute WARNING: 2021/03/22 08:56:20 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing" 2021-03-22 08:56:22.714422 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "error:context deadline exceeded" took too long (2.000809373s) to execute WARNING: 2021/03/22 08:56:22 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing" 2021-03-22 08:56:24.726541 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "error:context canceled" took too long (2.0001346s) to execute WARNING: 2021/03/22 08:56:24 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"

and these errors from kube-apiserver-ubuntu pod:
E0322 08:56:39.554970 1 controller.go:223] unable to sync kubernetes service: etcdserver: request timed out I0322 08:56:42.068619 1 trace.go:205] Trace[852842062]: "GuaranteedUpdate etcd3" type:*coordination.Lease (22-Mar-2021 08:56:35.065) (total time: 7002ms): Trace[852842062]: [7.002594255s] [7.002594255s] END E0322 08:56:42.068713 1 status.go:71] apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:"etcdserver: request timed out"} I0322 08:56:42.069066 1 trace.go:205] Trace[1481089662]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu,user-agent:kubelet/v1.20.2 (linux/amd64) kubernetes/faecb19,client:192.168.1.6 (22-Mar-2021 08:56:35.065) (total time: 7003ms): Trace[1481089662]: [7.003223867s] [7.003223867s] END I0322 08:56:42.077354 1 trace.go:205] Trace[118326597]: "GuaranteedUpdate etcd3" type:*core.Event (22-Mar-2021 08:56:32.557) (total time: 9519ms): Trace[118326597]: [9.519452184s] [9.519452184s] END E0322 08:56:42.077399 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"} E0322 08:56:42.077581 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"} I0322 08:56:42.077663 1 trace.go:205] Trace[580221338]: "Patch" url:/api/v1/namespaces/kube-system/events/kube-apiserver-ubuntu.166e9a79ea68f758,user-agent:kubelet/v1.20.2 (linux/amd64) kubernetes/faecb19,client:192.168.1.6 (22-Mar-2021 08:56:32.557) (total time: 9519ms): Trace[580221338]: [9.519855616s] [9.519855616s] END I0322 08:56:42.078864 1 trace.go:205] Trace[1326891703]: "Get" url:/api/v1/namespaces/kube-system/pods/kube-apiserver-ubuntu,user-agent:kubelet/v1.20.2 (linux/amd64) kubernetes/faecb19,client:192.168.1.6 (22-Mar-2021 08:56:32.559) (total time: 9519ms): Trace[1326891703]: [9.519297381s] [9.519297381s] END

May you assist me with my issue? It has been a week, I am struggling with my development environment.
Thanks

[tstromberg]

I don’t yet have a clear way to replicate this issue. Do you mind adding some additional details? Here is additional information that would be helpful:

• The exact minikube start command line used (I'm curious about your Kubernetes version, for example)
• The full output of the minikube start command
• The full output of minikube logs (I am curious what it reports about load averages, for example)

I've seen this in the past when etcd has been abused by some process to store state, but I haven't seen an indication here. See for example kubernetes/kubernetes#70082

[MoeHamdan]

Hello @tstromberg
Thanks for your reply. Here are the details requested:

sudo minikube start --embed-certs --driver=none

minikubeLog.txt

[MoeHamdan]

anybody there ?

[ilya-zuyev]

@MoeHamdan do you really have to use none driver? It deploys all k8s services in your host OS, provides no isolation and makes it really easy to affect the setup by 3rd party. May be the docker driver would work for you?

[MoeHamdan]

@ilya-zuyev "none" driver satisfy my needs in a development environment. I hope if anybody could help resolve the issue without using other driver.

[seksham]

Same issue. Even I require "none" driver. Any solutions @MoeHamdan

[wajeehulhassanvii]

@MoeHamdan @ilya-zuyev I am also having a similar issue. Should I change the driver to Docker?

[spowelljr]

@wajeehulhassanvii Using the Docker driver should yield an easier experience with minikube.

[sharifelgamal]

I wonder if it has anything to do with the metallb addon with the none driver. I haven't personally tried that combination.

[klaases]

Hi @MoeHamdan, @seksham, @wajeehulhassanvii were you able to try minikube with the docker driver?

[MoeHamdan]

Hello

Well guys recently, I cannot remember since which version though, I am not having the issue anymore. I have around 80+ pods and all good.

[AlagiefNget]

@MoeHamdan ,
this will like most have to do with your image being too big.

One thing I would do is in your deployment file,

in the spec for the conatainer, add a resources with a request and a limit,eg
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi