New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes Mounting Cinder Volume gives “Timeout” error #73887

Open
harshkumar1 opened this Issue Feb 10, 2019 · 6 comments

Comments

Projects
None yet
4 participants
@harshkumar1
Copy link

harshkumar1 commented Feb 10, 2019

What happened:
Kubernetes Services failed in deployment since it was not able to mount Persistent Volumes

What you expected to happen:
Kubernetes Services deployment should happen post mounting Persistent Volumes

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
Persistent Volumes are being provisioned using Dynamic Provisioning

Environment:

  • Kubernetes version (use kubectl version): 1.12.3

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

  • Cloud provider or hardware configuration: Openstack

  • OS (e.g. from /etc/os-release): CentOS7

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

  • Kernel (e.g. uname -a):

Linux k8s-master-0 4.14.35-1844.0.7.el7uek.x86_64 #2 SMP Wed Dec 12 19:48:02 PST 2018 x86_64 x86_64 x86_64 GNU/Linux

  • Install tools:

  • Others:
    k8s Error Logs,

Feb 8 11:30:08 k8s-minion-1 kubelet: E0208 11:30:08.962255 21297 pod_workers.go:186] Error syncing pod 46a0d40f-2b94-11e9-8db8-fa163e315068 ("eis-scu-75fc888b5d-lstnb_edison-core(46a0d40f-2b94-11e9-8db8-fa163e315068)"), skipping: timeout expired waiting for volumes to attach or mount for pod "edison-core"/"eis-scu-75fc888b5d-lstnb". list of unmounted volumes=[certs]. list of unattached volumes=[edison-core-vol certs default-token-qjdc5]

@harshkumar1

This comment has been minimized.

Copy link
Author

harshkumar1 commented Feb 10, 2019

/sig openstack
/sig storage

@adisky

This comment has been minimized.

Copy link

adisky commented Feb 11, 2019

@harshkumar1 can you share kube-controller-manger logs as well? you can also reach us at kubernetes slack #sig-openstack channel

@desaintmartin

This comment has been minimized.

Copy link
Contributor

desaintmartin commented Feb 13, 2019

Out of curiosity, what is your IaaS provider? I had such issues when the control plane of OpenStack of my provider was too overloaded to answer in a timely fashion.

@harshkumar1

This comment has been minimized.

Copy link
Author

harshkumar1 commented Feb 14, 2019

@desaintmartin
Cloud Provider - Openstack
What do you mean by too overloaded, we are seeing other issues as well like Node toggling between Ready / Not Ready state. Did you see it because of NW Topology?

@harshkumar1

This comment has been minimized.

Copy link
Author

harshkumar1 commented Feb 16, 2019

@adisky so we are redirecting all logs to /var/log/messages and it would be split into multiple files and tough for you to be able to make sense. Hence am posting content from it.
The logs which I am posting below happened for a pod sw-label (pod name is different that the original post, but the behavior is similar)

Feb 14 09:48:43 k8s-minion-0 kubelet: I0214 09:48:43.970148 10038 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-b62815dd-303d-11e9-a725-fa163ee64060" (UniqueName: "kubernetes.io/cinder/da1609d0-23b4-44a9-8e7e-43de539c7161") pod "sw-label-b75c7dbf8-46bcr" (UID: "b69c50bb-303d-11e9-85b6-fa163e8691c9")
Feb 14 09:48:43 k8s-minion-0 kubelet: I0214 09:48:43.970219 10038 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-zqqcf" (UniqueName: "kubernetes.io/secret/b69c50bb-303d-11e9-85b6-fa163e8691c9-default-token-zqqcf") pod "sw-label-b75c7dbf8-46bcr" (UID: "b69c50bb-303d-11e9-85b6-fa163e8691c9")
Feb 14 09:48:43 k8s-minion-0 kubelet: E0214 09:48:43.970372 10038 nestedpendingoperations.go:267] Operation for ""kubernetes.io/cinder/da1609d0-23b4-44a9-8e7e-43de539c7161"" failed. No retries permitted until 2019-02-14 09:48:44.470339465 +0000 UTC m=+4381.941141866 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume "pvc-b62815dd-303d-11e9-a725-fa163ee64060" (UniqueName: "kubernetes.io/cinder/da1609d0-23b4-44a9-8e7e-43de539c7161") pod "sw-label-b75c7dbf8-46bcr" (UID: "b69c50bb-303d-11e9-85b6-fa163e8691c9") "

Feb 14 09:48:44 k8s-minion-0 kubelet: I0214 09:48:44.070879 10038 reconciler.go:252] operationExecutor.MountVolume started for volume "default-token-zqqcf" (UniqueName: "kubernetes.io/secret/b69c50bb-303d-11e9-85b6-fa163e8691c9-default-token-zqqcf") pod "sw-label-b75c7dbf8-46bcr" (UID: "b69c50bb-303d-11e9-85b6-fa163e8691c9")
Feb 14 09:48:44 k8s-minion-0 kernel: IPv6: ADDRCONF(NETDEV_UP): cali94e65181613: link is not ready
Feb 14 09:48:44 k8s-minion-0 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): cali94e65181613: link becomes ready
Feb 14 09:48:44 k8s-minion-0 systemd: Started Kubernetes transient mount for /var/lib/kubelet/pods/b69c50bb-303d-11e9-85b6-fa163e8691c9/volumes/kubernetes.io~secret/default-token-zqqcf.
Feb 14 09:48:44 k8s-minion-0 kubelet: I0214 09:48:44.239864 10038 operation_generator.go:568] MountVolume.SetUp succeeded for volume "default-token-zqqcf" (UniqueName: "kubernetes.io/secret/b69c50bb-303d-11e9-85b6-fa163e8691c9-default-token-zqqcf") pod "sw-label-b75c7dbf8-46bcr" (UID: "b69c50bb-303d-11e9-85b6-fa163e8691c9")
Feb 14 09:48:44 k8s-minion-0 kubelet: 2019-02-14 15:18:44.249 [INFO][6856] k8s.go 267: Populated endpoint ContainerID="05361d672a71f0fb1fc7fcf954c57b921e011b62704449895599fb3a70f9de51" Namespace="ct-recon" Pod="ig-controller-75c57dcc67-mnvm4" WorkloadEndpoint="k8s--minion--0-k8s-ig--controller--75c57dcc67--mnvm4-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"k8s--minion--0-k8s-ig--controller--75c57dcc67--mnvm4-eth0", GenerateName:"", Namespace:"ct-recon", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"projectcalico.org/orchestrator":"k8s", "app":"recon-controller", "pod-template-hash":"75c57dcc67", "projectcalico.org/namespace":"ct-recon"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"k8s-minion-0", ContainerID:"", Pod:"ig-controller-75c57dcc67-mnvm4", Endpoint:"eth0", IPNetworks:[]string{"10.233.70.115/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"kns.ct-recon"}, InterfaceName:"", MAC:"", Ports:[]v3.EndpointPort(nil)}}
Feb 14 09:48:44 k8s-minion-0 kubelet: Calico CNI using IPs: [10.233.70.115/32]

@harshkumar1

This comment has been minimized.

Copy link
Author

harshkumar1 commented Feb 16, 2019

The other observations which is there is that this seems to happen most of the times with services deployed on k8s-minion-0 (first node of the cluster)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment