Couldn't read CA certificate: open : no such file or directory #2518

dreamlover · 2017-10-25T01:02:19Z

Environment

I installed a single-node kubernetes cluster on CentOS7 using kubeadm according to this manual, then installed the kubernetes-dashboard extension.

Dashboard version:  1.7.1
Kubernetes version: 1.7.5
Operating system: CentOS 7
Node.js version:
Go version: 1.8.3

Observed result

[root@ay pki]# kubectl get pods -n kube-system
NAME                                    READY     STATUS             RESTARTS   AGE
etcd-ay                                 1/1       Running            0          12d
kube-apiserver-ay                       1/1       Running            0          12d
kube-controller-manager-ay              1/1       Running            0          12d
kube-dns-209315428-666w5                3/3       Running            0          12d
kube-proxy-92ss6                        1/1       Running            0          12d
kube-scheduler-ay                       1/1       Running            0          12d
kubernetes-dashboard-1092119393-n9ww6   0/1       CrashLoopBackOff   185        15h
weave-net-wtf68                         2/2       Running            22         1d

[root@ay run]# kubectl logs kubernetes-dashboard-1092119393-n9ww6 -n kube-system
2017/10/25 00:57:16 Using in-cluster config to connect to apiserver
2017/10/25 00:57:16 Starting overwatch
2017/10/25 00:57:16 Using service account token for csrf signing
2017/10/25 00:57:16 No request provided. Skipping authorization
2017/10/25 00:57:16 Successful initial request to the apiserver, version: v1.7.5
2017/10/25 00:57:16 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2017/10/25 00:57:16 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2017/10/25 00:57:16 Initializing secret synchronizer synchronously using secret kubernetes-dashboard-key-holder from namespace kube-system
2017/10/25 00:57:16 Initializing JWE encryption key from synchronized object
2017/10/25 00:57:16 Creating in-cluster Heapster client
2017/10/25 00:57:16 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2017/10/25 00:57:16 Serving securely on HTTPS port: 8443
2017/10/25 00:57:16 Couldn't read CA certificate: open : no such file or directory

The text was updated successfully, but these errors were encountered:

maciaszczykm · 2017-10-26T10:41:24Z

Could you paste output from kubectl describe kubernetes-dashboard -n kube-system and kubectl describe secret kubernetes-dashboard-certs - n kube-system?

dreamlover · 2017-10-27T00:32:31Z

@maciaszczykm

[root@ay ay.k8s.d]# kubectl describe pod kubernetes-dashboard -n kube-system
Name:		kubernetes-dashboard-1092119393-1np60
Namespace:	kube-system
Node:		ay/10.27.183.194
Start Time:	Thu, 26 Oct 2017 08:55:07 +0800
Labels:		k8s-app=kubernetes-dashboard
		pod-template-hash=1092119393
Annotations:	kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"kubernetes-dashboard-1092119393","uid":"4f7270cc-b9e8-11e7-b...
Status:		Running
IP:		10.32.0.4
Created By:	ReplicaSet/kubernetes-dashboard-1092119393
Controlled By:	ReplicaSet/kubernetes-dashboard-1092119393
Init Containers:
  kubernetes-dashboard-init:
    Container ID:	docker://4d814445778aa41bd2be53fbfe6bfb256c8f83d957f99c709cc11b57b36fb948
    Image:		registry.cn-hangzhou.aliyuncs.com/google-containers/kubernetes-dashboard-init-amd64:v1.0.1
    Image ID:		docker-pullable://registry.cn-hangzhou.aliyuncs.com/google-containers/kubernetes-dashboard-init-amd64@sha256:fb86ae64a1876a73ee68fa6428c94cbc14c85b549ea3e896f19e81eda00181ba
    Port:		<none>
    State:		Terminated
      Reason:		Completed
      Exit Code:	0
      Started:		Thu, 26 Oct 2017 08:55:09 +0800
      Finished:		Thu, 26 Oct 2017 08:55:09 +0800
    Ready:		True
    Restart Count:	0
    Environment:	<none>
    Mounts:
      /certs from kubernetes-dashboard-certs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kubernetes-dashboard-token-bmvm9 (ro)
Containers:
  kubernetes-dashboard:
    Container ID:	docker://93ee8337723b9e4810c3005302c93779e115164d19aa1264f7307557ae015a37
    Image:		registry.cn-hangzhou.aliyuncs.com/google-containers/kubernetes-dashboard-amd64:v1.7.1
    Image ID:		docker-pullable://registry.cn-hangzhou.aliyuncs.com/google-containers/kubernetes-dashboard-amd64@sha256:52b1aeb47e56a97e1278fcdede3dd84703e5e7cef8e0129aa26a73b5f4cadb76
    Port:		8443/TCP
    Args:
      --tls-key-file=/certs/dashboard.key
      --tls-cert-file=/certs/dashboard.crt
    State:		Waiting
      Reason:		CrashLoopBackOff
    Last State:		Terminated
      Reason:		Error
      Exit Code:	1
      Started:		Fri, 27 Oct 2017 08:30:33 +0800
      Finished:		Fri, 27 Oct 2017 08:30:33 +0800
    Ready:		False
    Restart Count:	281
    Liveness:		http-get https://:8443/ delay=30s timeout=30s period=10s #success=1 #failure=3
    Environment:	<none>
    Mounts:
      /certs from kubernetes-dashboard-certs (ro)
      /tmp from tmp-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kubernetes-dashboard-token-bmvm9 (ro)
Conditions:
  Type		Status
  Initialized 	True 
  Ready 	False 
  PodScheduled 	True 
Volumes:
  kubernetes-dashboard-certs:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	kubernetes-dashboard-certs
    Optional:	false
  tmp-volume:
    Type:	EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:	
  kubernetes-dashboard-token-bmvm9:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	kubernetes-dashboard-token-bmvm9
    Optional:	false
QoS Class:	BestEffort
Node-Selectors:	<none>
Tolerations:	node-role.kubernetes.io/master:NoSchedule
		node.alpha.kubernetes.io/notReady:NoExecute for 300s
		node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  FirstSeen	LastSeen	Count	From		SubObjectPath				Type		Reason		Message
  ---------	--------	-----	----		-------------				--------	------		-------
  23h		1m		282	kubelet, ay	spec.containers{kubernetes-dashboard}	Normal		Pulled		Container image "registry.cn-hangzhou.aliyuncs.com/google-containers/kubernetes-dashboard-amd64:v1.7.1" already present on machine
  23h		1m		282	kubelet, ay	spec.containers{kubernetes-dashboard}	Normal		Created		Created container
  23h		1m		282	kubelet, ay	spec.containers{kubernetes-dashboard}	Normal		Started		Started container
  23h		7s		6578	kubelet, ay	spec.containers{kubernetes-dashboard}	Warning		BackOff		Back-off restarting failed container
  23h		7s		6580	kubelet, ay						Warning		FailedSync	Error syncing pod

[root@ay ay.k8s.d]# kubectl describe secret kubernetes-dashboard-certs -n kube-system
Name:		kubernetes-dashboard-certs
Namespace:	kube-system
Labels:		k8s-app=kubernetes-dashboard
Annotations:	
Type:		Opaque

Data
====
dashboard.crt:	1123 bytes
dashboard.key:	1704 bytes

maciaszczykm · 2017-10-27T07:10:20Z

At a first glance, It looks like Dashboard cannot open certs/dashboard.crt file, which should be created on a mounted volume (https://github.com/kubernetes/dashboard/blob/master/src/deploy/recommended/kubernetes-dashboard.yaml#L116), but it seems to be there.

It is just a guess, but it might be some Docker problem (https://docs.docker.com/machine/reference/regenerate-certs/).

colinlabs · 2017-10-31T07:44:47Z

+1 same issue

Dashboard version: 1.7.1
Kubernetes version: 1.8.1
Operating system: Ubuntu 16.04

pennpeng · 2017-11-03T09:16:33Z

+1 same issue

Dashboard version: 1.7.1
Kubernetes version: v1.7.9+coreos.0
Operating system: centos 7.3

floreks · 2017-11-08T07:32:43Z

2017/10/25 00:57:16 Couldn't read CA certificate: open : no such file or directory

This is a bit weird error because it does not come from us. I agree with @maciaszczykm because from what I have checked it might indeed be related to docker. Even the path in this error is missing.

Similar issue: boot2docker/osx-installer#126

Sidney9217 · 2017-11-09T07:47:35Z

+1 same issue

maciaszczykm · 2017-11-09T08:28:05Z

Did anoyone try solution from #2518 (comment)?

floreks · 2017-11-09T08:35:46Z

Can all of you paste your docker version? I have found this error in one of our vendored deps:
https://github.com/docker/distribution/blob/b6e0cfbdaa1ddc3a17c95142c7bf6e42c5567370/vendor/github.com/docker/libtrust/key_manager.go#L141

It definitely looks like issue or conflict with docker daemon.

colinlabs · 2017-11-09T13:04:54Z

@floreks

docker version

Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 03:35:14 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.2-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 03:35:14 2017
 OS/Arch:      linux/amd64
 Experimental: false

docker info

Containers: 10
 Running: 10
 Paused: 0
 Stopped: 0
Images: 6
Server Version: 17.03.2-ce
Storage Driver: aufs
 Root Dir: /data/docker/aufs
 Backing Filesystem: extfs
 Dirs: 50
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-62-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.797 GiB
Name: iZ2ze1zz9zmbbh6ah52701Z
ID: 2WHB:6Y7P:YLW7:FKDB:2GEK:TU4L:VS6K:ODQE:5JLD:APC3:Q7W4:MYBI
Docker Root Dir: /data/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
Live Restore Enabled: false

WARNING: No swap limit support

shenkonghui · 2017-11-21T10:02:22Z

今天刚遇到这个问题，折腾了好久才发现是阿里云这个镜像有问题
registry.cn-hangzhou.aliyuncs.com/google-containers/kubernetes-dashboard-amd64:v1.7.1
换这个镜像
registry.cn-hangzhou.aliyuncs.com/kube_containers/kubernetes-dashboard-amd64:v1.7.1

page-fault-in-nonpaged-area · 2017-11-29T02:18:22Z

Same issue. Upgraded K8S from 1.7 to 1.8. Biggest mistake ever. Everything broken, dashboard crashloopbackoff, Cassandra crashes and restarts without logs, etc etc

floreks · 2017-11-29T08:07:45Z

Looks like some core issue or conflict with docker daemon. I'm afraid that we can't fix that directly in Dashboard.

pingod · 2018-05-06T14:34:49Z

@sillky 我改镜像也没用

maciaszczykm · 2019-01-10T09:24:04Z

Closing as stale.

/close

k8s-ci-robot · 2019-01-10T09:24:05Z

@maciaszczykm: Closing this issue.

In response to this:

Closing as stale.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

maciaszczykm added kind/bug Categorizes issue or PR as related to a bug. priority/P1 labels Oct 27, 2017

maciaszczykm added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. labels Feb 27, 2018

maciaszczykm removed priority/P1 labels Nov 9, 2018

k8s-ci-robot closed this as completed Jan 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Couldn't read CA certificate: open : no such file or directory #2518

Couldn't read CA certificate: open : no such file or directory #2518

dreamlover commented Oct 25, 2017 •

edited

Loading

maciaszczykm commented Oct 26, 2017

dreamlover commented Oct 27, 2017 •

edited

Loading

maciaszczykm commented Oct 27, 2017 •

edited

Loading

colinlabs commented Oct 31, 2017

pennpeng commented Nov 3, 2017

floreks commented Nov 8, 2017 •

edited

Loading

Sidney9217 commented Nov 9, 2017

maciaszczykm commented Nov 9, 2017

floreks commented Nov 9, 2017

colinlabs commented Nov 9, 2017 •

edited

Loading

shenkonghui commented Nov 21, 2017

page-fault-in-nonpaged-area commented Nov 29, 2017

floreks commented Nov 29, 2017

pingod commented May 6, 2018

maciaszczykm commented Jan 10, 2019

k8s-ci-robot commented Jan 10, 2019

Couldn't read CA certificate: open : no such file or directory #2518

Couldn't read CA certificate: open : no such file or directory #2518

Comments

dreamlover commented Oct 25, 2017 • edited Loading

Environment

Observed result

maciaszczykm commented Oct 26, 2017

dreamlover commented Oct 27, 2017 • edited Loading

maciaszczykm commented Oct 27, 2017 • edited Loading

colinlabs commented Oct 31, 2017

pennpeng commented Nov 3, 2017

floreks commented Nov 8, 2017 • edited Loading

Sidney9217 commented Nov 9, 2017

maciaszczykm commented Nov 9, 2017

floreks commented Nov 9, 2017

colinlabs commented Nov 9, 2017 • edited Loading

shenkonghui commented Nov 21, 2017

page-fault-in-nonpaged-area commented Nov 29, 2017

floreks commented Nov 29, 2017

pingod commented May 6, 2018

maciaszczykm commented Jan 10, 2019

k8s-ci-robot commented Jan 10, 2019

dreamlover commented Oct 25, 2017 •

edited

Loading

dreamlover commented Oct 27, 2017 •

edited

Loading

maciaszczykm commented Oct 27, 2017 •

edited

Loading

floreks commented Nov 8, 2017 •

edited

Loading

colinlabs commented Nov 9, 2017 •

edited

Loading