kubeadm: "failure loading ca certificate: the certificate is not valid yet"

[kierenj]

Please provide the following details:

Environment:

Minikube version (use minikube version): 0.26.1

• OS (e.g. from /etc/os-release): Windows 10 (build 16299)
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): hyperv
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): file://C:/Users/kiere/.minikube/cache/iso/minikube-v0.26.0.iso
• Install tools: ?
• Others: ?

What happened:

C:\Users\kiere> sudo minikube start --vm-driver hyperv --hyperv-virtual-switch "Primary Virtual Switch"
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Downloading kubeadm v1.10.0
Downloading kubelet v1.10.0
Finished Downloading kubeadm v1.10.0
Finished Downloading kubelet v1.10.0
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E0419 08:54:23.948090    2468 start.go:276] Error starting cluster:  kubeadm init error sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data
--ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  running command: : running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
 output: [init] Using Kubernetes version: v1.10.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
                                                                                                                                                                [WARNING Swap]: running with swap on is not supported. Please disable swap
Flag --admission-control has been deprecated, Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version.
failure loading ca certificate: the certificate is not valid yet
: running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
.: Process exited with status 1

What you expected to happen:
Minikube VM to start successfully.

How to reproduce it (as minimally and precisely as possible):
Follow the exact steps here: https://medium.com/@JockDaRock/minikube-on-windows-10-with-hyper-v-6ef0f4dc158c

Output of minikube logs (if applicable):

F0419 09:01:40.025689   19032 logs.go:50] Error getting cluster bootstrapper: getting kubeadm bootstrapper: getting ssh client: Error creating new ssh host from driver: Error getting ssh host name for driver: Host is not running

Anything else do we need to know:
Would be great to understand if the "WARNING Swap" is relevant/related. I'm unsure if it's referring to the host OS, or VM OS - or how to remedy?

[fabriziocucci]

Same issue on macOS Sierra.

[AmazingTurtle]

Same here. If you delete and recreate the minikube, it's saying that the apiserver certificate is wrong. I'm using VirtualBox. The vm driver is not the cause.

I can confirm that this issue does not occur in v0.25.2, starts perfectly.

$ minikube delete && minikube start
Deleting local Kubernetes cluster...
Machine deleted.
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E0504 20:16:11.725862   14520 start.go:276] Error starting cluster:  kubeadm init error sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  running command: : running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
 output: [init] Using Kubernetes version: v1.10.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[certificates] Using the existing ca certificate and key.
        [WARNING Swap]: running with swap on is not supported. Please disable swap
Flag --admission-control has been deprecated, Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version.
failure loading apiserver certificate: the certificate is not valid yet
: running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
.: Process exited with status 1
================================================================================
An error has occurred. Would you like to opt in to sending anonymized crash
information to minikube to help prevent future errors?
To opt out of these messages, run the command:
        minikube config set WantReportErrorPrompt false
================================================================================

[chrobotm]

same issue here

[fore5fire]

Experiencing the same issue, Linux with kvm2 driver, minikube v0.26.1. I started getting this error when trying to start my previously-working cluster:

minikube start --vm-driver kvm2
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E0514 21:16:43.158255    5693 start.go:281] Error restarting cluster:  running cmd: 
sudo kubeadm alpha phase certs all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase kubeconfig all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase controlplane all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase etcd local --config /var/lib/kubeadm.yaml
: Process exited with status 1

After trying to delete and recreate the cluster, I get the same issue as OP. After getting the "the certificate is not yet valid" error, If I try to start the vm again without deleting the vm first I get the same error that I was having before I deleted the cluster.

[jstangroome]

On Windows 10 version 1803 (OS Build 17134.48) with Virtualbox 5.2.12 (also 5.2.6) and Minikube 0.27.0 (also 0.26.1) I get the "certificate is not yet valid" error during a minikube start with no existing Minikube VM.

Comparing the host time to the VM time, I see a 3 second difference:

PS > get-date ;  minikube ssh -- date --utc ; get-date
Friday, 18 May 2018 14:21:47
Fri May 18 04:21:44 UTC 2018
Friday, 18 May 2018 14:21:47

Comparing the cert validity time to the cert file modified time I see the same 3 second difference:

$ minikube ssh -- stat /var/lib/localkube/certs/apiserver.crt
  File: /var/lib/localkube/certs/apiserver.crt
  Size: 1298            Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d      Inode: 3670105     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-05-18 04:41:51.768791568 +0000
Modify: 2018-05-18 04:41:50.237026124 +0000
Change: 2018-05-18 04:41:50.237026124 +0000
 Birth: -

$ minikube ssh -- cat /var/lib/localkube/certs/apiserver.crt | openssl x509 -noout -text | grep alidity -A2
        Validity
            Not Before: May 18 04:41:53 2018 GMT
            Not After : May 18 04:41:53 2019 GMT

---

Update: after leaving the host running for a few days without reboots, the clock discrepancy between the host and newly created VMs has reduced to subsecond and the "certificate is not yet valid" error is no longer encountered.

Since the clock discrepancy is between the host and a new VM (as opposed to existing VMs) and since it has impacted users with Virtualbox and users with Hyper-V, implementing a change to be less sensitive to time variations during minikube start would seem prudent.

[jstangroome]

If I run minikube start again following the "certificate is not yet valid" error but after enough time has elapsed for the VM clock to be after the certificate's Not Before time, then I get the same error as reported above by @LSmith130:

Starting cluster components...
E0518 15:54:06.919233   13964 start.go:281] Error restarting cluster:  running cmd:
sudo kubeadm alpha phase certs all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase kubeconfig all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase controlplane all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase etcd local --config /var/lib/kubeadm.yaml
: Process exited with status 1

Note the above output is with minikube log level 99.

If I run the supposedly failing command directly with minikube ssh it does not fail:

PS > minikube ssh -- "sudo kubeadm alpha phase certs all --config /var/lib/kubeadm.yaml &&
>> sudo /usr/bin/kubeadm alpha phase kubeconfig all --config /var/lib/kubeadm.yaml &&
>> sudo /usr/bin/kubeadm alpha phase controlplane all --config /var/lib/kubeadm.yaml &&
>> sudo /usr/bin/kubeadm alpha phase etcd local --config /var/lib/kubeadm.yaml ; echo EXITED `$?"
[certificates] Using the existing ca certificate and key.
[certificates] Using the existing apiserver certificate and key.
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [minikube] and IPs [192.168.99.101]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/var/lib/localkube/certs/"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
EXITED 0

Trying minikube start again after verifying the command succeeds once again reports failure for the same command.

However, minikube status reports healthy:

PS > minikube status
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.101

Meanwhile the storage-provisioner and kubernetes-dashboard are both in a crash loop:

PS > kubectl get pod --all-namespaces
NAMESPACE     NAME                                    READY     STATUS             RESTARTS   AGE
kube-system   etcd-minikube                           1/1       Running            0          7m
kube-system   kube-addon-manager-minikube             1/1       Running            0          6m
kube-system   kube-apiserver-minikube                 1/1       Running            0          7m
kube-system   kube-controller-manager-minikube        1/1       Running            0          7m
kube-system   kube-scheduler-minikube                 1/1       Running            0          7m
kube-system   kubernetes-dashboard-5498ccf677-t9jff   0/1       CrashLoopBackOff   5          7m
kube-system   storage-provisioner                     0/1       CrashLoopBackOff   5          7m

PS > kubectl logs -n kube-system storage-provisioner
F0518 06:04:00.140866       1 main.go:37] Error getting server version: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout

PS > kubectl logs -n kube-system kubernetes-dashboard-5498ccf677-t9jff
2018/05/18 06:06:47 Starting overwatch
2018/05/18 06:06:47 Using in-cluster config to connect to apiserver
2018/05/18 06:06:47 Using service account token for csrf signing
2018/05/18 06:06:47 No request provided. Skipping authorization
2018/05/18 06:07:17 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ

For reference, the kubeadm.yaml file contents:

PS > minikube ssh -- sudo cat /var/lib/kubeadm.yaml
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
api:
  advertiseAddress: 192.168.99.101
  bindPort: 8443
kubernetesVersion: v1.10.0
certificatesDir: /var/lib/localkube/certs/
networking:
  serviceSubnet: 10.96.0.0/12
etcd:
  dataDir: /data
nodeName: minikube
apiServerExtraArgs:
  admission-control: "Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"

[chrobotm]

v0.27 is working for me on mac osx

[aseempatni]

I'm getting failure loading ca certificate: the certificate is not valid yeton Mac with minikube version: v0.27.0

[kierenj]

@mbotmcc @aseempatni - see the linked pull request above, it looks like a fix is identified and ready to merge - but isn't merged yet. I don't think 0.27 would have anything other than intermittent success

[chrobotm]

@aseempatni I got this error too but after I deleted everything in .minikube and downloaded everything again it worked

[addisonhuddy]

Getting the same issue.

E0609 14:36:04.659245   61628 start.go:276] Error starting cluster:  ...

Workaround: minikube ssh and run the init command

sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI

kubectl can now connect to minikube. Still have no idea what is actually going on.

[h0nor]

I first installed v0.27.0 on MacOS 10.11.6 El Capitan.
Tried many things but it didn't work.

To make it work this is what I did:

1. cd /Users/user_name/.minikube/cache
2. rm -Rf localkube
3. If you have another folder like "v1.10.0" or "v1.9.0"
   rm -Rf v1.10.0
4. curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.25.2/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
5. minikube start

Thanks @AmazingTurtle for referring to a version that actually works!

[Arnavion]

The PR with the fix should be in 0.28.0, so this should be fixed now.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tstromberg]

Fixed in 0.28.0.

[tstromberg]

Reopening because we're seeing this issue in our CI environment, although it's only showing up in VirtualBox at the moment. The root cause is effectively that time in the VM is lagging compared to the host.

example:

[certificates] Generated etcd/healthcheck-client certificate and key.                                                               
failure loading etcd/ca certificate authority: the certificate is not valid yet   

[tstromberg]

Doing some reading, vbox doesn't sync time on VM start:

https://vilimpoc.org/blog/2014/02/09/better-time-synchronization-with-virtualbox-guest-additions/

I'm working on a PR to configure this.