Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

None w/ Docker 18.09: [ERROR SystemVerification]: unsupported docker version: 18.09.0 #3323

Closed
Ark-kun opened this Issue Nov 10, 2018 · 24 comments

Comments

Projects
None yet
@Ark-kun
Copy link

Ark-kun commented Nov 10, 2018

This is a BUG REPORT

VM-less Minikube fails to start on Docker 18.09:

[ERROR SystemVerification]: unsupported docker version: 18.09.0

Environment:
Minikube version: v1.12.2

  • OS: Debian 9
  • VM Driver : None

What happened:
Our test script that installs Docker and VM-less Minikube started failing. Upgrading Minikube to 1.12.2 did not help.

**Logs:

+ export MINIKUBE_WANTUPDATENOTIFICATION=false
+ MINIKUBE_WANTUPDATENOTIFICATION=false
+ export MINIKUBE_WANTREPORTERRORPROMPT=false
+ MINIKUBE_WANTREPORTERRORPROMPT=false
+ export MINIKUBE_HOME=/home/not-root
+ MINIKUBE_HOME=/home/not-root
+ export CHANGE_MINIKUBE_NONE_USER=true
+ CHANGE_MINIKUBE_NONE_USER=true
+ mkdir -p /home/not-root/.kube
+ touch /home/not-root/.kube/config
+ export KUBECONFIG=/home/not-root/.kube/config
+ KUBECONFIG=/home/not-root/.kube/config
+ sudo -E /usr/local/bin/minikube start --vm-driver=none --kubernetes-version=v1.12.2
Starting local Kubernetes v1.12.2 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Downloading kubeadm v1.12.2
Downloading kubelet v1.12.2
Finished Downloading kubeadm v1.12.2
Finished Downloading kubelet v1.12.2
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E1109 23:47:22.788065   12517 start.go:297] Error starting cluster:  kubeadm init error 
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
sudo /usr/bin/kubeadm alpha phase addon coredns
 running command: : running command: 
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
sudo /usr/bin/kubeadm alpha phase addon coredns

 output: [init] using Kubernetes version: v1.12.2
[preflight] running pre-flight checks
	[WARNING FileExisting-ebtables]: ebtables not found in system path
	[WARNING FileExisting-socat]: socat not found in system path
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 4.9.0-8-amd64
CONFIG_NAMESPACES: enabled
CONFIG_NET_NS: enabled
CONFIG_PID_NS: enabled
CONFIG_IPC_NS: enabled
CONFIG_UTS_NS: enabled
CONFIG_CGROUPS: enabled
CONFIG_CGROUP_CPUACCT: enabled
CONFIG_CGROUP_DEVICE: enabled
CONFIG_CGROUP_FREEZER: enabled
CONFIG_CGROUP_SCHED: enabled
CONFIG_CPUSETS: enabled
CONFIG_MEMCG: enabled
CONFIG_INET: enabled
CONFIG_EXT4_FS: enabled (as module)
CONFIG_PROC_FS: enabled
CONFIG_NETFILTER_XT_TARGET_REDIRECT: enabled (as module)
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled (as module)
CONFIG_OVERLAY_FS: enabled (as module)
CONFIG_AUFS_FS: not set - Required for aufs.
CONFIG_BLK_DEV_DM: enabled (as module)
DOCKER_VERSION: 18.09.0
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
	[WARNING Hostname]: hostname "minikube" could not be reached
	[WARNING Hostname]: hostname "minikube" lookup minikube on 169.254.169.254:53: no such host
[preflight] Some fatal errors occurred:
	[ERROR SystemVerification]: unsupported docker version: 18.09.0
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
: running command: 
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
sudo /usr/bin/kubeadm alpha phase addon coredns
@ryanolson

This comment has been minimized.

Copy link

ryanolson commented Nov 10, 2018

Confirmed. I see the same.

@afbjorklund

This comment has been minimized.

Copy link
Contributor

afbjorklund commented Nov 10, 2018

You need to downgrade docker to 18.06, until 18.09 is validated and supported by kubeadm:

https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/system/docker_validator.go#L41

It was only recently that kubeadm started to support anything beyond version 17.03: #3223

Apparently you are also missing other binaries, such as socat and maybe also crictl...

@afbjorklund

This comment has been minimized.

Copy link
Contributor

afbjorklund commented Nov 10, 2018

See https://kubernetes.io/docs/setup/cri/#docker for instructions. Or just use minikube.iso.

Not sure if there was a bug report filed with kubernetes/kubeadm already, that would be next.

@thomas-riccardi

This comment has been minimized.

Copy link

thomas-riccardi commented Nov 13, 2018

@afbjorklund kubernetes pre-flight checks have 3 possible outcome: OK, WARNING, ERROR.

The docker version check can return the 3 outcomes:

  • supported version: OK
  • unsupported but recognized version: WARNING
  • all others: ERROR

Since docker 18.09 we get an ERROR instead of an expected WARNING.

Analysis

Code

https://github.com/kubernetes/kubernetes/blame/master/cmd/kubeadm/app/util/system/docker_validator.go#L73-L99

  • matched is for supported versions: in our case it's false
  • then it checks for recognized version with \d{2}\.\d+\.\d+-[a-z]{2} regex

Expected result:
the ver regex matches on the 18.09 docker version, and thus returns a WARNING.

Actual result:
the ver regex does no match, and thus returns an ERROR.

Docker versions

18.06

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:56 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false

18.09

$ docker version
Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:49:01 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:16:44 2018
  OS/Arch:          linux/amd64
  Experimental:     false

They dropped the -ce suffix in this new version => the regex should be updated.

@afbjorklund

This comment has been minimized.

Copy link
Contributor

afbjorklund commented Nov 13, 2018

Nice analysis! But still needs a kubeadm update ? Both to fix the regexp, and to validate if the new version "works"...

@tstromberg

This comment has been minimized.

Copy link
Contributor

tstromberg commented Nov 13, 2018

It's worth noting that our CI "None" instance recently upgraded docker to 18.09, so it's failing tests due to this issue as well.

@tstromberg tstromberg changed the title VM-less Minikube fails to start on Docker 18.09: None w/ Docker 18.09: [ERROR SystemVerification]: unsupported docker version: 18.09.0 Nov 13, 2018

@afbjorklund

This comment has been minimized.

Copy link
Contributor

afbjorklund commented Nov 13, 2018

@tstromberg : according to the kubeadm documentation, it is supposed to be on "hold" for now:

## Install docker.
apt-get update && apt-get install docker-ce=18.06.0~ce~3-0~ubuntu
## Install docker.
yum update && yum install docker-ce-18.06.1.ce

https://kubernetes.io/docs/setup/cri/#docker

I also dislike the way that the new "containerd.io" package now conflicts with "runc", breaking apt...

Unpacking containerd.io (1.2.0~rc.2-1) ...
dpkg: error processing archive /var/cache/apt/archives/containerd.io_1.2.0~rc.2-1_amd64.deb (--unpack):
 trying to overwrite '/usr/sbin/runc', which is also in package runc 1.0.0~rc2+docker1.13.1-0ubuntu1~16.04.1
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)

Then again "cri-o-1.10" uses "cri-o-runc", so I guess they can coexist. Just bad upgrade experience.

@balopat

This comment has been minimized.

Copy link
Contributor

balopat commented Nov 13, 2018

it's helpful to get the exact docker-ce version number on non-ubuntu debian versions:

apt-cache madison docker-ce | grep 18.06 | awk '{print $3}' | head -n 1
@balopat

This comment has been minimized.

Copy link
Contributor

balopat commented Nov 13, 2018

I downgraded the CI server so None driver tests are passing again.
Even though when 1.13 comes out we have to think about this a little bit: do we make 1.13 the default for None integration tests? Or we wait with the docker upgrade until 1.13 becomes the default in minikube?

@tsuna

This comment has been minimized.

Copy link

tsuna commented Nov 25, 2018

Since the fix in kubernetes/kubernetes#71001 merely consists in updating the regexp to ignore the -ce suffix, is there a known workaround we can use in the meantime?

@stucki

This comment has been minimized.

Copy link

stucki commented Nov 25, 2018

How about this:

kubeadm init [...] --ignore-preflight-errors=SystemVerification
@afbjorklund

This comment has been minimized.

Copy link
Contributor

afbjorklund commented Nov 25, 2018

Why not just use the older Docker version ?

@stucki

This comment has been minimized.

Copy link

stucki commented Nov 25, 2018

If I had known before the Docker update, then yes. Didn't know about it and don't like to downgrade.

@gatsbys

This comment has been minimized.

Copy link

gatsbys commented Nov 29, 2018

Removing the old version with :
sudo apt-get purge docker-ce
sudo rm -rf /var/lib/docker

And then reinstall as @afbjorklund said :

apt-get update && apt-get install docker-ce=18.06.0~ce~3-0~ubuntu

Worked for me.

@aashish-sheshadri

This comment has been minimized.

Copy link

aashish-sheshadri commented Dec 31, 2018

Is there a way to add --ignore-preflight-errors=SystemVerification?

@srikiz

This comment has been minimized.

Copy link

srikiz commented Jan 3, 2019

The above comments from @gatsbys worked for me perfectly.

@eltsdev

This comment has been minimized.

Copy link

eltsdev commented Jan 4, 2019

Happening to me too! I confirm that latest docker version (Docker 18.09) is not supported by minikube using no vm option in Ubuntu 16.04.

@tstromberg

This comment has been minimized.

Copy link
Contributor

tstromberg commented Jan 18, 2019

I haven't confirmed yet, but I think this should be resolved by using a newer Kubernetes release (v1.13.x) according to https://kubernetes.io/docs/setup/release/notes/

If you update to minikube v0.33.0, it defaults to Kubernetes v1.13.2, though you can also select it using --kubernetes-version in v0.32.0. Does someone mind confirming whether this is an issue any longer?

It would be nice to allow users to ignore SystemVerification errors. I've opened #3543 to address that feature request.

@thomas-riccardi

This comment has been minimized.

Copy link

thomas-riccardi commented Jan 18, 2019

@tstromberg I can confirm this now works with the following setup:

  • docker 18.09.0 or 18.09.1
  • minikube 0.32.0
  • --kuberentes-version 1.31.1 or 1.13.2

So this should work again out-of-the-box with minikube 0.33.0 which defaults to kubernetes 1.13.2.

@tstromberg tstromberg closed this Jan 23, 2019

@dlakatos847

This comment has been minimized.

Copy link

dlakatos847 commented Feb 13, 2019

@tstromberg confirmed, works out of the box 👍

  • OpenSuSE Tumbleweed x86-64 up-to-date at 2019.02.13. running on Linux kernel 4.20.6
  • Docker CE 18.09
  • Minikube v0.33.1
  • Kubernetes v1.13.2
@marcosdiez

This comment has been minimized.

Copy link

marcosdiez commented Mar 12, 2019

if you are using ubuntu 16.04, this is the magic command that solves it all
sudo apt-get install docker.io=18.06.1-0ubuntu1.2~16.04.1

@dperique

This comment has been minimized.

Copy link

dperique commented Mar 13, 2019

@marcosdiez yes, that worked for me. I had to install that and answered to the part where it asks if you want to restart docker now. I then restarted minikube and it seems happy again.

Thanks!

@marcosdiez

This comment has been minimized.

Copy link

marcosdiez commented Mar 14, 2019

For those who do not want to downgrade docker, I sent this PR #3879 which will allow us to pass a parameter to minikube to ignore docker changes.

After the PR gets merged, this will be the command to run sudo -E minikube start --vm-driver=none --kubernetes-version v1.11.8 --extra-config kubeadm.ignore-preflight-errors=SystemVerification

As of v0.35.0 this has not been merged.

@afbjorklund

This comment has been minimized.

Copy link
Contributor

afbjorklund commented Mar 14, 2019

Most likely minikube will support and use the same versions of Docker that kubeadm does:

https://kubernetes.io/docs/setup/cri/#docker

Version 18.06.2 is recommended, but 1.11, 1.12, 1.13, 17.03 and 18.09 are known to work as well. Keep track of the latest verified Docker version in the Kubernetes release notes.

Docker 18.09 is supported since Kubernetes 1.13, This is mostly an issue with none driver.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.