Latest docker version breaks kubelet in 1.5.0-alpha.3 #13281

0xmichalis · 2017-03-07T13:16:46Z

20s       20s       1         router-1-deploy           Pod                 Warning   FailedSync         {kubelet 10.34.129.45}   Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "Failed to check docker api version: docker: failed to parse docker version \"17.03.0-ce\": illegal zero-prefixed version component \"03\" in \"17.03.0-ce\""

@csrwng @jimmidyson

The text was updated successfully, but these errors were encountered:

0xmichalis · 2017-03-07T13:17:03Z

Running on Fedora 25

csrwng · 2017-03-07T13:23:10Z

Follow up to #13219

csrwng · 2017-03-07T15:32:08Z

I tried with the latest origin master and I get the same error (on fedora 25). However, I don't see the error on Docker for Mac.

justinclayton · 2017-03-07T19:21:39Z

I am seeing the issue with Docker for Mac 17.03.0-ce, commit 60ccb22.

oc version oc v1.4.1+3f9807a (latest from Homebrew)

csrwng · 2017-03-07T19:22:30Z

@justinclayton are you not able to run pods at all?

justinclayton · 2017-03-07T19:23:30Z

Sorry, I'm in the wrong issue. I'm trying to run oc cluster up. Looks like #13284 is what I'm looking for.

csrwng · 2017-03-07T19:24:34Z

@justinclayton you'll need an oc binary from the origin master

zak-hassan · 2017-03-08T19:03:13Z

On the latest Docker for Mac I'm experiencing issues running oc cluster up --metrics. The pods do not come up.
Any suggestions on what version of openshift cli I should use for this to work?

csrwng · 2017-03-08T19:05:17Z

@zmhassan what are your versions of docker and oc, and do you see any events in the openshift-infra namespace about the metrics pods?

zak-hassan · 2017-03-08T19:16:39Z

@csrwng

zhassan:~ zhassan$ oc cluster up --metrics
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... FAIL
   Error: Minor number must not contain leading zeroes "03"
zhassan:~ zhassan$ oc version
oc v1.5.0-alpha.3+cf7e336
kubernetes v1.5.2+43a9be4
features: Basic-Auth
Unable to connect to the server: EOF
zhassan:~ zhassan$ docker version
Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 10:40:59 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   3a232c8
 Built:        Tue Feb 28 07:52:04 2017
 OS/Arch:      linux/amd64
 Experimental: true

csrwng · 2017-03-08T19:20:00Z

@zmhassan you need the latest origin master. If you somehow have access to a cluster, you can build your own binary using https://github.com/csrwng/build-origin

zak-hassan · 2017-03-08T19:23:11Z

Hi @csrwng I could build from source as I already have all the openshift source. This step your suggestion is only going to create binary that I can easily download from github or build myself.

zak-hassan · 2017-03-08T19:23:45Z

So if I build latest it should work is what your saying?

csrwng · 2017-03-08T19:23:50Z

yes

zak-hassan · 2017-03-08T19:33:51Z

Nope doesn't work.

zhassan:kubernetes zhassan$ oc get pods
NAME                            READY     STATUS    RESTARTS   AGE
docker-registry-1-deploy        1/1       Running   0          54s
docker-registry-1-dpcwn         0/1       Running   0          28s
persistent-volume-setup-vh3pf   1/1       Running   0          54s
router-1-deploy                 1/1       Running   0          54s
router-1-ndsh6                  0/1       Running   0          29s
zhassan:kubernetes zhassan$ oc project openshift-infra
Now using project "openshift-infra" on server "https://127.0.0.1:8443".
zhassan:kubernetes zhassan$ oc get pods
NAME                         READY     STATUS    RESTARTS   AGE
metrics-deployer-pod-5ctbd   1/1       Running   0          15s
metrics-deployer-pod-lctjk   0/1       Error     0          1m
zhassan:kubernetes zhassan$ oc describe pod metrics-deployer-pod-lctjk
Name:			metrics-deployer-pod-lctjk
Namespace:		openshift-infra
Security Policy:	restricted
Node:			192.168.65.2/192.168.65.2
Start Time:		Wed, 08 Mar 2017 14:31:06 -0500
Labels:			controller-uid=c4d30a9d-0435-11e7-b875-c68515d70515
			job-name=metrics-deployer-pod
Status:			Failed
IP:			172.17.0.5
Controllers:		Job/metrics-deployer-pod
Containers:
  deployer:
    Container ID:	docker://046254a8829ecd5580eaa8430507bd1ecec285b2d5a9d62377e856907962720b
    Image:		openshift/origin-metrics-deployer:v1.5.0-alpha.3
    Image ID:		docker-pullable://openshift/origin-metrics-deployer@sha256:e3ae4eab3002fa532a30c3dcdfc320dd0a588e4adbc59896139cc1bff4c4f06e
    Port:
    State:		Terminated
      Reason:		Error
      Exit Code:	255
      Started:		Wed, 08 Mar 2017 14:31:47 -0500
      Finished:		Wed, 08 Mar 2017 14:31:57 -0500
    Ready:		False
    Restart Count:	0
    Volume Mounts:
      /etc/deploy from empty (rw)
      /secret from secret (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from metrics-deployer-token-cpn75 (ro)
    Environment Variables:
      PROJECT:				openshift-infra (v1:metadata.namespace)
      POD_NAME:				metrics-deployer-pod-lctjk (v1:metadata.name)
      IMAGE_PREFIX:			openshift/origin-
      IMAGE_VERSION:			v1.5.0-alpha.3
      MASTER_URL:			https://kubernetes.default.svc:443
      HAWKULAR_METRICS_HOSTNAME:	metrics-openshift-infra.127.0.0.1.nip.io
      MODE:				deploy
      REDEPLOY:				false
      IGNORE_PREFLIGHT:			false
      USE_PERSISTENT_STORAGE:		true
      CASSANDRA_NODES:			1
      CASSANDRA_PV_SIZE:		10Gi
      METRIC_DURATION:			7
      HEAPSTER_NODE_ID:			nodename
      METRIC_RESOLUTION:		10s
Conditions:
  Type		Status
  Initialized 	True
  Ready 	False
  PodScheduled 	True
Volumes:
  empty:
    Type:	EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  secret:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	metrics-deployer
  metrics-deployer-token-cpn75:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	metrics-deployer-token-cpn75
QoS Class:	BestEffort
Tolerations:	<none>
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  1m		1m		1	{default-scheduler }			Normal		Scheduled	Successfully assigned metrics-deployer-pod-lctjk to 192.168.65.2
  1m		1m		1	{kubelet 192.168.65.2}			Warning		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "Failed to check docker api version: docker: failed to parse docker version \"17.03.0-ce\": illegal zero-prefixed version component \"03\" in \"17.03.0-ce\""

  1m	1m	1	{kubelet 192.168.65.2}	spec.containers{deployer}	Normal	Pulling		pulling image "openshift/origin-metrics-deployer:v1.5.0-alpha.3"
  49s	49s	1	{kubelet 192.168.65.2}	spec.containers{deployer}	Normal	Pulled		Successfully pulled image "openshift/origin-metrics-deployer:v1.5.0-alpha.3"
  48s	48s	1	{kubelet 192.168.65.2}	spec.containers{deployer}	Normal	Created		Created container with docker id 046254a8829e; Security:[seccomp=unconfined]
  47s	47s	1	{kubelet 192.168.65.2}	spec.containers{deployer}	Normal	Started		Started container with docker id 046254a8829e
  47s	47s	1	{kubelet 192.168.65.2}					Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "deployer" with RunContainerError: "Failed to check docker api version: docker: failed to parse docker version \"17.03.0-ce\": illegal zero-prefixed version component \"03\" in \"17.03.0-ce\""

  35s	35s	1	{kubelet 192.168.65.2}	spec.containers{deployer}	Normal	Killing	Killing container with docker id 046254a8829e: Need to kill pod.

csrwng · 2017-03-08T19:46:06Z

@zmhassan what version of the images are you using ? (Are you specifying --version=blah with oc cluster up)

csrwng · 2017-03-08T20:27:12Z

@zmhassan so I tried locally and I did see the errors but eventually pods ran and metrics came up.

csrwng · 2017-03-13T17:15:31Z

Kube issue: kubernetes/kubernetes#42492

csrwng · 2017-03-13T18:53:58Z

It looks like something changed in the kubelet recently that broke things. Running cluster up with --version=v1.5.0-alpha.3 logs the message about the version parsing failing, but the pods eventually do start. With --version=v1.5.0-rc.0, the pods don't start anymore.

/cc @derekwaynecarr

mbechauf · 2017-03-13T21:49:39Z

When I try oc cluster up --version=v1.5.0-alpha.3 and create a project, the build fail immediately with

error: cannot connect to the server: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

I then tried v1.5.0-alpha.3 client tools with oc cluster up --version=v1.5.0-alpha.3 fails immediately with

-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... FAIL
Error: Minor number must not contain leading zeroes "03"

Looks like I'm stuck. I was trying to use origin for an internal class in a week. Is there a way to install an older docker engine on CentOS?

zak-hassan · 2017-03-14T14:24:56Z

I'm still having issues and I'm not seeing metrics show up at all.

csrwng · 2017-03-14T14:27:20Z

@mbechauf I believe the default centos yum repo has docker 1.12.x

@zmhassan are you getting any pods to start at all? Are you using --version=v1.5.0-alpha.3 to start your cluster?

zak-hassan · 2017-03-14T14:49:05Z

I managed to get it all working. Thank you.

shveik · 2017-03-16T19:10:27Z

got a little further, but the issue is now during build:
Failed sync Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: docker: failed to parse docker version "17.03.0-ce": illegal zero-prefixed version component "03" in "17.03.0-ce""

csrwng · 2017-03-16T19:12:11Z

@shveik is your registry up and running?

shveik · 2017-03-16T19:34:21Z

great question, please bare with me since it's my first day playing with this, and i am following instructions here: https://github.com/openshift/origin/blob/master/docs/cluster_up_down.md#macos-with-docker-for-mac

so, looks like the registry is running:
➜ ~ oc get svc docker-registry -n default
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
docker-registry 172.30.1.1 5000/TCP 16m

however, the login is not working, since OPENSHIFT_TOKEN is null:
➜ ~ OPENSHIFT_TOKEN=$(oc whoami -t)
error: no token is currently in use for this session

i do have an insecure registry configured for 172.30.0.0/16

csrwng · 2017-03-16T20:29:17Z

@shveik did you specify --version=v1.5.0-alpha.3 with 'oc cluster up'? If not, try that first

csrwng · 2017-03-16T20:30:06Z

If you are logged in as system:admin, you will not have a token, since you are logged in via certificate. If you need a token, you need to be a regular user.

mbechauf · 2017-03-16T20:39:22Z

All my attempts (even with previous Alpha versions) to get beyond the Docker version issues have failed. I ended up installing an old Docker 1.13 version from https://yum.dockerproject.org/. With that, v1.5.0-rc.0 has worked just fine. I hope it stays that way. Will need that system for an intro class to OpenShift next week ... Best, Michael

…

On Thu, Mar 16, 2017 at 12:34 PM, Alex Reznik ***@***.***> wrote: great question, please bare with me since it's my first day playing with this, and i am following instructions here: https://github.com/openshift/ origin/blob/master/docs/cluster_up_down.md#macos-with-docker-for-mac so, looks like the registry is running: ➜ ~ oc get svc docker-registry -n default NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE docker-registry 172.30.1.1 5000/TCP 16m however, the login is not working, since OPENSHIFT_TOKEN is null: ➜ ~ OPENSHIFT_TOKEN=$(oc whoami -t) error: no token is currently in use for this session i do have an insecure registry configured for 172.30.0.0/16 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13281 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJqqVDBDyEZHGrZQEfbhZ7zG6SM6sDo5ks5rmY7DgaJpZM4MVd22> .

shveik · 2017-03-16T20:49:11Z

@csrwng that was it, thank you. so when running with version, here is the output, so basically, it's the client and server versions that differ:
➜ ~ oc version
oc v1.5.0-rc.0+49a4a7a
kubernetes v1.5.2+43a9be4
features: Basic-Auth

Server https://127.0.0.1:8443
openshift v1.5.0-alpha.3+cf7e336
kubernetes v1.5.2+43a9be4

and here is without version:
➜ ~ oc version
oc v1.5.0-rc.0+49a4a7a
kubernetes v1.5.2+43a9be4
features: Basic-Auth

Server https://127.0.0.1:8443
openshift v1.5.0-rc.0+49a4a7a
kubernetes v1.5.2+43a9be4

cars10 · 2017-03-21T12:20:08Z

how to solve that when using the server binary instead of using oc cluster up?

I use ./openshift start and experience the same issue. any way to specify the version there?

stevekuznetsov · 2017-03-23T17:06:04Z

@mbechauf on CentOS for installing Docker, use yum install docker -- you will have 1.12 available to you in normal streams and 1.13 in @epel-testing

soul2zimate · 2017-03-31T06:48:21Z

I see a similar issue with current master:
openshift v3.6.0-alpha.0+611176d-121
kubernetes v1.5.2+43a9be4
etcd 3.1.0

Build is pending when I try to create a new ruby-ex app. See following message in Build -> Last Build -> Events tab:

Time Severity Reason Message
2:38:40 PM Warning Failed sync Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: docker: failed to parse docker version "17.03.0-ce": illegal zero-prefixed version component "03" in "17.03.0-ce""
18 times in the last 3 minutes

csrwng · 2017-03-31T11:59:01Z

The kubelet is still not working with the new Docker version string. Easiest thing to do is to downgrade Docker.

mbn18 · 2017-04-14T07:08:24Z

Got stuck on the first tutorial.

OS: Fedora25

Version:

oc v3.6.0-alpha.1+7044e57-29
kubernetes v1.5.2+43a9be4
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://192.168.1.100:8443
openshift v3.6.0-alpha.1+7044e57-29
kubernetes v1.5.2+43a9be4

The error:

Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: docker: failed to parse docker version "17.04.0-ce": illegal zero-prefixed version component "04" in "17.04.0-ce""

Tutorial worked on:

oc v1.4.1+3f9807a
kubernetes v1.4.0+776c994
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://192.168.1.100:8443
openshift v1.4.1+3f9807a
kubernetes v1.4.0+776c994

marioluan · 2017-04-15T14:21:51Z

Which docker version should I use to make it work, while we don't have the [final] working solution?

stevekuznetsov · 2017-04-17T11:49:07Z

OpenShift Origin 1.5 should use Docker 1.12.

simbo1905 · 2017-04-20T14:40:03Z

On mac os with latest Docker for Mac Version 17.03.1-ce-mac5 (16048) I was able to oc cluster up with the oc from openshift-origin-client-tools-v1.5.0 but was unable to deploy an image seeing:

"runContainer: docker: failed to parse docker version" 17.03 illegal zero-prefixed version component "03" in "17.03.1-ce""

changing the oc cluster up command to name the alpha version in the chain above solved it:

./oc cluster up --version=v1.5.0-alpha.3   \
                --use-existing-config   \
                --host-data-dir=/oc_data \
                --metrics=true

The deployment just automatically ran and succeeded on the older version.

csrwng · 2017-04-20T14:42:50Z

An alternative for now is also to downgrade to Docker 1.13.1 for Mac:
https://download.docker.com/mac/stable/1.13.1.15353/Docker.dmg

soltysh · 2017-04-27T13:37:41Z

We will need to cherry-pick kubernetes/kubernetes#44068, probably to solve the problem

soltysh · 2017-04-27T13:39:24Z

I'll pick it up once the upstream PR merges and the rebase lands

soltysh · 2017-05-12T09:06:17Z

I've cherry-picked the upstream fix in attached PR.

csrwng assigned derekwaynecarr Mar 7, 2017

metmajer mentioned this issue Mar 7, 2017

Latest docker version breaks oc cluster up in 1.5.0-alpha.3 #13284

Closed

pweil- added component/kubernetes priority/P2 kind/bug Categorizes issue or PR as related to a bug. labels Mar 7, 2017

pweil- mentioned this issue Mar 8, 2017

Failing to load metrics: oc cluster up --metrics not working #13310

Closed

csrwng mentioned this issue Mar 13, 2017

cluster up: Use loopback interface for nodename and default server IP #13112

Merged

csrwng mentioned this issue Mar 14, 2017

[1.5] cluster up: warn on error parsing Docker version #13204

Merged

kcd83 mentioned this issue Mar 23, 2017

Latest docker CE version kubelet failed to "StartContainer" for StatefulSet #13520

Closed

This was referenced Apr 10, 2017

Application creation failed on local cluster running on docker version 17.03.1-ce #13695

Closed

docker-registry is in pending state #13711

Closed

cdrage mentioned this issue Apr 12, 2017

Adding kompose up/down tests for openshift kubernetes/kompose#460

Merged

soltysh mentioned this issue Apr 27, 2017

Kubernetes issues: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory #13865

Closed

soltysh self-assigned this Apr 27, 2017

soltysh added this to the 3.6.0 milestone Apr 27, 2017

soltysh mentioned this issue May 12, 2017

UPSTREAM: 44068: Use Docker API Version instead of docker version #14158

Merged

openshift-bot closed this as completed in #14158 May 21, 2017

Montecito mentioned this issue Jul 7, 2017

Use template with local openshift (oc cluster up) Gepardec/Hogarama#50

Closed

4 tasks

Latest docker version breaks kubelet in 1.5.0-alpha.3 #13281

Latest docker version breaks kubelet in 1.5.0-alpha.3 #13281

Comments

0xmichalis commented Mar 7, 2017

0xmichalis commented Mar 7, 2017

csrwng commented Mar 7, 2017

csrwng commented Mar 7, 2017

justinclayton commented Mar 7, 2017 • edited

csrwng commented Mar 7, 2017

justinclayton commented Mar 7, 2017 • edited

csrwng commented Mar 7, 2017

zak-hassan commented Mar 8, 2017

csrwng commented Mar 8, 2017

zak-hassan commented Mar 8, 2017 • edited by stevekuznetsov

csrwng commented Mar 8, 2017

zak-hassan commented Mar 8, 2017

zak-hassan commented Mar 8, 2017

csrwng commented Mar 8, 2017

zak-hassan commented Mar 8, 2017 • edited by stevekuznetsov

csrwng commented Mar 8, 2017

csrwng commented Mar 8, 2017

csrwng commented Mar 13, 2017

csrwng commented Mar 13, 2017

mbechauf commented Mar 13, 2017

zak-hassan commented Mar 14, 2017

csrwng commented Mar 14, 2017

zak-hassan commented Mar 14, 2017

shveik commented Mar 16, 2017

csrwng commented Mar 16, 2017

shveik commented Mar 16, 2017

csrwng commented Mar 16, 2017

csrwng commented Mar 16, 2017

mbechauf commented Mar 16, 2017 via email

shveik commented Mar 16, 2017 • edited

cars10 commented Mar 21, 2017 • edited

stevekuznetsov commented Mar 23, 2017

soul2zimate commented Mar 31, 2017

csrwng commented Mar 31, 2017

mbn18 commented Apr 14, 2017

marioluan commented Apr 15, 2017

stevekuznetsov commented Apr 17, 2017

simbo1905 commented Apr 20, 2017 • edited

csrwng commented Apr 20, 2017

soltysh commented Apr 27, 2017

soltysh commented Apr 27, 2017

soltysh commented May 12, 2017

justinclayton commented Mar 7, 2017 •

edited

justinclayton commented Mar 7, 2017 •

edited

zak-hassan commented Mar 8, 2017 •

edited by stevekuznetsov

zak-hassan commented Mar 8, 2017 •

edited by stevekuznetsov

shveik commented Mar 16, 2017 •

edited

cars10 commented Mar 21, 2017 •

edited

simbo1905 commented Apr 20, 2017 •

edited