GCE/GKE jobs need newer Docker versions for multi-arch images

[dims]

Thanks @cjwagner we now have multi-arch images for all the Dockerfiles under test/images directory in main k/k repository (see gcr.io/kubernetes-e2e-test-images)

kubernetes/kubernetes#66608 aims to kick the tires of these images.

However looking at pull-kubernetes-e2e-gce and pull-kubernetes-e2e-gke jobs, we seem to have 17.03 version of docker which does not support multi-arch images

Seems we need at least moby/moby#35772 which seems to be Docker 18.01 per docker-archive/docker-ce@85c031e

Do we have cos-stable-* that support newer docker versions?

[dims]

related to kubernetes/kubernetes#38067

[dims]

cc @mkumatag @BenTheElder @cjwagner

[mkumatag]

@dims we don't have to worry about updating docker cli(client), because docker client supports manifest images from very long time. Until and unless if they are pushing manifests, we can still use the older version you have.

[dims]

@mkumatag not talking about "docker manifest" commands, the docker engine itself needs to be updated. See the links above :)

[mkumatag]

moby/moby#35772 is just a test scripts update, can you point me to the error what we are facing?

[dims]

Take a look at https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/66608/pull-kubernetes-e2e-gce/45548/artifacts/e2e-45548-674b9-minion-group-l2qs/docker.log, the docker version is as follows:

time="2018-07-25T14:18:56.779260481Z" level=info msg="Docker daemon" commit=f5ec1e2 graphdriver=overlay2 version=17.03.2-ce

search for gcr.io/kubernetes-e2e-test-images/mounttest:1.0 in the log file. The image is clearly present in the registry but is not getting pulled. (checked using manifest-tool and by looking at the url https://console.cloud.google.com/gcr/images/kubernetes-e2e-test-images/GLOBAL/mounttest?gcrImageListsize=50 )

moby#35772 is the first commit that enables fat manifest support and it's clearly not present in 17.03

[mkumatag]

same image I could able to pull on my machine which has same version:

root@some-host:~# docker pull gcr.io/kubernetes-e2e-test-images/mounttest:1.0
1.0: Pulling from kubernetes-e2e-test-images/mounttest
85b77b13b8c4: Pull complete
Digest: sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2
Status: Downloaded newer image for gcr.io/kubernetes-e2e-test-images/mounttest:1.0
root@some-host:~# docker version
Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 03:35:14 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.2-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 03:35:14 2017
 OS/Arch:      linux/amd64
 Experimental: false
root@some-host:~#

[mkumatag]

in that PR https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/66608/pull-kubernetes-e2e-gce/45548/, I see only 5 tests are failing now and they are because of we didn't push gcr.io/k8s-authenticated-test/serve-hostname:1.1 image which needed for testing ReplicaSet should serve a basic image on each replica with a private image testcase

[dims]

Aha! let me revert just that one

[dims]

/close