Validate docker v1.12 #28698

vishh · 2016-07-08T18:08:25Z

Multiple RCs have been released for docker v1.12 already. We should start testing docker v1.12 soon. @Random-Liu is our automated testing suite active? Are we already validating docker v1.12?

Random-Liu · 2016-07-08T18:23:30Z

@vishh Yeah, we already have a daily running docker validation test running e2e test against newest docker release with kubernetes head.
Please check continuous-docker-validation-gci. It has tested docker 1.12-rc2 and 1.12-rc3, and they are all green now. :)

And thanks to @wonderfly

Random-Liu · 2016-07-08T18:24:12Z

With #28213, we'll have another docker validation test against node e2e. PTAL~ :)

vishh · 2016-07-08T21:41:02Z

@Random-Liu I'm assigning this issue to you. You can close it whenever you think that we have enabled adequate testing.
Feel free to re-assign back to me if you'd not prefer owning this.

Random-Liu · 2016-07-12T02:00:17Z

FYI, The docker validation test starts failing from Jul 9.

wonderfly · 2016-07-12T02:06:17Z

08:48:43  - Invalid value for field 'resource.disks[0].initializeParams.sourceImage': 'https://www.googleapis.com/compute/v1/projects/container-vm-image-staging/global/images/gci-dev-53-8530-6-0'. The referenced image resource cannot be found.

Looks like an issue related to the container-vm-image-staging project. I'll take a look.

Random-Liu · 2016-07-12T02:07:10Z

@wonderfly Thanks a lot~ :)

wonderfly · 2016-07-12T02:41:45Z

The image is back there but I think the test will still fail due to kubernetes/test-infra#266 (comment)

wonderfly · 2016-07-12T17:03:15Z

The continuous e2e tests are back to green now.

Random-Liu · 2016-07-13T18:49:44Z

FYI, docker 1.12-rc4 came out yesterday. Our first run of validation test is red because some system pods in kube-system are Pending with image pulling error.

Error pulling image (1.3) from gcr.io/google_containers/kibana, Get https://gcr.io/v1/images/88ad1cfaa37a08b7ab19b3e4d150b3dd4d027dd1f38464b80dc178ac38c67ccd/ancestry: dial tcp 74.125.132.82:443: i/o timeout "

Random-Liu · 2016-07-13T23:57:10Z

This seems to be a image pulling flake. I started a gci-canary with docker 1.12-rc4 myself and pulled the image gcr.io/google_containers/kibana:1.3. Everything is fine.
Let's wait for the result today.

wonderfly · 2016-07-14T00:00:36Z

Or, you can log onto Jenkins and trigger a manual build, if you want the result sooner.

Random-Liu · 2016-07-14T00:04:14Z

@wonderfly Done! Don't know I have the permission to do that, Thanks a lot!

Random-Liu · 2016-07-23T02:12:14Z

FYI, we've got the first green node e2e build with:

DOCKER_VERSION=1.12.0-rc4
GCI_IMAGE=gci-test-54-8618-0-0

vishh · 2016-07-25T17:46:22Z

@Random-Liu What about testing with old container VM?

Random-Liu · 2016-07-25T18:00:13Z

@vishh To test old container VM, we need to a good way to install newest docker release on the containerVM. Maybe download and unpack binary directly.

yujuhong · 2016-08-01T16:59:07Z

The memory leakage problem issue in v1.11: moby/moby#24420
I haven't checked if this has been fixed on v1.12 or not.

Random-Liu · 2016-08-01T17:12:57Z

The memory leakage problem issue in v1.11: moby/moby#24420
I haven't checked if this has been fixed on v1.12 or not.

It seems not. The fix is just merged today
moby/moby#25283 (comment)

Random-Liu · 2016-08-12T21:54:48Z

Just FYI, till this morning all continuous docker validation node e2e tests are green.
However, this noon, we've got a red build.
During the test, all of a sudden, all docker operations start to timeout:

operation timeout: context deadline exceeded

In kernel log and docker log, no clue is found around that time:
Kernel log:

Aug 12 18:51:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: device veth74f4a99e left promiscuous mode
Aug 12 18:51:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(veth74f4a99e) entered disabled state
Aug 12 18:51:04 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 3(veth61b3ed9a) entered forwarding state
Aug 12 18:53:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Aug 12 18:53:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Aug 12 18:53:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: device vetha78ce6be entered promiscuous mode
Aug 12 18:53:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vetha78ce6be) entered forwarding state
Aug 12 18:53:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vetha78ce6be) entered forwarding state
Aug 12 18:53:17 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vetha78ce6be) entered forwarding state
Aug 12 18:55:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vetha78ce6be) entered disabled state
Aug 12 18:55:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: device vetha78ce6be left promiscuous mode
Aug 12 18:55:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vetha78ce6be) entered disabled state
Aug 12 19:00:51 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Aug 12 19:00:51 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Aug 12 19:00:51 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: device vetha2a0c962 entered promiscuous mode
Aug 12 19:00:51 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vetha2a0c962) entered forwarding state
Aug 12 19:00:51 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vetha2a0c962) entered forwarding state
Aug 12 19:01:06 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vetha2a0c962) entered forwarding state
Aug 12 19:02:51 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vetha2a0c962) entered disabled state
Aug 12 19:02:51 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: device vetha2a0c962 left promiscuous mode
Aug 12 19:02:51 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vetha2a0c962) entered disabled state
Aug 12 19:04:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Aug 12 19:04:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Aug 12 19:04:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: device vethb8115292 entered promiscuous mode
Aug 12 19:04:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vethb8115292) entered forwarding state
Aug 12 19:04:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vethb8115292) entered forwarding state
Aug 12 19:05:07 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vethb8115292) entered forwarding state
Aug 12 19:06:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vethb8115292) entered disabled state
Aug 12 19:06:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: device vethb8115292 left promiscuous mode
Aug 12 19:06:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(vethb8115292) entered disabled state
Aug 12 19:08:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Aug 12 19:08:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Aug 12 19:08:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: device veth45b06f24 entered promiscuous mode
Aug 12 19:08:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(veth45b06f24) entered forwarding state
Aug 12 19:08:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(veth45b06f24) entered forwarding state
Aug 12 19:09:07 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(veth45b06f24) entered forwarding state
Aug 12 19:10:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(veth45b06f24) entered disabled state
Aug 12 19:10:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: device veth45b06f24 left promiscuous mode
Aug 12 19:10:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: cbr0: port 2(veth45b06f24) entered disabled state
Aug 12 19:12:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Docker log:

Aug 12 18:50:50 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 docker[1056]: time="2016-08-12T18:50:50.787615193Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead."
Aug 12 18:51:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 docker[1056]: time="2016-08-12T18:51:02.555813491Z" level=info msg="Container e7ba7b2e63b8480991b9135b46d0d355b874f6f888f4130c4209cf6fab5f4858 failed to exit within 30 seconds of signal 15 - using the force"
Aug 12 18:51:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 docker[1056]: time="2016-08-12T18:51:02.557375343Z" level=info msg="Container c4743572c0f90a05addda012307aa9540f4415908f5633400f4a4418c8631479 failed to exit within 30 seconds of signal 15 - using the force"
Aug 12 18:51:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 docker[1056]: time="2016-08-12T18:51:02.560527484Z" level=info msg="Container f46b1e3f77df6d400d70df969b1f36029337618786eb39685322878504a7025f failed to exit within 30 seconds of signal 15 - using the force"
Aug 12 18:51:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 docker[1056]: time="2016-08-12T18:51:02.560841000Z" level=info msg="Container 3a16cd7ac6165a605f646d9a8e71337584a7927640664d298052290a24715958 failed to exit within 30 seconds of signal 15 - using the force"
Aug 12 18:51:02 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 docker[1056]: time="2016-08-12T18:51:02.561153367Z" level=info msg="Container 301ed4733b9a523f95c8d52cf93a081439fc2f11b94dcd361c6e595ca4cd4f67 failed to exit within 30 seconds of signal 15 - using the force"
Aug 12 18:52:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 docker[1056]: time="2016-08-12T18:52:52.084698223Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead."
Aug 12 19:00:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 docker[1056]: time="2016-08-12T19:00:52.082521219Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead."
Aug 12 19:04:52 tmp-node-e2e-61c9844e-gci-test-54-8699-0-0 docker[1056]: time="2016-08-12T19:04:52.087175680Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead."

yujuhong · 2016-08-12T22:01:32Z

@Random-Liu what's the link to the build?

Random-Liu · 2016-08-12T22:24:23Z

Here is the link: https://console.cloud.google.com/storage/browser/kubernetes-jenkins/logs/continuous-node-e2e-docker-validation-gci/94

Random-Liu · 2016-09-03T00:22:04Z

v1.11 results copied from this comment

Introduction
I benchmarked docker v1.12 using the docker micro benchmark tool.

Benchmark Environment

Cloud Provider: GCE
VM Instance: n1-standard-2 (2 vCPUs, 7.5 GB memory)
OS: Debian GNU/Linux 8.3 (jessie)
Docker version:

Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:02:53 2016
 OS/Arch:      linux/amd64
Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:02:53 2016
 OS/Arch:      linux/amd64

1. Benchmark list/inspect with varying number of containers

Using benchmark.sh -c

Latency
- Docker 1.11.1
- Docker 1.12.1
CPU Usage
- Docker 1.11.1
- Docker 1.12.1

2. Benchmark list/inspect with varying operation intervals

Using benchmark.sh -i

ps [all=true] Latency
- Docker 1.11.1
- Docker 1.12.1
ps [all=false] Latency
- Docker 1.11.1
- Docker 1.12.1
inspect Latency
- Docker 1.11.1
- Docker 1.12.1
CPU Usage
- Docker 1.11.1
- Docker 1.12.1

3. Benchmark list/inspect with varying number of goroutines

Using benchmark.sh -r

ps [all=true] Latency
- Docker 1.11.1
- Docker 1.12.1
inspect Latency
- Docker 1.11.1
- Docker 1.12.1
CPU Usage
- Docker 1.11.1
- Docker 1.12.1

Conclusion
The cpu looks so spiky that I'm not sure whether the benchmark tool is working well. There is no much performance difference between docker 1.11.1 and docker 1.12.1. The only thing I can observe is that the list container latency is higher in docker 1.12.1, looks like 1/3 higher.

dlorenc · 2016-09-09T01:56:05Z

I see this was closed, does that mean v1.4 has been validated against docker v1.12?

Random-Liu · 2016-09-09T02:34:59Z

@dlorenc In v1.4 we introduced automated docker validation test. We use node e2e to validate newest docker with Kubernetes HEAD. Currently, no critical issues found at least with this test.
http://k8s-testgrid.appspot.com/docker

vishh · 2016-09-09T21:29:57Z

@dlorenc Officially, k8s v1.4 still recommends using docker v1.11 and that
is what GCE and GKE clusters will use.

On Thu, Sep 8, 2016 at 7:35 PM, Lantao Liu notifications@github.com wrote:

@dlorenc https://github.com/dlorenc In v1.4 we introduced automated
docker validation test. We use node e2e to validate newest docker with
Kubernetes HEAD. Currently, no critical issues found at least with this
test.
http://k8s-testgrid.appspot.com/docker

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#28698 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKLbFZaWlWIzkh15axSVGj6qSvDKTks5qoMYGgaJpZM4JIQ6i
.

dlorenc · 2016-09-10T00:22:00Z

@vishh Thanks! Is there anywhere that's written down that I can follow along?

dchen1107 · 2016-09-10T00:28:02Z

@dlorenc, each kubernetes release, we document the validated & supported runtime (docker & rkt) along with the known issues. You can find those information in release notes.

tristanz · 2016-09-11T06:51:24Z

@dchen1107 It would be nice to put this somewhere more prominently. It's pretty hard to track down in changelog and it seems to be OS specific. I'm not sure everybody realizes how important validation is given docker's instability.

@timstclair

Automatic merge from submit-queue Docker Micro Benchmark: Add dockerd and containerd support. @timstclair Can you help me review this? I remember you made similar change locally before. I used this to get the result kubernetes/kubernetes#28698 (comment), but forgot to submit the PR at that time.

yujuhong · 2017-03-02T00:01:50Z

Known issue in 1.11.2 that needs further validation in 1.12

Docker leaking processes: Processes may be leaked when docker is killed repeatedly in a short time frame #41450

Know issues about 1.12

Docker reporting incorrect exit codes for containers: Docker reports incorrect exit code for containers if the docker service was restarted #41516
Shared PID namespace doesn't work properly: Propose rollout for Docker shared PID namespace community#207 (comment)

animeshsingh · 2017-05-24T22:58:04Z

So where does it stand now? Do we have Docker 1.12 working in Minikube?

vishh added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Jul 8, 2016

vishh added this to the v1.4 milestone Jul 8, 2016

vishh assigned Random-Liu Jul 8, 2016

yujuhong mentioned this issue Jul 21, 2016

Re-writing of the resolv.conf file generated by docker #29378

Merged

Random-Liu mentioned this issue Jul 28, 2016

[WIP] Trying PID Namespace sharing from Docker 1.12 release candidates #29677

Closed

r2d4 mentioned this issue Aug 1, 2016

Docker v1.12.0 kubernetes/minikube#435

Closed

thaJeztah mentioned this issue Aug 1, 2016

Flannel install command hangs moby/moby#25301

Closed

icecrime mentioned this issue Sep 7, 2016

Backport fix on debain to 1.11.2 moby/moby#26359

Closed

dchen1107 closed this as completed Sep 8, 2016

dlorenc mentioned this issue Sep 12, 2016

upstep docker to version 1.12.1 (default ubuntu shipped) kubernetes/minikube#525

Closed

dchen1107 mentioned this issue Sep 24, 2016

draft release notes for kubernetes v1.4.0 #33410

Merged

Random-Liu mentioned this issue Oct 5, 2016

Docker Micro Benchmark: Add dockerd and containerd support. kubernetes-retired/contrib#1840

Merged

resouer mentioned this issue Jan 19, 2017

Docker Vulnerability CVE-2016-9962 #40061

Closed

supereagle mentioned this issue Feb 9, 2017

Docker release tracking supereagle/experiences#5

Open

r2d4 mentioned this issue May 2, 2017

Upgrade to docker 1.12 kubernetes/minikube#1439

Closed

aclisp mentioned this issue Jun 9, 2017

K8s和Docker的版本兼容纪要 aclisp/voyage#6

Open

yguo0905 mentioned this issue Aug 22, 2017

Validating docker 1.13.x and 17.03 #42926

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Validate docker v1.12 #28698

Validate docker v1.12 #28698

vishh commented Jul 8, 2016

Random-Liu commented Jul 8, 2016 •

edited

Loading

Random-Liu commented Jul 8, 2016 •

edited

Loading

vishh commented Jul 8, 2016 •

edited

Loading

Random-Liu commented Jul 12, 2016 •

edited

Loading

wonderfly commented Jul 12, 2016

Random-Liu commented Jul 12, 2016

wonderfly commented Jul 12, 2016

wonderfly commented Jul 12, 2016

Random-Liu commented Jul 13, 2016

Random-Liu commented Jul 13, 2016

wonderfly commented Jul 14, 2016

Random-Liu commented Jul 14, 2016

Random-Liu commented Jul 23, 2016 •

edited

Loading

vishh commented Jul 25, 2016

Random-Liu commented Jul 25, 2016

yujuhong commented Aug 1, 2016 •

edited

Loading

Random-Liu commented Aug 1, 2016

Random-Liu commented Aug 12, 2016

yujuhong commented Aug 12, 2016

Random-Liu commented Aug 12, 2016

Random-Liu commented Sep 3, 2016 •

edited

Loading

dlorenc commented Sep 9, 2016

Random-Liu commented Sep 9, 2016

vishh commented Sep 9, 2016

dlorenc commented Sep 10, 2016

dchen1107 commented Sep 10, 2016

tristanz commented Sep 11, 2016

yujuhong commented Mar 2, 2017

animeshsingh commented May 24, 2017

Validate docker v1.12 #28698

Validate docker v1.12 #28698

Comments

vishh commented Jul 8, 2016

Random-Liu commented Jul 8, 2016 • edited Loading

Random-Liu commented Jul 8, 2016 • edited Loading

vishh commented Jul 8, 2016 • edited Loading

Random-Liu commented Jul 12, 2016 • edited Loading

wonderfly commented Jul 12, 2016

Random-Liu commented Jul 12, 2016

wonderfly commented Jul 12, 2016

wonderfly commented Jul 12, 2016

Random-Liu commented Jul 13, 2016

Random-Liu commented Jul 13, 2016

wonderfly commented Jul 14, 2016

Random-Liu commented Jul 14, 2016

Random-Liu commented Jul 23, 2016 • edited Loading

vishh commented Jul 25, 2016

Random-Liu commented Jul 25, 2016

yujuhong commented Aug 1, 2016 • edited Loading

Random-Liu commented Aug 1, 2016

Random-Liu commented Aug 12, 2016

yujuhong commented Aug 12, 2016

Random-Liu commented Aug 12, 2016

Random-Liu commented Sep 3, 2016 • edited Loading

1. Benchmark list/inspect with varying number of containers

2. Benchmark list/inspect with varying operation intervals

3. Benchmark list/inspect with varying number of goroutines

dlorenc commented Sep 9, 2016

Random-Liu commented Sep 9, 2016

vishh commented Sep 9, 2016

dlorenc commented Sep 10, 2016

dchen1107 commented Sep 10, 2016

tristanz commented Sep 11, 2016

yujuhong commented Mar 2, 2017

animeshsingh commented May 24, 2017

Random-Liu commented Jul 8, 2016 •

edited

Loading

Random-Liu commented Jul 8, 2016 •

edited

Loading

vishh commented Jul 8, 2016 •

edited

Loading

Random-Liu commented Jul 12, 2016 •

edited

Loading

Random-Liu commented Jul 23, 2016 •

edited

Loading

yujuhong commented Aug 1, 2016 •

edited

Loading

Random-Liu commented Sep 3, 2016 •

edited

Loading