Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Latest package on Ubuntu trusty/14.04 (18.06.2~ce~3-0~ubuntu) is broken with runc #591

Open
olemartinorg opened this issue Feb 12, 2019 · 32 comments

Comments

@olemartinorg
Copy link

commented Feb 12, 2019

  • This is a bug report
  • This is a feature request
  • I searched existing issues before opening this one

Expected behavior

Starting containers will work

Actual behavior

OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:297: copying bootstrap data to pipe caused "write init-p: broken pipe"": unknow

Steps to reproduce the behavior

On Ubuntu 14.04, install the latest packages (18.06.2ce3-0~ubuntu), which was released yesterday it seems.

docker info (snipped):

runc version: a592beb5bc4c4092b1b1bac971afed27687340c5 (expected: 69663f0bd4b60df09991c08812a60108003fa340)
# docker-runc --version
runc version 1.0.0-rc5+dev.docker-18.06
commit: a592beb5bc4c4092b1b1bac971afed27687340c5
spec: 1.0.0

Seems to suggest that the docker-runc version shipped with the docker-ce package is incompatible.

@olemartinorg

This comment has been minimized.

Copy link
Author

commented Feb 12, 2019

Guessing this has something to do with CVE-2019-5736 and a rushed-out release.

@iay

This comment has been minimized.

Copy link

commented Feb 12, 2019

Same failure here. Upgraded to 18.06.2 using apt-get update / apt-get dist-upgrade as part of normal system maintenance, now failing to start any existing containers.

Same runc version: hashes in docker info.

Attempt to run a new container results in:

ce081455c4f0adc0034edaa2dcd8aaedf39181c8a052492b4ac9371d68d1561d
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 23\"": unknown.

Downgrading brings things back to operating condition:

apt-get install docker-ce=18.06.1~ce~3-0
@seemethere

This comment has been minimized.

Copy link
Member

commented Feb 12, 2019

What is your kernel version?

@iay

This comment has been minimized.

Copy link

commented Feb 12, 2019

# uname -a
Linux srv-ut01 3.13.0-165-generic #215-Ubuntu SMP Wed Jan 16 11:46:47 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

I think I'm right in saying that a new kernel came in during the same apt-get dist-upgrade invocation.

@olemartinorg

This comment has been minimized.

Copy link
Author

commented Feb 12, 2019

3.13.0-165-generic here as well

@cpuguy83

This comment has been minimized.

Copy link
Collaborator

commented Feb 12, 2019

jeancochrane pushed a commit to project-icp/bee-pollinator-app that referenced this issue Feb 12, 2019

Jean Cochrane
Pin Docker to 18.06.1 during provisioning
Docker CE 18.06.2 is incompatible with the Linux kernel 3.13.
Since Docker is easier to down/upgrade than the kernel version,
downgrade Docker to 18.06.1 during provisioning.

See:
  * https://docs.docker.com/engine/release-notes/#18092
  * docker/for-linux#591
@anitgandhi

This comment has been minimized.

Copy link

commented Feb 12, 2019

I'm observing the same thing on Linux kernel 3.13.0-164-generic and 3.13.0-165-generic.

I tried with this https://github.com/rancher/runc-cve , on 18.06.1 and it worked, to at-least have a patched runc.

apt install -y docker-ce=18.06.1~ce~3-0~ubuntu jq
# Since we need to pin to 18.06, we need to use a backport fix for CVE-2019-5736
/bin/mv /usr/bin/docker-runc /usr/bin/docker-runc.orig.$(date -Iseconds)
curl -LO https://github.com/rancher/runc-cve/releases/download/CVE-2019-5736-build2/runc-v18.06.1-amd64
/bin/mv runc-v18.06.1-amd64 /usr/bin/docker-runc
chmod +x /usr/bin/docker-runc
# test
docker run alpine echo OK
@thaJeztah

This comment has been minimized.

Copy link
Member

commented Feb 12, 2019

Kernel 3.13 does not have the features needed to mitigate the vulnerability (see https://docs.docker.com/engine/release-notes/#18092

Ubuntu 14.04 customers using a 3.13 kernel will need to upgrade to a supported Ubuntu 4.x kernel

There is a variation of the patch for 3.13 kernels, which will be included in a future update, but updating to a 4.x kernel might be the best solution.

@seemethere

This comment has been minimized.

Copy link
Member

commented Feb 12, 2019

@anitgandhi I'm not seeing the same results on a 3.13 kernels:

My run with the patched runc
root@vagrant-ubuntu-trusty-64:/home/vagrant/go/src/github.com/opencontainers/runc# apt install -y --force-yes docker-ce=18.06.1~ce~3-0~ubuntu
Reading package lists... Done
Building dependency tree
Reading state information... Done
Recommended packages:
  aufs-tools cgroupfs-mount cgroup-lite pigz
The following packages will be DOWNGRADED:
  docker-ce
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 93 not upgraded.
Need to get 39.7 MB of archives.
After this operation, 17.4 kB disk space will be freed.
Get:1 https://download.docker.com/linux/ubuntu/ trusty/edge docker-ce amd64 18.06.1~ce~3-0~ubuntu [39.7 MB]
Fetched 39.7 MB in 3s (13.1 MB/s)
dpkg: warning: downgrading docker-ce from 18.06.2~ce~3-0~ubuntu to 18.06.1~ce~3-0~ubuntu
(Reading database ... 90960 files and directories currently installed.)
Preparing to unpack .../docker-ce_18.06.1~ce~3-0~ubuntu_amd64.deb ...
docker stop/waiting
Unpacking docker-ce (18.06.1~ce~3-0~ubuntu) over (18.06.2~ce~3-0~ubuntu) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up docker-ce (18.06.1~ce~3-0~ubuntu) ...
docker start/running, process 18689
root@vagrant-ubuntu-trusty-64:/home/vagrant/go/src/github.com/opencontainers/runc# cd
root@vagrant-ubuntu-trusty-64:~# curl -LO https://github.com/rancher/runc-cve/releases/download/CVE-2019-5736-build2/runc-v18.06.1-amd64
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   610    0   610    0     0    875      0 --:--:-- --:--:-- --:--:--   875
100 7914k  100 7914k    0     0  3384k      0  0:00:02  0:00:02 --:--:-- 7489k
root@vagrant-ubuntu-trusty-64:~# /bin/mv runc-v18.06.1-amd64 /usr/bin/docker-runc
root@vagrant-ubuntu-trusty-64:~# chmod +x /usr/bin/docker-runc
root@vagrant-ubuntu-trusty-64:~# docker run alpine echo ok
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:297: copying bootstrap data to pipe caused \"write init-p: broken pipe\"": unknown.
root@vagrant-ubuntu-trusty-64:~# uname -a
Linux vagrant-ubuntu-trusty-64 3.13.0-155-generic #205-Ubuntu SMP Fri Aug 10 15:53:26 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
@anitgandhi

This comment has been minimized.

Copy link

commented Feb 12, 2019

@seemethere yeah I'm trying to recreate what I did and I don't know why but it's not working anymore :(

it's likely just what was already said, 3.13 just might not be supported at all. I'll keep digging and report back with what I find

@anitgandhi

This comment has been minimized.

Copy link

commented Feb 12, 2019

@seemethere @thaJeztah I'm confused on something, hoping you can help me understand.

It looks like the runc commit a592beb5bc4c4092b1b1bac971afed27687340c5 that 18.06.2 is shipping with is from this branch, which already contains this commit that references Ubuntu Trusty and the Linux 4.4 headers being problematic for this situation.

My interpretation was that that commit was added to provide 3.13 compatibility - is that accurate? Or is that completely different from the variation of the patch referenced above?

@seemethere

This comment has been minimized.

Copy link
Member

commented Feb 12, 2019

@anitgandhi That commit in particular was used to unblock our build processes since building for Trusty with the patch was not possible with the original patch even with linux-headers-virtual-generic-xenial installed. (at least that we found), cc @kolyshkin

@anitgandhi

This comment has been minimized.

Copy link

commented Feb 12, 2019

@seemethere Thank you, appreciate the quick explanation!

I'm not sure what I was doing above that worked, think I just got my wires/OSs crossed 😅

hectcastro added a commit to project-icp/bee-pollinator-app that referenced this issue Feb 13, 2019

Pin Docker to 18.06.1 during provisioning (#474)
Docker CE `18.06.2` is incompatible with the Linux kernel `3.13`. Since Docker is easier to down/upgrade than the kernel version, downgrade Docker to `18.06.1` during provisioning, and uninstall Docker on AMI builds once we're done provisioning.

See:
  * https://docs.docker.com/engine/release-notes/#18092
  * docker/for-linux#591
  * https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-5736
@RiRa12621

This comment has been minimized.

Copy link

commented Feb 13, 2019

seeing the same behaviour but tested a bit on it and what confuses me is that it all works fine with the plain binaries.

here's the run with the deb package installed:

root@dockertest001:~# apt-cache policy docker-ce |head -5
docker-ce:
  Installed: 18.06.2~ce~3-0~ubuntu
  Candidate: 18.06.2~ce~3-0~ubuntu
  Version table:
 *** 18.06.2~ce~3-0~ubuntu 0
root@dockertest001:~# docker --debug run alpine
DEBU[0000] [hijack] End of stdout
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:297: copying bootstrap data to pipe caused \"write init-p: broken pipe\"": unknown.
ERRO[0000] error waiting for container: context canceled

but when getting the static binaries and using those, everything works as expected:

wget https://download.docker.com/linux/static/stable/x86_64/docker-18.06.2-ce.tgz
     ls
     tar xzfv docker-18.06.2-ce.tgz
     ls /usr/bin/
     ls /usr/bin/docker
     ls /usr/bin/docker/
     ls /usr/bin/docker
     mkdir docker_binary_backup
     mv /usr/bin/docker* docker_binary_backup/
     ls docker_binary_backup/
     sudo cp docker/* /usr/bin/
     sudo dockerd &
root@dockertest001:~# docker --debug run alpine
INFO[0008] shim docker-containerd-shim started           address="/containerd-shim/moby/fac6716ce99d344ee15b0b7baf211ee2ed1e2fe37dc3d2df38339df860e0189c/shim.sock" debug=false pid=1969
INFO[0008] shim reaped                                   id=fac6716ce99d344ee15b0b7baf211ee2ed1e2fe37dc3d2df38339df860e0189c
INFO[2019-02-13T19:14:09.212649013Z] ignoring event                                module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
DEBU[0000] [hijack] End of stdout

Kernel:

root@dockertest001:~# uname -a
Linux dockertest001 3.13.0-157-generic #207-Ubuntu SMP Mon Aug 20 16:44:59 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
@kolyshkin

This comment has been minimized.

Copy link

commented Feb 13, 2019

As pointed out above and in release notes, the security fix for runc (CVE-2019-5736) requires either kernel >= 3.17, or a backport of kernel commits adding memfd_create() syscall (which, for example, was done by Debian in their 3.16-based kernel for Jessie and RHEL for their 3.10-based kernel for RHEL7, but not for Ubuntu Trusty 3.13-based kernel).

So, for Ubuntu Trusty, you need to

  1. install 4.4 lts kernel:

sudo apt-get install --install-recommends linux-generic-lts-xenial

  1. reboot into the newly installed kernel.

It would be great for docker-ce deb package for trusty to have that dependency...

@anitgandhi

This comment has been minimized.

Copy link

commented Feb 13, 2019

@RiRa12621 I downloaded the static build tar like you described, and if you check the docker/docker-runc --version it's this one which is the same as what 18.06.1 had, aka the one without the CVE fix, that's why it "worked".

@RiRa12621

This comment has been minimized.

Copy link

commented Feb 14, 2019

@kolyshkin indeed true. I think that should be clearly documented then. Because if I read the change logs correctly, until now it only states that you're required the kernel upgrade for the 18.09.XX versions.
@anitgandhi thanks for pointing that out.
The situation is now:
different binaries in the tar and in the deb for the same version of docker-ce.
The fix for CVE-2019-5736 requires the newer Kernel.
So imho that means that for one the binaries should actually be the same both in the archive and the deb package and furthermore it should be added to the documentation that you are required to have the newer Kernel installed in order for docker to work.

@iay

This comment has been minimized.

Copy link

commented Feb 14, 2019

Confirming that updating to the 4.4 kernel using sudo apt-get install --install-recommends linux-generic-lts-xenial makes my system compatible with the 18.06.2 release.

I'd also like to second the comment that the docker-ce package should make that dependency explicit. It's really a bad look to have a patch release brick a system like this, and pointing at the release notes doesn't help with that.

@mattnworb

This comment has been minimized.

Copy link

commented Feb 14, 2019

@thaJeztah

There is a variation of the patch for 3.13 kernels, which will be included in a future update, but updating to a 4.x kernel might be the best solution.

is there any timeframe for the variation of the patch for 3.13 kernels?

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Feb 14, 2019

We're trying to get it available as soon as possible; the tricky bit with the alternative approach is that it's not as battle-tested as the original patch, and thus for 4.x kernels may not be ideal. If we provide 14.04 packages that work on 3.13 kernels, they would still have to work for the 4.x kernels as well, so we're looking at options to switch at runtime between "variation A/B".

If possible, I would recommend upgrading the kernel, also with in mind that ubuntu 14.04 will reach EOL in April (https://wiki.ubuntu.com/Releases)

@tao12345666333

This comment has been minimized.

Copy link

commented Feb 15, 2019

The same issue using docker 18.09.2

[root@gpu80 ~]# docker run --rm my-registry/devops/ubuntu:gpu-cuda8 nvidia-smi      
docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container
process caused "process_linux.go:293: copying bootstrap data to pipe caused \"write init-p: broken pipe\"": unknown.

[root@gpu80 ~]# uname -a
Linux gpu80 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@gpu80 ~]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
[root@gpu80 ~]# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core)

and runc version

root@gpu80 ~]# runc --version
runc version 1.0.0-rc6+dev
commit: 09c8266bf2fcf9519a651b04ae54c967b9ab86ec
spec: 1.0.1-dev

The dockerd debug log

Feb 15 14:37:29 gpu80 dockerd[29799]: time="2019-02-15T14:37:29.155174304+08:00" level=info msg="API listen on /var/run/docker.sock"                                                                                
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.321063149+08:00" level=debug msg="Calling GET /_ping"                                                                                               
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.322163414+08:00" level=debug msg="Calling POST /v1.39/containers/create"                                                                            
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.322489707+08:00" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":[\"nvidia-smi\"],\"Domainname\":\"\",\"Entrypoint\":null,\"Env\":[],\"HostConfig\":{\"AutoRemove\":true,\"Binds\":null,\"BlkioDeviceReadBps\":null,\"BlkioDeviceReadIOps\":null,\"BlkioDeviceWriteBps\":null,\"BlkioDeviceWriteIOps\":null,\"BlkioWeight\":0,\"BlkioWeightDevice\":[],\"CapAdd\":null,\"CapDrop\":null,\"Cgroup\":\"\",\"CgroupParent\":\"\",\"ConsoleSize\":[0,0],\"ContainerIDFile\":\"\",\"CpuCount\":0,\"CpuPercent\":0,\"CpuPeriod\":0,\"CpuQuota\":0,\"CpuRealtimePeriod\":0,\"CpuRealtimeRuntime\":0,\"CpuShares\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"DeviceCgroupRules\":null,\"Devices\":[],\"DiskQuota\":0,\"Dns\":[],\"DnsOptions\":[],\"DnsSearch\":[],\"ExtraHosts\":null,\"GroupAdd\":null,\"IOMaximumBandwidth\":0,\"IOMaximumIOps\":0,\"IpcMode\":\"\",\"Isolation\":\"\",\"KernelMemory\":0,\"Links\":null,\"LogConfig\":{\"Config\":{},\"Type\":\"\"},\"MaskedPaths\":null,\"Memory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwappiness\":-1,\"NanoCpus\":0,\"NetworkMode\":\"default\",\"OomKillDisable\":false,\"OomScoreAdj\":0,\"PidMode\":\"\",\"PidsLimit\":0,\"PortBindings\":{},\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyPaths\":null,\"ReadonlyRootfs\":false,\"RestartPolicy\":{\"MaximumRetryCount\":0,\"Name\":\"no\"},\"SecurityOpt\":null,\"ShmSize\":0,\"UTSMode\":\"\",\"Ulimits\":null,\"UsernsMode\":\"\",\"VolumeDriver\":\"\",\"VolumesFrom\":null},\"Hostname\":\"\",\"Image\":\"harbor-registry.inner.youdao.com/devops/ubuntu:gpu-cuda8\",\"Labels\":{},\"NetworkingConfig\":{\"EndpointsConfig\":{}},\"OnBuild\":null,\"OpenStdin\":false,\"StdinOnce\":false,\"Tty\":false,\"User\":\"\",\"Volumes\":{},\"WorkingDir\":\"\"}"                                                                       
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.353272927+08:00" level=debug msg="container mounted via layerStore: &{/disk1/docker/overlay2/530aa7cb6b51ff868ab7cddbca07b21b36ebc6105a19ae60e4c69975180246df/merged 0x7f81f77fb8c0 0x7f81f77fb8c0}"
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.381727268+08:00" level=debug msg="Calling POST /v1.39/containers/65387bf2d0c000a0f409b4e05fd4b102832beb5b26d6178f811839fe6eb5c1d5/attach?stderr=1&stdout=1&stream=1"
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.381911380+08:00" level=debug msg="attach: stderr: begin"                                                                                            
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.381920462+08:00" level=debug msg="attach: stdout: begin"                                                                                            
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.382290770+08:00" level=debug msg="Calling POST /v1.39/containers/65387bf2d0c000a0f409b4e05fd4b102832beb5b26d6178f811839fe6eb5c1d5/wait?condition=removed"
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.382870043+08:00" level=debug msg="Calling POST /v1.39/containers/65387bf2d0c000a0f409b4e05fd4b102832beb5b26d6178f811839fe6eb5c1d5/start"            
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.384056488+08:00" level=debug msg="container mounted via layerStore: &{/disk1/docker/overlay2/530aa7cb6b51ff868ab7cddbca07b21b36ebc6105a19ae60e4c69975180246df/merged 0x7f81f77fb8c0 0x7f81f77fb8c0}"
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.384573999+08:00" level=debug msg="Assigning addresses for endpoint compassionate_ellis's interface on network bridge"                               
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.384611698+08:00" level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[])"                                                         
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.384654141+08:00" level=debug msg="Request address PoolID:172.17.0.0/16 App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0 Serial:false PrefAddress:<nil> "                                                                                               
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.387915759+08:00" level=debug msg="Assigning addresses for endpoint compassionate_ellis's interface on network bridge"                               
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.450473182+08:00" level=debug msg="Programming external connectivity on endpoint compassionate_ellis (63da3baf1b7922c54646bdd92f30d45b2d484c1d168e5a70a552b1ccfcf95804)"
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.468455702+08:00" level=debug msg="EnableService 65387bf2d0c000a0f409b4e05fd4b102832beb5b26d6178f811839fe6eb5c1d5 START"                             
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.468481048+08:00" level=debug msg="EnableService 65387bf2d0c000a0f409b4e05fd4b102832beb5b26d6178f811839fe6eb5c1d5 DONE"                              
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.470246259+08:00" level=debug msg="bundle dir created" bundle=/var/run/docker/containerd/65387bf2d0c000a0f409b4e05fd4b102832beb5b26d6178f811839fe6eb5c1d5 module=libcontainerd namespace=moby root=/disk1/docker/overlay2/530aa7cb6b51ff868ab7cddbca07b21b36ebc6105a19ae60e4c69975180246df/merged                                                                         
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.528203191+08:00" level=error msg="stream copy error: reading from a closed fifo"                                                                    
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.528241422+08:00" level=error msg="stream copy error: reading from a closed fifo"                                                                    
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.528283256+08:00" level=debug msg="attach: stdout: end"                                                                                              
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.528287304+08:00" level=debug msg="attach: stderr: end"                                                                                              
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.528317143+08:00" level=debug msg="attach done"                                                                                                      
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.533285071+08:00" level=debug msg="Revoking external connectivity on endpoint compassionate_ellis (63da3baf1b7922c54646bdd92f30d45b2d484c1d168e5a70a552b1ccfcf95804)"
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.535663522+08:00" level=debug msg="DeleteConntrackEntries purged ipv4:0, ipv6:0"                                                                     
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.603512002+08:00" level=debug msg="Releasing addresses for endpoint compassionate_ellis's interface on network bridge"                               
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.603551653+08:00" level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)"                                                           
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.603584533+08:00" level=debug msg="Released address PoolID:LocalDefault/172.17.0.0/16, Address:172.17.0.2 Sequence:App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65532, Sequence: (0xe0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:3"                                                                                    
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.606017590+08:00" level=error msg="65387bf2d0c000a0f409b4e05fd4b102832beb5b26d6178f811839fe6eb5c1d5 cleanup: failed to delete container from containerd: no such container"
Feb 15 14:38:00 gpu80 dockerd[29799]: time="2019-02-15T14:38:00.609233675+08:00" level=error msg="Handler for POST /v1.39/containers/65387bf2d0c000a0f409b4e05fd4b102832beb5b26d6178f811839fe6eb5c1d5/start returned
error: OCI runtime create failed: container_linux.go:344: starting container process caused \"process_linux.go:293: copying bootstrap data to pipe caused \\\"write init-p: broken pipe\\\"\": unknown"

Maybe the CentOS 7's 3.10.0-327 kernel is too old.

And upgrade to 3.10.0-693.el7.x86_64 it works fine.

@albertoal

This comment has been minimized.

Copy link

commented Feb 20, 2019

Thanks @kolyshkin / @iay , upgrading the Kernel using the command sudo apt-get install --install-recommends linux-generic-lts-xenial also makes my Ubuntu 14.04 LTS - Trusty Tahr server compatible with 18.06.3~ce~3-0~ubuntu.

If anyone is wondering where this command comes from, it's the one reccomended by Ubuntu for Kernel upgrades - see https://wiki.ubuntu.com/Kernel/LTSEnablementStack . I doubled-checked as I got confused by the fact that my Ubuntu release is Trusty and not Xenial but that is how the HWE works.

@wy193777

This comment has been minimized.

Copy link

commented Feb 21, 2019

sudo apt-get install --install-recommends linux-generic-lts-xenial

This command really helps. Thanks @albertoal !

@ISHITADG

This comment has been minimized.

Copy link

commented Feb 22, 2019

When running this command to upgrade the Kernel, my installation gets stuck at :

Unable to find an initial ram disk that I know how to handle.
Will not try to make an initrd.

I also tried uninstalling and reinstalling initramfs-tools

@ISHITADG

This comment has been minimized.

Copy link

commented Feb 22, 2019

Can I install the previous version of docker-ce just before this release update that used to work fine with Ubuntu 14.04 kernel 3.13 ???

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Feb 22, 2019

Can I install the previous version of docker-ce just before this release update that used to work fine with Ubuntu 14.04 kernel 3.13 ???

You can, but I would not recommend doing so, as the CVE that's being addressed by this update is critical (allows a container escape, so processes inside the container can get root access on your host).

lvnilesh added a commit to beacloudgenius/ntier that referenced this issue Mar 9, 2019

lvnilesh added a commit to beacloudgenius/ntier that referenced this issue Mar 9, 2019

@xhocquet

This comment has been minimized.

Copy link

commented Apr 15, 2019

@thaJeztah Hey, sorry for the ping, but was wondering if there's any tracking of the work being done for this patch and if it is actually being worked on? I know for a fact that many developers in this thread and otherwise have had to temporarily pin their Docker versions to 18.06.3~ce~3-0~ubuntu, which leaves the vulnerability in place.

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Apr 15, 2019

The 18.06 release reached end of life, so no 18.06.4 will be released; if you're still on Ubuntu 14.04 with the original 3.13 kernel, you can upgrade to the ubuntu supported 4.x kernels.

Note however that Ubuntu 14.04 reached EOL at the end of this month, so I would highly recommend upgrading to a currently supported Ubuntu LTS version (Ubuntu 18.04 LTS or 16.04 LTS), as 14.04 won't be receiving updates or security fixes

@gangh

This comment has been minimized.

Copy link

commented Apr 29, 2019

apt install -y docker-ce=18.06.1~ce~3-0~ubuntu jq

helpful 👍
it is useful or my Ubuntu 14.04(3.13.0-105-generic) , thank you very much

@play1921

This comment has been minimized.

Copy link

commented May 15, 2019

well... I downgrade to docker-ce 18.06.1ce3-0-ubuntu
it seems work.
but yeah, we have to upgrade kernel.

@nigerninja

This comment has been minimized.

Copy link

commented May 23, 2019

As pointed out above and in release notes, the security fix for runc (CVE-2019-5736) requires either kernel >= 3.17, or a backport of kernel commits adding memfd_create() syscall (which, for example, was done by Debian in their 3.16-based kernel for Jessie and RHEL for their 3.10-based kernel for RHEL7, but not for Ubuntu Trusty 3.13-based kernel).

So, for Ubuntu Trusty, you need to

  1. install 4.4 lts kernel:

sudo apt-get install --install-recommends linux-generic-lts-xenial

  1. reboot into the newly installed kernel.

It would be great for docker-ce deb package for trusty to have that dependency...

Works for me.
Thanks.

@charlie-charlie

This comment has been minimized.

Copy link

commented Jun 17, 2019

yeah, thanks for sharing. after running apt-get install --install-recommends linux-generic-lts-xenial, I can run my docker now. thx again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.