-
Notifications
You must be signed in to change notification settings - Fork 84
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docker-ce-18.09.2 and/or containerd.io-1.2.2 prevent containers from running #595
Comments
Note that CentOS uses a rolling release model, which means that older versions (including their kernels) reach EOL if a newer version is released. Kernel 3.10.0-229 is a really old version of the CentOS kernel, so definitely not recommended to be running. Also make sure you don't have a custom |
There's no |
Easiest way to find is to use |
Thanks for the swift reply! I see no mention of
|
ok, thanks! looks like there's indeed no Looking at the error again ( |
There is no new information in this comment. Only condensed recreate steps and confirmation of previous observations that the issue goes away upon containerd downgrade or kernel upgrade. Minimal Recreate
Result:
Confirmed relief via containerd downgrade or kernel upgradeAs observed by @pmoris (thanks!), issue goes away when downgrading
... or when upgrading to latest kernel ( |
Pasted the install steps into a vagrant shell provisioner and bisected by vagrant box version: test:
|
Thanks @trapier - so, the runc fix requires a kernel feature that was added in kernel 3.17, but was backported in RHEL kernels. I wonder if kernel For Docker Engine Community, this is not an issue (as it is not supported on RHEL, only on CentOS, so only the latest kernel version is supported), but for Docker Engine Enterprise, we need to check if there's still versions of docker that are supported on RHEL 7.2 (if so, an alternative fix is needed) |
Docker EE does not have anymore versions supported on RHEL 7.2 https://success.docker.com/article/compatibility-matrix |
Maybe my test is helpful: |
@leeningli so in each case, you start two MySQL containers; one with its own networking namespace, and one with Is there anything in the system- or daemon logs? (also might want to check audit logs to see if SELinux is involved) |
So how can I downgrade my docker-ce on centos 7? My production server cannot be restarted. |
Let me close this ticket for now, as it looks like it went stale. |
Expected behavior
Containers should start and remain running.
Actual behavior
I suspect that the update to
docker-ce-18.09.2
and/orcontainerd.io-1.2.2
crashed my running containers and prevents the creation of new ones. Both actions lead to following error:Cannot start service redis: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:293: copying bootstrap data to pipe caused \"write init-p: broken pipe\"": unknown
All running containers (which were managed by
docker-compose
) show the exit statusExited (128)
(postgres, redis, nginx and django) orExited (137)
(celery-worker and celery-beat), which from what I've gathered points to an OOM error? However, the container logs show that my processes received shutdown requests (SIGTERM
) and don't mention memory issues.docker inspect container
The same error appears when I try to spin-up a new container.
Update log
Note that there's a 1 hour difference due to the configuration of the timezone on my OS compared to the containers. It also reports the same error as the inspect command showed:
var/log/messages
I added this as an attachment (var-log-messages.txt). It shows that an update was initiated right before the containers crashed.
OS and kernel
Downgrade attempt
yum downgrade docker-ce
(3:18.09.1-3.el7
) still results in the same error message when I try to recreate my containers.Downgrading
containerd.io
(containerd.io.x86_64 0:1.2.2-3.el7
) in addition todocker-ce
does allow me to recreate the containers.EDIT: upgrading the kernel to
3.10.0-957.5.1.el7.centos.plus.x86_64
also fixes the issue.The text was updated successfully, but these errors were encountered: