New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot ssh into a running pod/container -- rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:110: decoding init error from pipe caused \"read parent: connection reset by peer\"" command terminated with exit code 126 #21590

Open
sverma37 opened this Issue Dec 2, 2018 · 19 comments

Comments

Projects
None yet
@sverma37
Copy link

sverma37 commented Dec 2, 2018

Cannot ssh into a running pod/container

Version

openshift v3.9.0+ba7faec-1
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.16

Steps To Reproduce
  1. Installed openshift 3.9 with one master and 2 nodes
  2. Deployed Camunda image using openshift. The image got deployed and the service is running. However, I can not ssh into the pod
Current Result

oc rsh camunda-1-bk6h8

rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:110: decoding init error from pipe caused "read parent: connection reset by peer""
command terminated with exit code 126

Expected Result

Logged in the container's shell

@AleksanderGrzybowski

This comment has been minimized.

Copy link

AleksanderGrzybowski commented Dec 4, 2018

Hello, we have the same problem, all containers with shell probes are stuck in a crashloop because of that. Removing the probe looks like a workaround for now, but there is no way to run commands in containers.

@ephracis

This comment has been minimized.

Copy link

ephracis commented Dec 4, 2018

I am seeing similar issues today. I don't think it is related to OpenShift but to something else.

Steps to reproduce:

  1. Create virtual machine with CentOS (haven't tried on RHEL)
  2. yum -y install docker
  3. systemctl start docker
  4. docker run -d --name test nginx
  5. docker exec test id

Some info:

$ cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
$ uname -a
Linux test.example.com 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 5eda6f6fd0c2884c2c8e78a6e7119e8d0ecedb77 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
 selinux
Kernel Version: 3.10.0-862.14.4.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 1
Total Memory: 7.639 GiB
Name: test.example.com
ID: G6RY:7BY4:IBRN:B7RO:XU54:QBBN:YK4K:7EEL:OVVO:FYFV:UWIN:OJEV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)
@senax

This comment has been minimized.

Copy link

senax commented Dec 4, 2018

I've seen the same with a different container; In my case the fix was to downgrade docker from release 84 to 75.

yum downgrade docker-1.13.1-75.git8633870.el7.centos.x86_64 docker-client-1.13.1-75.git8633870.el7.centos.x86_64 docker-common-1.13.1-75.git8633870.el7.centos.x86_64

@LalatenduMohanty

This comment has been minimized.

Copy link

LalatenduMohanty commented Dec 4, 2018

I have also seen the same issue in CDK https://issues.jboss.org/browse/CDK-380. With latest docker we are seeing the issue, but with older docker 1.13.1-75 the issue is not there.


$ sudo rpm -qa \| grep -i docker
--
docker-client-1.13.1-84.git07f3374.el7.x86_64
docker-1.13.1-84.git07f3374.el7.x86_64
docker-rhel-push-plugin-1.13.1-84.git07f3374.el7.x86_64
docker-common-1.13.1-84.git07f3374.el7.x86_64
@mrunalp

This comment has been minimized.

Copy link
Member

mrunalp commented Dec 4, 2018

@lsm5 @runcom Looks like we have a regression in build 84.

@AwesomeKai

This comment has been minimized.

Copy link

AwesomeKai commented Dec 5, 2018

Experiencing the same 🤔 glad i'm not the only one

version 1.13.1, build 07f3374/1.13.1

@rainmor

This comment has been minimized.

Copy link

rainmor commented Dec 5, 2018

I've seen the same with a different container; In my case the fix was to downgrade docker from release 84 to 75.

yum downgrade docker-1.13.1-75.git8633870.el7.centos.x86_64 docker-client-1.13.1-75.git8633870.el7.centos.x86_64 docker-common-1.13.1-75.git8633870.el7.centos.x86_64

thanks for your solution。

@simongrzebieta

This comment has been minimized.

@cassiussa

This comment has been minimized.

Copy link

cassiussa commented Dec 6, 2018

The suggestion that @senax provided above worked for me as well (on the heketi container).

yum downgrade docker-1.13.1-75.git8633870.el7.centos.x86_64 docker-client-1.13.1-75.git8633870.el7.centos.x86_64 docker-common-1.13.1-75.git8633870.el7.centos.x86_64

Edited to add that it also seems to have solved an issue I was having with containerized etcd on a clean 3.11 installation.

@rootsongjc

This comment has been minimized.

Copy link

rootsongjc commented Dec 6, 2018

I've seen the same issue with my kubernetes-vagrant-centos-cluster today, after downgrade docker, it works for me.

My environment

  • Docker 1.13.1-84 before, 1.13.1-75 now
  • Kubernetes v1.13.0
  • CentOS Linux release 7.4.1708 (Core)
  • Linux 3.10.0-693.11.6.el7.x86_64

You can also use docker-ce instead of the RedHat official docker release.

rootsongjc added a commit to rootsongjc/rootsongjc.github.io that referenced this issue Dec 6, 2018

@mrunalp

This comment has been minimized.

Copy link
Member

mrunalp commented Dec 6, 2018

An updated fixed build is on the way.

xsgordon added a commit to xsgordon/okd-kubevirt-demo that referenced this issue Dec 6, 2018

Pin Docker version to last known good
Pin Docker version to last known good version before:

openshift/origin#21590
@zhachaoy

This comment has been minimized.

Copy link

zhachaoy commented Dec 7, 2018

I've seen the same with a different container; In my case the fix was to downgrade docker from release 84 to 75.

yum downgrade docker-1.13.1-75.git8633870.el7.centos.x86_64 docker-client-1.13.1-75.git8633870.el7.centos.x86_64 docker-common-1.13.1-75.git8633870.el7.centos.x86_64

nice

@yllierop

This comment has been minimized.

Copy link

yllierop commented Dec 8, 2018

I've seen this same issue as well. @mrunalp what's the status of a fix?

@simongrzebieta

This comment has been minimized.

Copy link

simongrzebieta commented Dec 9, 2018

Red hat have said it's been fixed in version docker-1.13.1-88

https://access.redhat.com/solutions/3734981
This issue is resolved in docker-1.13.1-88 in https://access.redhat.com/errata/RHBA-2018:3796.

@rcsavage

This comment has been minimized.

Copy link

rcsavage commented Dec 10, 2018

yum downgrade docker-1.13.1-75.git8633870.el7.centos.x86_64 docker-client-1.13.1-75.git8633870.el7.centos.x86_64 docker-common-1.13.1-75.git8633870.el7.centos.x86_64

This EXACT issue is happening to me as well. I had to downgrade docker the exact same way to overcome the OCI runtime error.

@rcsavage

This comment has been minimized.

Copy link

rcsavage commented Dec 10, 2018

If RH says the ""fix"" is in docker RPM docker -1.13.1-88 that's great news, but it's not available in CentOS 7 yet. This work around (i.e. downgrade) will have to be our option for now until the CentOS community delivers a new package.

@dsroark

This comment has been minimized.

Copy link

dsroark commented Dec 10, 2018

Encountered this exact same issue where a container was failing because it was unable to launch a PostStartHook command. I was able to upgrade to version ...-88 just now on Cent7 and that resolved the issue

@xsgordon

This comment has been minimized.

Copy link

xsgordon commented Dec 14, 2018

If RH says the ""fix"" is in docker RPM docker -1.13.1-88 that's great news, but it's not available in CentOS 7 yet. This work around (i.e. downgrade) will have to be our option for now until the CentOS community delivers a new package.

http://mirror.centos.org/centos/7/extras/x86_64/Packages/docker-1.13.1-88.git07f3374.el7.centos.x86_64.rpm

@AwesomeKai

This comment has been minimized.

Copy link

AwesomeKai commented Dec 27, 2018

I've seen the same with a different container; In my case the fix was to downgrade docker from release 84 to 75.
yum downgrade docker-1.13.1-75.git8633870.el7.centos.x86_64 docker-client-1.13.1-75.git8633870.el7.centos.x86_64 docker-common-1.13.1-75.git8633870.el7.centos.x86_64

nice

👍 Great work around, works

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment