Docker exec suddenly not working

[koehn]

Description

Suddenly after upgrading to 18.09, I am unable to exec into any container; when I try I get returned to my normal shell. If I try to exec a command that doesn't exist in a container, I get the proper error message.

Steps to reproduce the issue:

1. docker exec -it [container-name] [command e.g., /bin/sh]
2. docker exec -it [container-name] [bad command e.g., /bin/foobar]

Describe the results you received:

1. If I run a command that's valid (e.g., /bin/sh), I get returned to the command line of the host.
2. If I run a bad command, I get the expected error message:

OCI runtime exec failed: exec failed: container_linux.go:344: starting container process caused "exec: \"/bin/foobar\": stat /bin/foobar: no such file or directory": unknown

Describe the results you expected:

1. Expect the command to run inside the container. This worked reliably prior to the upgrade.

Additional information you deem important (e.g. issue happens only occasionally):
If I run docker -l debug exec I get the following:

DEBU[0000] [hijack] End of stdin                        
DEBU[0000] Error sendStdin: read /dev/stdin: resource temporarily unavailable 
DEBU[0000] [hijack] End of stdout               

I tried running cat /dev/stdin and it works as expected.

Output of docker version:

Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:35:31 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:02:44 2019
  OS/Arch:          linux/amd64
  Experimental:     true

Output of docker info:

Containers: 27
 Running: 26
 Paused: 0
 Stopped: 1
Images: 234
Server Version: 18.09.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 523
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 96ec2177ae841256168fcf76954f7177af9446eb
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-43-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 31.4GiB
Name: turnkey.koehn.com
ID: YOGX:6IOO:NUM4:2ZK3:LOR6:5TNC:CPGE:KTIV:QRBU:FT36:KTY4:EUWB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: koehn
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical box running Ubuntu bionic on bare metal. I tried removing docker-ce with apt-get purge -y docker-ce && apt-get autormove -y and re-installing it with apt-get install -y docker-ce to no avail.

[systemmonkey42]

Can I ask why you closed this? Did you resolve it somehow? I have a production server stuck in this state right now and I don't really have the scope to reboot it for some time.

[koehn]

It turned out the shell I was using was corrupted somehow; as soon as I exited that shell and opened another the problem went away. Another symptom in my case was if I ran apt-get with the -y the command also terminated, just like docker-exec did.

[AkihiroSuda]

Potentially related: docker/cli#1707

@koehn Were you using remote a daemon (DOCKER_HOST=tcp://... or DOCKER_HOST=ssh://...) when you hit the issue?

[koehn]

@AkihiroSuda no, I was using a local daemon.