docker run -i -a stdin hanging #18543

Open
Benjamin-Dobell opened this Issue Dec 9, 2015 · 9 comments

Comments

Projects
None yet
5 participants
@Benjamin-Dobell

I've just updated to the latest Docker (1.9.1) and docker run -a -a stdin <image> <command> is hanging 100% of the time on my machine.

Docker prints the container ID and the container shuts down when the command completes, but the docker process itself hangs around. This is breaking a lot of scripts.

I'm assuming this is a somewhat isolated case, because imagine this would cause huge problems for everyone if it wasn't limited to something to do with my setup.

3.13.0-71-generic #114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 14.04
Linode - But we're running a generic kernel thanks to KVM and Grub 2 (rather than the outdated Linode provided kernel).

Any suggestions on how I can try identify the root cause?

@Benjamin-Dobell

This comment has been minimized.

Show comment
Hide comment
@Benjamin-Dobell

Benjamin-Dobell Dec 9, 2015

It's probably worth noting that:

docker run -i -d <image> <command>

works just fine, printing the container ID as expected.

It's probably worth noting that:

docker run -i -d <image> <command>

works just fine, printing the container ID as expected.

@Benjamin-Dobell

This comment has been minimized.

Show comment
Hide comment
@Benjamin-Dobell

Benjamin-Dobell Dec 9, 2015

$ sudo docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64
$ sudo docker info
Containers: 26
Images: 579
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 717
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-71-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 8
Total Memory: 15.67 GiB
Name: uat
ID: EF2K:SNH2:RXTW:YA27:HUNZ:RTO3:WYPM:TJIT:DZ3C:TTNN:OVST:BNTO
WARNING: No swap limit support

uname -a, as provided above:
`Linux uat 3.13.0-71-generic #114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux'

Reproduction is literally execute:

docker run -i -a stdin <image> <command>

with any image and any command. As mentioned above, after the command finishes the container stops running and the container ID is printed as expected, but the Docker process (on the host) never returns.

$ sudo docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64
$ sudo docker info
Containers: 26
Images: 579
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 717
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-71-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 8
Total Memory: 15.67 GiB
Name: uat
ID: EF2K:SNH2:RXTW:YA27:HUNZ:RTO3:WYPM:TJIT:DZ3C:TTNN:OVST:BNTO
WARNING: No swap limit support

uname -a, as provided above:
`Linux uat 3.13.0-71-generic #114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux'

Reproduction is literally execute:

docker run -i -a stdin <image> <command>

with any image and any command. As mentioned above, after the command finishes the container stops running and the container ID is printed as expected, but the Docker process (on the host) never returns.

@Benjamin-Dobell

This comment has been minimized.

Show comment
Hide comment
@Benjamin-Dobell

Benjamin-Dobell Dec 9, 2015

Interesting, if input is provided, in the form of:

echo "Some input" | sudo docker run -i -a stdin <image> <command>

Then the command returns as expected. Was this a recent change, is this expected behaviour now?

Interesting, if input is provided, in the form of:

echo "Some input" | sudo docker run -i -a stdin <image> <command>

Then the command returns as expected. Was this a recent change, is this expected behaviour now?

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Dec 9, 2015

Contributor

What if you don't do -a stdin?

I'll have a look at this to see what might be causing it.
The command should not hang waiting for stdin.

Contributor

cpuguy83 commented Dec 9, 2015

What if you don't do -a stdin?

I'll have a look at this to see what might be causing it.
The command should not hang waiting for stdin.

@cpuguy83 cpuguy83 self-assigned this Dec 9, 2015

@coolljt0725

This comment has been minimized.

Show comment
Hide comment
@coolljt0725

coolljt0725 Dec 10, 2015

Contributor

@cpuguy83 https://github.com/docker/docker/blob/master/api/client/hijack.go#L52
io.Copy(resp.Conn, inputStream) will not return even if the hijack connection is end, so close(stdinDone) never happen, if we only -a stdin, https://github.com/docker/docker/blob/master/api/client/hijack.go#L63 will block forever, so this go routine keep running. And if we attach -a stdout or -a stderr (which is default), io.Copy(outputStream, resp.Reader) will end if connection end, and https://github.com/docker/docker/blob/master/api/client/hijack.go#L63 will not block if container process end.

Contributor

coolljt0725 commented Dec 10, 2015

@cpuguy83 https://github.com/docker/docker/blob/master/api/client/hijack.go#L52
io.Copy(resp.Conn, inputStream) will not return even if the hijack connection is end, so close(stdinDone) never happen, if we only -a stdin, https://github.com/docker/docker/blob/master/api/client/hijack.go#L63 will block forever, so this go routine keep running. And if we attach -a stdout or -a stderr (which is default), io.Copy(outputStream, resp.Reader) will end if connection end, and https://github.com/docker/docker/blob/master/api/client/hijack.go#L63 will not block if container process end.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Dec 10, 2015

Contributor

Yep, seems so -- add that to on the daemon + tls connections, CloseWrite is never called on the stdin stream because tls conns don't implement CloseWrite.

Contributor

cpuguy83 commented Dec 10, 2015

Yep, seems so -- add that to on the daemon + tls connections, CloseWrite is never called on the stdin stream because tls conns don't implement CloseWrite.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Dec 10, 2015

Member

@cpuguy83 you're working on this? Should it be marked as "bug"?

Member

thaJeztah commented Dec 10, 2015

@cpuguy83 you're working on this? Should it be marked as "bug"?

@cpuguy83 cpuguy83 added the kind/bug label Dec 14, 2015

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Dec 14, 2015

Contributor

Yes, marked as bug.
Thinking about the cleanest way to fix this.
We may have to call wait on the container in a separate goroutine for this case.

Contributor

cpuguy83 commented Dec 14, 2015

Yes, marked as bug.
Thinking about the cleanest way to fix this.
We may have to call wait on the container in a separate goroutine for this case.

@Benjamin-Dobell Benjamin-Dobell referenced this issue in sehrope/dokku-logging-supervisord Jan 26, 2016

Open

Runs the Docker container commands in detached mode #33

dblock added a commit to dblock/code.dblock.org that referenced this issue Feb 10, 2016

@LK4D4 LK4D4 added the area/cli label Sep 16, 2016

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Sep 16, 2016

Contributor

@cpuguy83 looks like still an issue.

Contributor

LK4D4 commented Sep 16, 2016

@cpuguy83 looks like still an issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment