Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker Exec does not resize terminal #35407

Closed
thaJeztah opened this issue Nov 4, 2017 · 26 comments
Closed

Docker Exec does not resize terminal #35407

thaJeztah opened this issue Nov 4, 2017 · 26 comments

Comments

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Nov 4, 2017

I noticed this on both master (and docker 17.11-rc2, both on Ubuntu and Docker for Mac);

Client:
 Version:      17.11.0-ce-rc2
 API version:  1.34
 Go version:   go1.8.3
 Git commit:   d7062e5
 Built:        Wed Nov  1 22:11:59 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.11.0-ce-rc2
 API version:  1.34 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   d7062e5
 Built:        Wed Nov  1 22:10:29 2017
 OS/Arch:      linux/amd64
 Experimental: false

When running docker exec, the terminal doesn't resize:

$ docker run -d --name foo nginx:alpine
$ docker exec -it foo sh
/ # 0123456789012345678901234567890123456789012345678901234567890123456789012345
6789

Checking the size with stty size does show the correct dimensions:

/ # stty size
93 410

This only happens on docker exec; doing docker run -it works correct:

$ docker run -it --rm nginx:alpine sh
/ # 01234567890123456789012345678901234567890123456789012345678901234567890123456789

(and shows the same output for stty size):

/ # stty size
93 410

Checking the daemon logs, the resize parameters are passed to the API for both docker run and docker exec;

time="2017-11-04T00:17:19.801321056Z" level=debug msg="Calling POST /v1.34/containers/378120448a6af3b289c12574445dde6f7a5a8f6fb09a311a46d1ffc735fab010/resize?h=93&w=410"


time="2017-11-04T00:16:37.697677138Z" level=debug msg="Calling POST /v1.34/exec/c57124724228e4af259281af94206ce9f02c64e472408198f665b342c1347f03/resize?h=93&w=410"

ping @vieux @mlaventure @crosbymichael

@thaJeztah

This comment has been minimized.

Copy link
Member Author

@thaJeztah thaJeztah commented Nov 4, 2017

I'm not consistently getting this; investigating a bit more, and checking if the resize is actually called;

watch 'curl -sL localhost:1338/v1/metrics | grep ResizePty'
@thaJeztah

This comment has been minimized.

Copy link
Member Author

@thaJeztah thaJeztah commented Nov 4, 2017

So, the metrics are not enabled by default. From discussion on slack;

in /var/run/docker/containerd/containerd.toml;

Add:

[metrics]
    address = "..."

(the code assume it'll be tcp socket)

After updating the config, kill containerd to have it restart and reread the config. Do not restart docker, as it will override the changes.

@crosbymichael

This comment has been minimized.

Copy link
Member

@crosbymichael crosbymichael commented Nov 7, 2017

you are not running the original nginx container with a tty so execs are not going to have one either. it has to be setup when the container is created so a correct /dev/console and /dev/pty are created that later exec's fork off new ptys from

@thaJeztah

This comment has been minimized.

Copy link
Member Author

@thaJeztah thaJeztah commented Nov 8, 2017

Oh! That does make sense, and didn't think of that; let me try;

that does seem to work indeed; it's a bit confusing because the docker exec accepts a -t flag as well 🤔

@thaJeztah

This comment has been minimized.

Copy link
Member Author

@thaJeztah thaJeztah commented Nov 8, 2017

So, thanks! Looks like this is a PEBCAK - let me close

@thaJeztah thaJeztah closed this Nov 8, 2017
@thaJeztah

This comment has been minimized.

Copy link
Member Author

@thaJeztah thaJeztah commented Nov 15, 2017

Still not sure if this could be related to my setup, but there does appear to be a weird issue.

I'm testing a pull-request, so started an interactive development container;

$ make BIND_DIR=. binary shell

Inside the container, installing, no wrapping occurs;

root@5f5c4dc29878:/go/src/github.com/docker/docker# ds;kljads;lkajsd;lkadsj;lasdkjads;lkjasdl;kasdj;laksdjad;slkjads;lkadsj;aldksjad;lkadjs;ladksjad;lkjads;lakdja;dlkjads;lkajd;adlskjads;lkadjs;ladksjasd;lkjad;lkadsj;ladksjasd;lkjads;lkdasj;adlskjads;lkadj;ladkjads;lkadj;ladskjads;lkj^C

I then start the daemon; and output from the daemon also doesn't wrap:

root@5f5c4dc29878:/go/src/github.com/docker/docker# make install
root@5f5c4dc29878:/go/src/github.com/docker/docker# dockerd --debug
DEBU[2017-11-15T01:17:23.552063465Z] Listener created for HTTP on unix (/var/run/docker.sock)
INFO[2017-11-15T01:17:23.553092593Z] libcontainerd: started new docker-containerd process  pid=2915
INFO[0000] starting containerd                           module=containerd revision=2b8ed96d2a422bf205adcdfa05272dc12c81613b version=v1.0.0-beta.3
INFO[0000] changing OOM score to -500                    module=containerd
INFO[0000] loading plugin "io.containerd.content.v1.content"...  module=containerd type=io.containerd.content.v1
INFO[0000] loading plugin "io.containerd.snapshotter.v1.btrfs"...  module=containerd type=io.containerd.snapshotter.v1
WARN[0000] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
INFO[0000] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  module=containerd type=io.containerd.snapshotter.v1
INFO[0000] loading plugin "io.containerd.metadata.v1.bolt"...  module=containerd type=io.containerd.metadata.v1
WARN[0000] could not use snapshotter btrfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt"
INFO[0000] loading plugin "io.containerd.differ.v1.walking"...  module=containerd type=io.containerd.differ.v1

In a different shell, verify that a TTY is enabled for the container:

$ docker inspect $(docker ps -q -n1) | grep Tty
            "Tty": true,

And open a second session (docker exec) into the container:

$ docker exec -it $(docker ps -q -n1) bash

Output produced by docker correctly fits to the size, but anything that I'm typing is wrapped to 80-characters

root@5f5c4dc29878:/go/src/github.com/docker/docker# docker run --help

Usage:	docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Run a command in a new container

Options:
      --add-host list                  Add a custom host-to-IP mapping (host:ip)
  -a, --attach list                    Attach to STDIN, STDOUT or STDERR
      --blkio-weight uint16            Block IO (relative weight), between 10 and 1000, or 0 to disable (default 0)

# (snip)

      --volumes-from list              Mount volumes from the specified container(s)
  -w, --workdir string                 Working directory inside the container
terminal wraps878:/go/src/github.com/docker/docker# but when I start typing the

See the last line in the output above (I typed "but when I start typing the terminal wraps");

terminal wraps878:/go/src/github.com/docker/docker# but when I start typing the
@mpepping

This comment has been minimized.

Copy link

@mpepping mpepping commented Dec 8, 2017

This indeed seems an issues since the early 17.11 releases. It's quite easy to reproduce:

  • make sure your terminal is > 80x24 chars.
  • use docker engine 17.11 (eg. 17.11.0-ce)
  • start a container /w tty +interactive: docker run -ti --name shelltest ubuntu bash
  • start top and verify it's in full screen/terminal
  • open a new terminal and make sure your terminal is > 80x24 chars.
  • run docker exec -ti shelltest bash
  • start top and see it starts in a 80x24 boundary

No issue in =< 17.10.

@mpepping

This comment has been minimized.

Copy link

@mpepping mpepping commented Jan 6, 2018

So, this still seems to be an issue. My workaround for now is setting the COLUMNS and LINES environment variables in the Exec.

docker exec -ti --env COLUMNS=`tput cols` --env LINES=`tput lines` shelltest bash
@thaJeztah

This comment has been minimized.

Copy link
Member Author

@thaJeztah thaJeztah commented Jan 6, 2018

@mempepping are you on Docker for Mac, or something else?

@mpepping

This comment has been minimized.

Copy link

@mpepping mpepping commented Jan 6, 2018

@thaJeztah Same outcome on Docker for Mac and with Docker Engine on a Fedora host.

@thaJeztah

This comment has been minimized.

Copy link
Member Author

@thaJeztah thaJeztah commented Jan 6, 2018

Thanks! Was wondering, because others were reporting not seeing the issue, and only saw it when using Docker for Mac

@lgfausak

This comment has been minimized.

Copy link

@lgfausak lgfausak commented Jan 10, 2018

I am on docker/mac, I am seeing term size issues. For me they started when i upgraded to 17.12 (i think i was on 17.09 before). I've tried setting LINES/COLUMNS/TERM in both the run and exec docker calls. When I exec in, and immediately do a stty size, I see 0 0 as the answer. If I do a reset, or if I resize my window, stty will then report the correct term size and I can carry on inside the docker exec.

@mpepping

This comment has been minimized.

Copy link

@mpepping mpepping commented Jan 11, 2018

The suggestion of resizing the window is an interesting one. It makes it seem that the SIGWINCH signal propagates the terminal size correctly. A small test ..

Start a TTY via docker exec:

docker exec -ti shelltest bash
$ stty size
0 0

On the host, send a SIGWINCH to the docker exec process:

kill -SIGWINCH $(ps -ef | grep 'docker exec -ti shelltest' | grep -v grep | awk '{ print $2}')

Re-run stty in the same TTY as the first step:

$ stty size
68 212

It seems that an initial SIGWINCH on the start of the TTY is missed or isn't sent (correctly).

@mpepping

This comment has been minimized.

Copy link

@mpepping mpepping commented Jan 11, 2018

Two issues that describe the same behavior: #10341, #33794

@brianbruggeman

This comment has been minimized.

Copy link

@brianbruggeman brianbruggeman commented Jan 11, 2018

My office is on a mac; we all see the terminal issue using docker exec. Easy to test with tput lines, tput cols or stty size.

Work-arounds:

* A resize of the terminal fixes the issue.
* docker exec -it $container -e COLUMNS=`tput cols` -e LINES=`tput lines` /bin/bash -l -i
@mpepping

This comment has been minimized.

Copy link

@mpepping mpepping commented Jan 12, 2018

The docker exec -ti performs a "Calling POST /v1.35/exec/${EXEC-ID}/resize" during it's start:

time="2018-01-12T21:15:41.725755283Z" level=debug msg="Calling GET /_ping"
time="2018-01-12T21:15:41.726438011Z" level=debug msg="Calling GET /v1.35/containers/youthful_fermi/json"
time="2018-01-12T21:15:41.728114278Z" level=debug msg="Calling POST /v1.35/containers/youthful_fermi/exec"
time="2018-01-12T21:15:41.728217094Z" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":true,\"AttachStdout\":true,\"Cmd\":[\"bash\"],\"Detach\":false,\"DetachKeys\":\"\",\"Env\":null,\"Privileged\":false,\"Tty\":true,\"User\":\"\",\"WorkingDir\":\"\"}"
time="2018-01-12T21:15:41.736287790Z" level=debug msg="Calling POST /v1.35/exec/f1382509b171dfbd9c5d8c8fa2f1a6f3acf4309dfa491196ef98bd40f450c18f/start"
time="2018-01-12T21:15:41.736377670Z" level=debug msg="form data: {\"Detach\":false,\"Tty\":true}"
time="2018-01-12T21:15:41.738118385Z" level=debug msg="starting exec command f1382509b171dfbd9c5d8c8fa2f1a6f3acf4309dfa491196ef98bd40f450c18f in container 1f0da635b423808b3438e2d338fe11ed6049c428e2768451bc937f507dcde1ac"
time="2018-01-12T21:15:41.740337256Z" level=debug msg="Calling POST /v1.35/exec/f1382509b171dfbd9c5d8c8fa2f1a6f3acf4309dfa491196ef98bd40f450c18f/resize?h=61&w=212"
time="2018-01-12T21:15:41.739369481Z" level=debug msg="attach: stdin: begin"
time="2018-01-12T21:15:41.741206281Z" level=debug msg="attach: stdout: begin"
time="2018-01-12T21:15:41Z" level=debug msg="event published" module="containerd/tasks" ns=moby topic="/tasks/exec-started" type=containerd.events.TaskExecStarted 
time="2018-01-12T21:15:41.806260976Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/exec-started
time="2018-01-12T21:15:41.806261138Z" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/exec-started
time="2018-01-12T21:15:41.806302812Z" level=warning msg="unknown container" container=1f0da635b423808b3438e2d338fe11ed6049c428e2768451bc937f507dcde1ac module=libcontainerd namespace=plugins.moby

Although it seems that this POST happens somewhat too early, since stty size shows:

root@1f0da635b423:/# stty size
0 0

Now, if the exact same "Calling POST /v1.35/exec/${EXEC-ID}/resize" call is performed via curl, right after the TTY session is started, it corrects the stty size:

On the Docker host:

curl --unix-socket /var/run/docker.sock -H "Content-Type: application/json" -X POST \
"http:/v1.35/exec/f1382509b171dfbd9c5d8c8fa2f1a6f3acf4309dfa491196ef98bd40f450c18f/resize?h=61&w=212"

And in the container:

# stty size
61 212

This makes this tty issue seem to be a race condition, or timing issue, during the start of the exec. The correct "Calling POST /v1.35/exec/${EXEC-ID}/resize" call is performed during the start, but the STTY size is set incorrectly. Re-running the same call a somewhat later sets the correct STTY size.

@thaJeztah

This comment has been minimized.

Copy link
Member Author

@thaJeztah thaJeztah commented Jan 15, 2018

Yes, it looks like a race condition, and likely the initial resize is triggered to early; cli/command/container/tty.go#L48. Moving the resize to the end of that function (or adding another resize after monitoring is started; cli/command/container/exec.go#L156-L160) seems to help slightly, but it's still racy.

So, this may actually be the same issue as #31223

ping @mlaventure

@mlaventure

This comment has been minimized.

Copy link
Contributor

@mlaventure mlaventure commented Jan 16, 2018

Yes, sorry this is still in my list. Unfortunately, I still haven't gotten around setting up a proper working env for docker-ce (the mono-repo doesn't make it easy 😅). I'll try to dedicate some time to it in the next couple weeks.

jschwartzentruber added a commit to jschwartzentruber/docker-bootstrap that referenced this issue Apr 6, 2018
@mkurz

This comment has been minimized.

Copy link

@mkurz mkurz commented Apr 12, 2018

Any updates here?
@mlaventure Three months since your comment, did you find time to fix this?

@mlaventure

This comment has been minimized.

Copy link
Contributor

@mlaventure mlaventure commented Apr 12, 2018

@mkurz yes and no. I did make a PR to fix it (#36147) but the actual solution doesn't have consensus. And I haven't had much time lately to work on things Docker unfortunately.

Anyone is welcome to carry on/address the comment on that PR :)

retr0h added a commit to retr0h/molecule that referenced this issue Apr 12, 2018
Containers accessed via `molecule login` are not getting
$COLUMNS and $LINES set.  Update docker exec to pass the
proper $COLUMNS and $LINES env vars [1] as a work-a-round.

    [1] moby/moby#35407

Fixes: ansible-community#1244
retr0h added a commit to ansible-community/molecule that referenced this issue Apr 12, 2018
Containers accessed via `molecule login` are not getting
$COLUMNS and $LINES set.  Update docker exec to pass the
proper $COLUMNS and $LINES env vars [1] as a work-a-round.

    [1] moby/moby#35407

Fixes: #1244
fiuzagr added a commit to iaguara/iaguara that referenced this issue Apr 15, 2018
achekulaev added a commit to docksal/docksal that referenced this issue May 17, 2018
…e in fin bash characters may overlap

For details see:
moby/moby#33794
moby/moby#35407
tadas-s added a commit to tadas-s/dotfiles that referenced this issue May 21, 2018
… terminal

Taken from [this discussion][1].

Otherwise, using binding.pry with this 'dce' alias is nearly impossible.

[1]: moby/moby#35407 (comment)
@lmakarov lmakarov mentioned this issue May 31, 2018
@grigoryosifov

This comment has been minimized.

Copy link

@grigoryosifov grigoryosifov commented Jun 7, 2018

The SIGWINCH workaround, suggested by @mpepping is much better than setting COLUMNS and LINES envs because it also supports resizing of the terminal window, without the need to restart Docker exec every time your terminal size changes. However the issue was that it had to be manually run every time after docker exec, so here's my workaround:

If you use Makefile, you can add:

.PHONY: sh
sh:
#	Fix Docker terminal to resize properly:
#	https://github.com/moby/moby/issues/35407#issuecomment-357046159
	@./fix-docker-terminal.bash &
	@docker exec -it -u www-data container zsh

and create fix-docker-terminal.bash in the same directory:

#!/bin/bash
sleep 1
ps -ef | grep 'docker exec -it' | grep -v grep | awk '{ print $2}' | xargs kill -SIGWINCH

Do not forget to chmod +x fix-docker-terminal.bash.

If you don't use Makefile, you can go with this one line:
./fix-docker-terminal.bash & docker exec -it -u www-data container zsh

@mpepping

This comment has been minimized.

Copy link

@mpepping mpepping commented Jun 7, 2018

Nice workaround, @grigoryosifov! Luckily it seems that de definitive fix for this issue is getting posture: #37172

@thaJeztah

This comment has been minimized.

Copy link
Member Author

@thaJeztah thaJeztah commented Jun 8, 2018

Fixed on master through #37172

@mpepping

This comment has been minimized.

Copy link

@mpepping mpepping commented Jun 9, 2018

@thaJeztah Thanks! This will probably make a lot of people happy! 👍

GMTA added a commit to GMTA/docksal that referenced this issue Dec 10, 2018
Originally introduced by 6e53837 and other commits; this workaround is no
longer necessary since the bug in Docker was squashed:

moby/moby#35407
grahamlyons added a commit to mergermarket/cdflow that referenced this issue Apr 5, 2019
Workaround the issue in Docker where the terminal has a strange size:
moby/moby#35407

This issue is purportedly fixed but I'm still seeing it.
grahamlyons added a commit to mergermarket/cdflow that referenced this issue Jul 29, 2019
Workaround the issue in Docker where the terminal has a strange size:
moby/moby#35407

This issue is purportedly fixed but I'm still seeing it.
@uptycs-shivurh

This comment has been minimized.

Copy link

@uptycs-shivurh uptycs-shivurh commented Jan 28, 2020

So, this still seems to be an issue. My workaround for now is setting the COLUMNS and LINES environment variables in the Exec.

docker exec -ti --env COLUMNS=`tput cols` --env LINES=`tput lines` shelltest bash

This helped me on mac -

$ tput lines
39
$ tput cols
178

$ docker exec -e COLUMNS="178" -e LINES="39" -it CONTAINER_NAME /bin/bash

@thaJeztah

This comment has been minimized.

Copy link
Member Author

@thaJeztah thaJeztah commented Jan 28, 2020

@uptycs-shivurh what version of docker are you on? I think this issue should be fixed on current versions of the docker cli

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

10 participants
You can’t perform that action at this time.