UTF-8 character in the default shell profile causes rendering issues without a $LANG or locale configured

[griwes]

I've tried this with both bash and zsh. They are both broken, but in slightly different ways; I think the breakage in bash is worse, but they both are not pretty.

Steps to reproduce and demonstrate what I mean:

1. Create a bash/zsh process in a devcontainer attached to a tty.
2. Press enter repeatedly so that you are at the bottom of the tty in your terminal emulator.
3. Type in a long command that overflows the line length of your terminal emulator.
4. Run that command.
5. Next step depends on the shell being used:
   1. For bash, press the up arrow, and observe how the overflowing command does not appear fully, and how another line of text in the terminal disappears. If you now repeatedly press the down and up arrows to recall a command from history and clear it, you will see that for each of the pairs of arrow key presses, one line of text that was previously in the terminal vanishes.
   2. For zsh, press the up arrow, then the down arrow. You will see that the cursor is two cells further into the line than it should be; if you go up twice and then down, two characters from a previous command will appear in those two cells, and will not change until you force a new line to be sent to the terminal.

I believe that this is related to this problem, though the linked original talks specifically about non-printable characters, and the arrow is definitely printable. Nevertheless, with it being a multi-byte character, it appears to me that using the solution normally used for the non-printable characters, i.e. wrapping it with \[\] in PS1, fixes the issue for bash. I am unsure if this is a viable solution to this problem, which is why I'm filing this as an issue and not with a specific patch. I assume that the zsh method of escaping the characters in PROMPT would similarly work, but I have not verified this.

[jkeech]

Thanks for reporting! I believe all of the non-printable characters are already wrapped with escaped square brackets in https://github.com/devcontainers/features/blob/main/src/common-utils/scripts/bash_theme_snippet.sh as well as the zsh variant, but the theme prompt is quite large so it's possible there's a missing character somewhere in there. We'll need to take a closer look to double check.

[griwes]

All the actual non-printable characters are wrapped, but the arrow is causing issues and removing or wrapping it fixes those issues. It's not non-printable, which is why I don't know if wrapping it as if it was is a reasonable idea or not :)

[jkeech]

All the actual non-printable characters are wrapped, but the arrow is causing issues and removing or wrapping it fixes those issues. It's not non-printable, which is why I don't know if wrapping it as if it was is a reasonable idea or not :)

Thanks for clarifying! I'm not sure whether that would cause any problems either, but it seems like it would since bash will still have an incorrect understanding of the actual printable width of the line. I'm not able to reproduce the problem you describe. Here's a recording of my test in zsh.

Screen.Recording.2023-08-18.at.5.34.26.PM.mov

Could you capture a video of the issue in VS Code with Developer: Toggle Screencast Mode enabled to show the key strokes as you make changes in the terminal? It would also be helpful to get the following information:

• OS Version
• Client used for connecting to the devcontainer (VS Code desktop with local devcontainers, VS Code desktop with Codespaces, VS Code in the browser with Codespaces, etc)
• Devcontainer config or repo with a minimal reproduction

[griwes]

Attached a screencap.

I'm on Debian sid, I'm using devcontainer exec bash; I see the same behavior with neovim terminal, tmux, and alacritty, so I assume that this is just a case of the prompt confusing bash in one way or the other.

Also noticed you don't actually have to be at the bottom of the tty to see this while recording the screencap below.

An example of what I can repro it with is https://github.com/NVIDIA/cccl/blob/main/.devcontainer/cuda12.2-gcc12/devcontainer.json.

screencap.mp4

[jkeech]

I'm on Debian sid, I'm using devcontainer exec bash; I see the same behavior with neovim terminal, tmux, and alacritty, so I assume that this is just a case of the prompt confusing bash in one way or the other.

I'm able to reproduce using devcontainer exec bash but not in any other client (VS code terminals, ssh, etc), so I suspect this is a bug in the way devcontainer exec bash is attaching the pty/tty. I'm going to transfer this to the CLI repo since this looks like a CLI bug.

[jkeech]

@chrmarti can you take a look at this?

[chrmarti]

I can also reproduce with docker exec. Does this indicate that the PS1 value needs additional wrapping?

[jkeech]

I can also reproduce with docker exec. Does this indicate that the PS1 value needs additional wrapping?

I checked all of the escapes in https://github.com/devcontainers/features/blob/main/src/common-utils/scripts/bash_theme_snippet.sh is they look correct to me. All of the non-printable sequences are wrapped in square brackets.

I was not able to reproduce in pure ssh connections (using the sshd feature and forwarding the ssh port as described here) or in VS Code integrated terminals which makes me think it's not an issue in the shell profile. Perhaps it's a bug in docker exec, which would explain why it also shows up in devcontainer exec which wraps docker exec.

[jkeech]

Here's a more minimal repro:

docker run --rm -it mcr.microsoft.com/devcontainers/base:ubuntu
echo ➜

Then hit the up and down arrow back and forth and notice one character is overwritten at a time.

Screen.Recording.2023-08-22.at.10.21.01.AM.mov

If you clear out the profile and set it to a minimal profile config before doing the echo, the same thing happens:

docker run --rm -it mcr.microsoft.com/devcontainers/base:ubuntu
export PS1="\s-\v\$"
echo ➜

Screen.Recording.2023-08-22.at.10.23.39.AM.mov

Similarly, spawning the initial bash shell with --norc also still reproduces the issue.

docker run --rm -it mcr.microsoft.com/devcontainers/base:ubuntu /bin/bash --norc  
echo ➜

The UTF-8 character in the output seems to be messing up width calculation somehow. If you try with the ubuntu:latest image, it doesn't reproduce, but it also doesn't accept unescaped UTF-8 characters as input on the terminal without installing additional locales, so I wonder if that's related.

docker run --rm -it ubuntu:latest
apt update && apt install locales
dpkg-reconfigure locales
<enter 160 and then 3 on the options>
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8
bash
echo ➜

I did notice that changing the locale configured in mcr.microsoft.com/devcontainers/base:ubuntu from POSIX to en_US.utf8 seems to fix the issue, but it's not entirely clear why when the ubuntu:latest image also comes configured out of the box with the POSIX locale which works fine.

docker run --rm -it mcr.microsoft.com/devcontainers/base:ubuntu
export LC_ALL=en_US.utf8
bash
echo ➜

I don't know enough about how locales are configured to explain this difference, and it's still not clear to me why this only reproduces in docker exec and devcontainer exec rather than in shells spawned through other tools.

[jkeech]

The line length is definitely wrong in the base devcontainers image when no locale is explicitly set.

bash-5.1# echo ➜ | wc --max-line-length
0
bash-5.1# export LC_ALL=en_US.utf8
bash-5.1# echo ➜ | wc --max-line-length
1

Compared to ubuntu:latest which shows the correct value without setting the locale explicitly:

bash-5.1# echo ➜ | wc --max-line-length
1

Knowing that it's a locale issue also helps explain why it works in VS Code interactive terminals and SSH shells. In VS Code and SSH, LANG=en_US.UTF-8 is being applied which also fixes the issue.

At this point, it's definitely not a CLI bug. It's either in the image or the common-utils feature, although it's tied to the locales rather than the PS1 escaping. Transferring back to the features repo for now.

[jkeech]

Walking up the stack a bit on the image where it's broken, the upstream image buildpack-deps:jammy-curl https://github.com/devcontainers/images/blob/main/src/base-ubuntu/.devcontainer/Dockerfile#L3 also reproduces this issue without any devcontainer-specific configuration.

docker run --rm -it buildpack-deps:jammy-curl
echo ➜

[jkeech]

Looking at the Dockerfile for buildpack-deps and bisecting where it start breaking, it's right after gnupg is installed at https://github.com/docker-library/buildpack-deps/blob/93d6db0797f91ab674535553b7e0e762941a02d0/ubuntu/jammy/curl/Dockerfile#L14.

[jkeech]

Opened docker-library/buildpack-deps#148

[ridhwaans]

same. does LANG not stay exported after common-utils install?

features/src/common-utils/main.sh

Lines 144 to 149 in 084df81

	# Ensure at least the en_US.UTF-8 UTF-8 locale is available = common need for both applications and things like the agnoster ZSH theme.
	if [ "${LOCALE_ALREADY_SET}" != "true" ] && ! grep -o -E '^\s*en_US.UTF-8\s+UTF-8' /etc/locale.gen > /dev/null; then
	echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
	locale-gen
	LOCALE_ALREADY_SET="true"
	fi

on debian

locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=