Terminal on Windows: CJK bytes escaped as M-... in the prompt, and IME (pinyin) input does not work

[KannaKuron]

Summary

Two problems in the DSH terminal on Windows (Git Bash via the official terminal), both affecting CJK users:

1. Non-ASCII bytes in the prompt are escaped (M-…) while pwd prints the same path correctly.
2. An IME (pinyin) cannot enter text in the terminal.

They look independent (one is the shell's prompt string, the other is the frontend input path), so they are reported separately below.

Environment

• Windows 11 · DSH 0.1.7-rc.2 (dsh --version) · web GUI
• terminal: the built-in sidebar terminal, shell = Git Bash
• working directory contains non-ASCII (CJK) characters
• (Additional data point collected on macOS with a real pty, see the reproduction.)

Problem 1 — CJK bytes in the prompt become M-…

Actual: the prompt shows the working directory as escaped 8-bit bytes:

~/Documents/<workspace-with-cjk>/M-iM-;M-^XM-hM-.M-$M-eM-7M-%M-dM-=M-^\M-eM-^LM-:> 

Same terminal, same directory: pwd prints it correctly:

/c/Users/<user>/Documents/<workspace-with-cjk>

So the path itself is fine; only the prompt rendering is affected. (This is the same mechanism the user reported as "path display is broken".)

Minimal reproduction / what I could confirm

I could not run the Windows build, so I reproduced the mechanism on macOS through a real pty with a CJK directory and PS1='\w> ':

locale passed to the shell	prompt
LC_ALL=C	/private/tmp/…/M-iM-;M-^XM-hM-.M-$M-eM-7M-%M-dM-=M-^\M-eM-^LM-:> (escaped)
LC_ALL=en_US.UTF-8	escaped identically
LC_ALL=C + TERM=dumb	escaped identically

Two findings worth noting:

• A literal non-ASCII PS1 (PS1='中文目录> ') is not escaped in any of those cases. The escaping happens where the shell itself decodes the working directory into the prompt (\w / \W, which the default Git Bash prompt uses). It is therefore not the terminal renderer.
• Contrary to my initial hypothesis, the locale did not change the behaviour on that macOS/bash build, so I am not claiming the cause. The Windows check is the missing piece — see below.

Readings requested on Windows (this is the decisive control):

echo "LANG=[$LANG] LC_ALL=[$LC_ALL] LC_CTYPE=[$LC_CTYPE]"
locale
PS1='\w> '          # does the escaped form come back?
export LC_ALL=en_US.UTF-8   # or C.UTF-8, whichever Git Bash ships
PS1='\w> '

Expected: the prompt persists the path in its original non-ASCII form (or at least the terminal shows readable CJK), like pwd does.
Actual: the prompt shows M-… escapes.

A code lead (read-only inspection of my local checkout): the terminal backend does not pass any locale to the shell. packages/terminal/terminal-bash/src/index.ts childEnvironment() (around :64-88) sets TERM: 'dumb', PAGER, GIT_PAGER, DSH_*, PS1, PROMPT_COMMAND — and no LANG/LC_ALL/LC_CTYPE; the environment is the subprocess provider's scrubbed ambient base, which on Windows has no locale variables. There is no other LC_ALL in packages/terminal/*, packages/api/terminal-controller or packages/subprocess/subprocess-local except Linux sandbox helpers pinning LC_ALL=C (subprocess-local/src/linux-scope.ts). Passing a UTF-8 locale for the terminal shell (or allowing a per-row env map) would be an upstream change; a third-party plugin cannot do it today because the terminal row's Config only carries shell.path / shell.name / shell.args.

Problem 2 — IME (pinyin) does not work in the terminal

Actual: typing pinyin in the DSH terminal does not produce characters (composition/candidate window does not commit text).

Not reproduced locally (the headless macOS environment has no IME), so this half is reported as observed behaviour plus a code-level pointer rather than a confirmed cause.

Code-level pointer: the terminal frontend is xterm.js (packages/client/ui-sidebar-terminal/src/client/terminal.tsx, new Terminal({…})), so composition handling is xterm's own; DSH's terminal code contains no composition* handling for the terminal surface — the only composition-aware code in that package is the title rename input (TerminalTitle.tsx, which skips Enter while isComposing is true). If the terminal's hidden textarea loses focus, is covered by an overlay, or the key handler swallows keyCode 229 / composition*, the IME never commits. Related reports exist for the chat input (#4233, #1417, #6138) but not for the terminal.

Impact

Any CJK (and generally non-ASCII) user who works in a directory with a localized name sees a mangled prompt, and cannot type their language into the terminal at all. The second one makes the terminal unusable for them, not just ugly.

Ask

• Problem 1: is the terminal shell supposed to receive a UTF-8 locale (and would you accept passing one, or a per-row env map)? If the escaping is expected on Windows, is there a recommended shell-side setup?
• Problem 2: is IME input in the sidebar terminal supposed to work today? If yes, any pointer to what handles composition there would help us aim the report.

Happy to run anything on Windows, or to provide the exact pty script and raw byte dumps.

[KannaKuron]

Update — the reporter ran the Windows read-outs: the locale hypothesis is ruled out.

Windows 11, Git Bash via the terminal, same CJK workspace:

• LANG= (empty), LC_ALL= (empty)
• LC_CTYPE="C.UTF-8", and LC_NUMERIC / LC_TIME / LC_COLLATE / LC_MONETARY / LC_MESSAGES are all C.UTF-8

So the shell is not in a C/POSIX locale, yet the prompt still escapes the CJK bytes. That matches our local pty reproduction, where LC_ALL=C, LC_ALL=en_US.UTF-8 and LC_ALL=C + TERM=dumb all produced the identical escaping.

Next read-out we are asking for (this should separate "bash's own cwd expansion" from "the terminal renderer"):

cd ~/Documents/<workspace-with-cjk>
PS1='$PWD> '     # direct variable expansion
PS1='\w> '       # bash's own \w expansion

If $PWD renders correctly while \w escapes, the escaping happens inside bash's prompt expansion rather than in the renderer — which would make the PS1 value set by the environment relevant (packages/terminal/terminal-bash/src/index.ts sets TERM/PS1/PROMPT_COMMAND but no locale variables), and would give both sides a concrete fix (avoid \w, or pass a UTF-8 locale).

[KannaKuron]

Correction + next narrowing steps (from the reporter's own control runs)

Withdrawing one explanation I gave above. I wrote that the escaping happens where the shell expands \w under a non-UTF-8 locale. That is not supported by the reporter's runs on the affected machine, and I also could not reproduce it on macOS. The corrected state of the investigation:

What the affected machine shows

check	reading
echo $LANG / echo $LC_ALL	empty
LC_CTYPE and friends	C.UTF-8
PS1='$PWD> '	CJK path renders correctly
PS1='\w> '	CJK path renders correctly
the terminal's default prompt	CJK path appears as M-… escapes

So: not the locale, and not \w on its own. The trigger has to be one of the other components of the default prompt.

One thing worth knowing before anything else

DSH's own shell prompt is plain ASCII: CONTROLLED_PROMPT = 'dsh> ' (packages/terminal/terminal-bash/src/sanitize.ts), exported into the child as PS1 and re-asserted before every prompt through PROMPT_COMMAND (packages/terminal/terminal-bash/src/index.ts, childEnvironment()). A prompt that displays the working directory is therefore not the prompt DSH sets — it comes from the shell's own startup files (Git Bash's default prompt / a user PS1). That is where the escaping must be, which also means this may be a Git Bash/readline-side rendering question rather than a DSH-side one (which is exactly what the control below decides).

Two commands that narrow it down (paste as-is)

Only colours:

PS1='\[\033[33m\]\w\[\033[0m\]> '

Only the OSC window-title sequence:

PS1='\[\033]0;\w\007\]\w> '

Whichever of the two starts showing M-… isolates the class of sequence responsible. (The Git Bash default prompt contains both — an OSC title built from $PWD, and ANSI-coloured \w inside \[ \] — which is why the plain-text tests above pass while the default prompt does not.)

Please also paste the two readings of the prompt actually in use on the affected machine:

echo "$PS1"

bash --version

The decisive attribution control (please run)

Open a native Git Bash window (minTTY), completely outside DSH, cd into the same CJK directory and look at the prompt:

• If it is escaped there too → this is Git Bash/readline-side rendering, independent of DSH, and belongs upstream in Git for Windows (or needs a shell-side workaround).
• If it is correct there but escaped inside DSH → the terminal layer (pty/TERM, or how the prompt bytes are redrawn) is part of the problem, and DSH is the right place to fix it.

What I could and could not reproduce locally

I re-ran a real-pty matrix on macOS (bash 3.2) with a CJK working directory: plain \w, \[\033[33m\]\w\[\033[0m\], \[\033]0;\w\007\]\w, an OSC title built from $PWD, and a Git-Bash-shaped default prompt — none of them escaped, under LC_ALL=C or a UTF-8 locale. So the ANSI/OSC hypothesis is not confirmed by me; it is the leading candidate given the reporter's controls, and the two commands plus the native-Git-Bash run above are what would settle it on Windows.

Happy to help interpret whatever those four commands print.

[KannaKuron]

Where the prompt actually comes from: two different terminal paths

Following up on the correction above — I traced which code sets a prompt, and the answer explains why the prompt in the sidebar terminal is the shell's own one rather than DSH's controlled prompt.

dsh> belongs to the agent-facing shell tool, not the sidebar terminal

• packages/terminal/terminal-bash registers a backend into the terminal registry: ctx.terminals.registerBackend(new BashTerminalBackend(...)) (terminal-bash/src/index.ts, apply()), and its childEnvironment() (terminal-bash/src/index.ts) sets TERM: 'dumb', PS1: CONTROLLED_PROMPT ('dsh> ', terminal-bash/src/sanitize.ts) and re-asserts that PS1 before every prompt through PROMPT_COMMAND.
• The consumers of that registry are the agent shell tools (packages/shell/tool-bash-persistent, packages/shell/tool-pwsh-persistent), i.e. dsh> is the prompt of the persistent shell an agent drives.

The web sidebar terminal is a different path and sets no prompt at all

• packages/api/terminal-controller creates a terminal with
  subprocess.spawnTerminal({ argv: [shell.path, ...shell.args], cwd, cols, rows, terminalType: 'xterm-256color', env: { DSH_SESSION_ID: agent.id } })
  (terminal-controller/src/index.ts, spawn()), and subprocess-local adds only TERM: spec.terminalType (subprocess-local/src/index.ts, spawnTerminal()).
• So on this path there is no PS1, no PROMPT_COMMAND and no locale — by construction. The prompt you see is the shell's own (Git Bash's PS1, which its startup files install, including the OSC window title and coloured \w).
• This also means a third-party terminal takeover did not bypass a controlled prompt: the sidebar terminal never had one. A plugin can only choose which shell runs, through the official shell.path / shell.args config of the same row (terminal-controller config schema), which is what those fields are for.

Consequence for this report

• The escaping is happening inside/around that shell-side prompt string, not in a DSH-controlled prompt, and not in DSH's renderer (the renderer strips escapes for its own bookkeeping only on the agent path).
• Would it make sense for the web sidebar terminal to also receive a normalized environment (a prompt it controls, and/or a UTF-8 locale), or is showing the user's own shell prompt intentional? That is the design question this report is really asking.

Readings that would still settle it

1. echo "$PS1" — on the affected machine, in the DSH terminal.
2. bash --version.
3. PS1='\[\033[33m\]\w\[\033[0m\]> ' (colours only) — escaped or not?
4. PS1='\[\033]0;\w\007\]\w> ' (OSC window title only) — escaped or not?
5. Native Git Bash window (minTTY), same CJK directory, no DSH involved — is its prompt escaped too? Same result ⇒ Git Bash/readline side; correct there but escaped inside DSH ⇒ the terminal layer is involved.

Status: still not confirmed which of those two it is; the code path above only rules out DSH's controlled prompt as the source.