Keep a ZWJ emoji sequence in one cell - #26
Merged
Conversation
A ZWJ sequence is one grapheme cluster and one glyph, but every component after the first opened its own cell. Measured before this change: a woman-technologist claimed 4 cells and a family claimed 8, of which 2 and 6 respectively were left blank. A renderer that shapes the cluster into one glyph — which is the right thing to do — then draws it at the first cell and leaves a run of empty cells after it, so emoji appear with wide gaps and everything after them sits in the wrong column. The ZWJ itself already merged, because IsCombiningCharacter lists it. What did not merge was the character AFTER it: an ordinary emoji, passing no combining test of its own, so Print gave it a new cell. The position of a merged ZWJ is now remembered, and a character printed at exactly that position continues the cluster. A position rather than a flag, so it invalidates itself. Anything that moves the cursor between the ZWJ and the next character — an escape sequence, a newline, a cursor address — leaves it pointing somewhere the next Print is not, and the continuation is dropped instead of joining two unrelated characters. There is a test for that case specifically; it passes either way, and is there to keep the fix from growing that failure later. Five tests. Four fail against the current behaviour with the counts above; 733 pass with the change.
tomlm
added a commit
to tomlm/Iciclecreek.Avalonia.Terminal
that referenced
this pull request
Aug 24, 2026
* Let the Example demo use a font a terminal can render Example hardcoded FontFamily="Cascadia Mono" on all three terminals. That font does not exist on macOS, so Avalonia fell back to the system UI font and the demo rendered a terminal in a proportional face — glyphs drifting out of their columns, which is the exact failure TerminalView.DefaultFontFamily exists to prevent. Removing the override lets that default apply. It already carries a cross-platform monospace fallback chain, so every platform gets something appropriate without the demo naming a font at all. * Make Shift+navigation behave the way a text field does Fixes #63, and the four further gaps that fixing it exposed. Reported: Ctrl+Shift+Left moved to the word boundary but lost the selection. Control|Shift matched neither gate — TryExtendKeyboardSelection wanted exactly Shift, the word-motion path from #49 wanted exactly Alt or Control — so it fell through to the blanket selection-clear and then sent the modified-cursor sequence. The shell acted on that and moved by a word. Both halves were firing, in the wrong place. Ctrl+Shift and Alt+Shift now extend by a word. Alt because on macOS Ctrl+arrow belongs to Mission Control, so Option is the only gesture that reaches the app. Cmd+Left and Cmd+Right are line-start and line-end there, and with Shift select to that edge. They did nothing at all before, swallowed by the Meta passthrough — a Mac keyboard has no Home or End, so neither was reachable. They send what Home and End send, so this is an alias rather than a second code path. The caret follows the selection's moving edge, as it does in every text field. Only where it is DRAWN changes: the shell still owns the real cursor and is never told, because the buffer position is where the shell writes next and moving it would put output in the wrong place. Typing over a selection replaces it, and Backspace or Delete removes it. The view cannot edit the line — the shell owns it — so the selection becomes the keystrokes that would have removed it. Backspace in both directions rather than Delete for a forward selection: forward-delete is not reliably bound, and zsh with no rc does not know ESC[3~ — it swallows the ESC[3 and TYPES the tilde. The deletion and the replacing character go out as ONE write, because sending them separately loses the race against the next keystroke: "there" typed over a selection arrived as "heret" against a real bash. Only a keyboard selection is editable that way. A mouse selection can sit anywhere on screen, including the scrollback, with no fixed relationship to the shell's cursor, so typing over one clears it without deleting. The alternate buffer is excluded throughout: a full-screen app owns its own editing. A selection is bounded at both ends. It stops where the editable input starts, because the prompt is not the user's to edit and readline will not delete it either — so a selection covering it could not be replaced. It stops at the end of the input too: a terminal grid is padded to full width with blanks, and without a ceiling Shift+Right walks into the empty rest of the screen a cell at a time. Where the input starts is taken from OSC 133 when the shell reports it. XTerm.NET already parses those markers, so this subscribes rather than adding a parser; B is emitted straight after the prompt is drawn, so the cursor is exactly where input begins. Measured against a 12-character prompt, the marker gives column 12. Nothing is required of the shell: bash 3.2 as shipped on macOS emits nothing, and the heuristic still applies there, sampling at the first keystroke after the shell moves to a new row. Wide glyphs are one character throughout. A CJK or emoji cell is followed by a width-0 placeholder whose content is empty, which read as whitespace: the line-end scan stopped on the glyph's first column and word movement stopped between its two halves, so a selection covered half a character. Known limit, documented at the call site: a trailing space the user typed is not counted as input. The emulator pre-fills the grid with spaces and a typed space is identical to an unwritten cell in every field — same content, width and code point — so distinguishing them needs the buffer to record that a cell was written, which is a change in XTerm.NET. Also of note for emoji: XTerm.NET gives every component of a ZWJ sequence its own cell pair, so a family emoji claims eight cells for a glyph two wide. Selection over those stays odd until tomlm/XTerm.NET#26 lands; this change makes plain wide characters behave. Tests assert what reaches the pty rather than internal state, and the ones that can only be true end to end drive a real bash. 217 pass. * Use XTerm's word definition, and drop a doc block from the wrong method TryExtendKeyboardSelection carried two stacked XML doc comments: its own, and a copy of TakeKeyboardSelectionDeletion's pasted below it. The copy was also stale — it described a selection made rightwards as "that many Deletes", which is the approach this PR deliberately rejects, since zsh with no rc file does not know ESC[3~ and types the tilde instead. So the wrong method documented an abandoned design, which is how a later reader ends up "fixing" the code to match the comment. Removed, and the surviving block now covers the word and line-edge chords it gained rather than only Shift. WordBoundary also invented a second definition of a word. Whitespace-delimited is defensible on its own — it is what readline's ESC-b and ESC-f do — but SelectionManager.IsWordChar already answers the same question for double-click expansion, and letters, digits and underscore is not the same answer. The terminal told you "foo-bar" was one word if you held Ctrl+Shift and "bar" if you double-clicked, over the same text, in the same line. Snapped to XTerm's definition. Placeholder cells behind a wide glyph are still never separators; that part was right and is why the scan does not split a CJK character in half. The existing word tests all use "hello world", which the two definitions agree on, so none of them could see the difference. The new test uses a hyphen and asserts both halves of the claim: the keyboard gesture stops at it, and a Word-mode selection over the same text says the same thing. It fails on the old definition with "foo-bar" against "bar". 192 tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B3g78sC3oXs7TQq7pgw6xE * Let Ctrl+arrow reach a shell that reads Win32 input records Ctrl+Left and Ctrl+Right have meant word-motion since #49, which sends ESC-b and ESC-f because no default keymap binds the modified-cursor sequence. That is right for zsh, bash, fish and PSReadLine's emacs mode. It is wrong for cmd.exe, which binds neither. cmd.exe turns on win32-input-mode as it starts — CSI ?9001h, the third sequence in any session it runs. In that mode it wants the key event itself, and it already moves by word when it gets one. The translation ran first, so what reached it was a sequence it ignores: on Windows the chord did nothing at all, which is what prompted this. The translation now stands aside when that mode is on and the key falls through to the Win32 record path a few lines below. Nothing changes on Linux or macOS, where the mode is never enabled and ESC-b remains the right answer. Also moved a const that had been left between two methods at the wrong indentation, and confirmed no raw control characters ended up in the source. Four tests: Ctrl+arrow still sends ESC-b/ESC-f to a VT shell, and does NOT under win32-input-mode. Verified the second pair fails without the guard — "b" and "f" arrive where a key record should. 196 pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B3g78sC3oXs7TQq7pgw6xE --------- Co-authored-by: Tom Laird-McConnell <thermous@iciclecreek.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
tomlm
approved these changes
Aug 25, 2026
pull Bot
pushed a commit
to IvanJosipovic/XTerm.NET
that referenced
this pull request
Aug 25, 2026
Fixes tomlm#28. A flag emoji did not render oddly, it disappeared: both halves were stored with width 0, the cursor never advanced over them, and whatever came next overwrote them. "US flag" followed by "XXX" left the XXX at column 0 with no flag at all, and the rest of the line misaligned by two columns. The pairing test lived in GetStringCellWidth, which is called once per printed character. The two halves of a flag arrive in separate Print calls, so the local count was always 1, always odd, and the width was always 0 -- the pair branch could never fire. It would only have worked if a whole flag were ever measured in a single call, which it never is. There is a second defect behind that one: even measured together, the pair added 1 where a flag is 2 columns wide. So fixing only the pairing would have left a flag drawn in a single column. The width routine now needs no case for indicators at all. Each is worth 1, so a pair comes to 2 -- the flag -- and a lone one comes to 1, which is the boxed letter it renders as. Pairing moves to Print, where the state to do it can survive the call, mirroring _zwjContinuation from tomlm#26. Two regional indicators are one grapheme cluster under UAX tomlm#29, so this is the same class of thing that change addressed rather than a special case. Positions rather than a flag, for the same reason as the ZWJ version: a second indicator pairs only when it lands exactly where the first left the cursor, so a cursor address or a newline in between leaves two separate characters, which is what they are. Eleven tests. Seven fail without the fix; the four that pass are the guards for the ZWJ and skin-tone clusters that share the width routine. They cover the reported case, a lone indicator, three and four in a row, the halves arriving in separate writes, a cursor move and a newline between them, and a pair with one column left -- which stays two characters rather than becoming a wide cell hanging off the edge. 744 tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B3g78sC3oXs7TQq7pgw6xE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Branched from current
main, one commit.A ZWJ sequence is one grapheme cluster and one glyph, but every component after the first opened its own cell. Measured against
mainbefore the change:世😀👩💻(woman + ZWJ + laptop)👨👩👧👦(four people, three ZWJs)A renderer that shapes the cluster into one glyph — which is the right thing to do — then draws it at the first cell and leaves a run of empty cells after it. Emoji appear with wide gaps, and everything after them sits in the wrong column.
Cause
The ZWJ itself already merged:
IsCombiningCharacterlistsU+200D. What did not merge was the character after it — an ordinary emoji, passing no combining test of its own, soPrintgave it a new cell.Fix
The position of a merged ZWJ is remembered, and a character printed at exactly that position continues the cluster.
A position rather than a flag, so it invalidates itself. Anything that moves the cursor between the ZWJ and the next character — an escape sequence, a newline, a cursor address — leaves it pointing somewhere the next
Printis not, and the continuation is dropped rather than joining two unrelated characters.Tests
Five. Four fail against current
mainwith exactly the counts tabulated above:That last one passes either way. It is there to keep the fix from growing that failure later, not to prove it today.
733 tests pass.
Where this came from
Found while working on keyboard selection in
Iciclecreek.Avalonia.Terminal, which counts cells: selecting a family emoji needed 8 cells for a glyph 2 wide. The rendering side was already handled downstream by shaping the cluster into one glyph — this is the other half.🤖 Generated with Claude Code