fix(editor): two Monaco defaults that only hurt non-Latin scripts (#393) - #546
Merged
Conversation
…scripts Both come out of the audit in #393. Neither is a bug in Markpad — they are Monaco defaults that are right for code in English and wrong for prose in a script that behaves differently. ## The space bar under a CJK IME is outlined #462 turned off `unicodeHighlight.ambiguousCharacters` and closed #186 and #94 with it. The boxes came off the fullwidth punctuation and stayed on U+3000 IDEOGRAPHIC SPACE, which is what the space bar produces under every Chinese and Japanese IME — the same reporters, the same typing, the same hover offering an **Adjust settings** button that leads nowhere in standalone Monaco. That character is caught by `invisibleCharacters`, a different flag, whose 465-entry set cannot be narrowed by `allowedLocales`: `strings.js` getData() flattens every locale bucket into one set, so U+3000 is in it unconditionally. Only `allowedCharacters` reaches it. The flag itself stays on, and that is deliberate. An NBSP after `-` stops a list from parsing, and nobody types one on purpose. Worth recording that the comment justifying it was wrong in the other direction: it claimed the highlighter "never fires on something typed on purpose", and U+3000 is exactly that. ## Word-wise navigation treats a Chinese clause as one word ⌥←/→, double-click-to-select and ⌥⌫ split on `wordSeparators`. A language that puts no spaces between its words supplies none, so the whole clause is one word: ⌥→ crosses the sentence, double-click selects it, ⌥⌫ deletes it. `wordSegmenterLocales` switches on `Intl.Segmenter`, which knows where the words are. It reads like a whitelist and is closer to a switch. ICU dispatches its dictionary breaking by SCRIPT, not by this list: Thai, Khmer, Lao, Burmese and Tibetan all segment without being named, and `['zh']`, `['ja']` and `['zh','ja']` produce identical output — including on Han text, where the two dictionaries might have been expected to differ. Pinned in a test, because the obvious tidy-up is to trim the list to match the comment, which would change nothing while making the code claim something false. Space-delimited languages lose nothing: Korean, Vietnamese, Russian and English segment word-for-word identically to splitting on whitespace. That is why this can be on for everyone rather than keyed to the UI language — which would be the wrong key anyway, since it is the document's script that decides, not the language of the menus. ## Tests Driven through Monaco's own code rather than by asserting the options are spelled correctly: `UnicodeTextModelHighlighter.computeUnicodeHighlights` for the first, `getMapForWordSeparators` — the function `wordOperations.js` and `cursorWordOperations.js` both call — for the second. Each fixture is also checked to still reproduce the original bug once the fix is taken away, so a passing test cannot mean the fixture stopped exercising anything. Verified in Node's ICU. The dictionary breaking is standard across ICU builds, but it was not exercised in WKWebView, WebView2 or WebKitGTK.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two items from the audit in #393. Neither is a bug in Markpad — both are Monaco
defaults that are right for code in English and wrong for prose in a script
that behaves differently.
The space bar under a CJK IME is outlined
#462 turned off
unicodeHighlight.ambiguousCharactersand closed #186 and #94with it. The boxes came off the fullwidth punctuation and stayed on U+3000
IDEOGRAPHIC SPACE — which is what the space bar produces under every Chinese
and Japanese IME. Same reporters, same typing, same hover offering an Adjust
settings button that leads nowhere in standalone Monaco.
That character is caught by
invisibleCharacters, a different flag, andallowedLocalescannot narrow it:strings.jsgetData()flattens everylocale bucket into one 465-entry set, so U+3000 is in it unconditionally. Only
allowedCharactersreaches it.The flag itself stays on, deliberately — an NBSP after
-stops a list fromparsing, and nobody types one on purpose. Worth recording that the comment
justifying it was wrong in the other direction: it claimed the highlighter
"never fires on something typed on purpose", and U+3000 is exactly that.
Word-wise navigation treats a Chinese clause as one word
⌥←/→, double-click-to-select and ⌥⌫ split on
wordSeparators. A language thatputs no spaces between its words supplies none, so the whole clause is one
word: ⌥→ crosses the sentence, double-click selects it, ⌥⌫ deletes it.
wordSegmenterLocalesswitches onIntl.Segmenter, which knows where thewords are.
这是一个用于测试的段落goes from one word to seven.It reads like a whitelist and is closer to a switch. ICU dispatches its
dictionary breaking by SCRIPT, not by this list:
and
['zh'],['ja']and['zh','ja']produce identical output — including onHan text, where the two dictionaries might have been expected to differ. Pinned
in a test, because the obvious tidy-up is to trim the list to match the comment,
which would change nothing while making the code claim something false.
Space-delimited languages lose nothing. Korean, Vietnamese, Russian and English
segment word-for-word identically to splitting on whitespace — which is why
this can be on for everyone rather than keyed to the UI language. That would be
the wrong key anyway: it is the document's script that decides, not the
language of the menus.
Tests
Driven through Monaco's own code rather than by asserting the options are
spelled correctly —
UnicodeTextModelHighlighter.computeUnicodeHighlightsforthe first,
getMapForWordSeparators(the functionwordOperations.jsandcursorWordOperations.jsboth call) for the second. Each fixture is alsochecked to still reproduce the original bug once the fix is removed, so a
passing test cannot mean the fixture stopped exercising anything.
Verified
By hand on macOS: the IME space is no longer outlined, ⌥←/→ and double-click
move by word in Chinese prose, and English in the same document behaves exactly
as before.
The segmentation results above were measured in Node's ICU. Dictionary breaking
is standard across ICU builds, but it was not exercised inside WKWebView,
WebView2 or WebKitGTK.