"Arabic/RTL Text Support with BiDi Rendering" #9774
arabic text not displays disconnected
i have a working implementation ensures text renders
usage in the terminal for reading other language files even the " cat " command works great and can show the language in a right way . the integration i think will work with other languages but i have test only the arabic not the other languages i have some issues with the selecting the text with mouse in arabic ( RTL ) position . working on that I loved Ghostly and the community behind it so hopefully this will be in the " main repo !! " branch : https://github.com/commandlinetips/ghostty/tree/main
echo "Arabic: مرحبا (hello) - letters should connect"
- this showing the before and after
- this showing the arabic language in terminal using cat command .
- this showing writeeing arabic in claude code cli .
- this showing the terminal test with different position for the text .
|
Replies: 5 comments 10 replies
|
there is already an accepted issue related to RTL (meaning PRs are welcome) I think before you open any PRs benchmarks would be appreciated to make sure LTR doesnt have a large performance impact, there are some benchmarks in the repo you can run im not sure how you are supposed to run them though |
|
Closing as there doesn't seem to be much activity here, like I noted above BIDI / RTL is accepted just keep in mind smaller PRs are generally better. Think about the cost of supporting RTL (it shouldnt greatly impact LTR performance) and unit tests would also be useful. |
|
I have update the Arabic terminal and it's working perfect even with OpenCode |
|
I've been building Bayan, an Arabic-first terminal in Rust, and hit most of these problems from the other direction. Sharing the edge cases that cost me the most, in case they're useful — especially around selection, since that's listed as still open here. Worth stating the asymmetry up front: Bayan solves the inverse transform. Claude Code (Ink) applies BiDi itself on Windows and emits Arabic already reordered into visual order, so my BiDi module reverses UAX#9 rule L2 to recover logical order before the shaper sees it. That's the opposite of what you're doing. But the segmentation decisions underneath are the same either direction — you have to agree with the algorithm about where runs begin and end, and that's where I lost the most time. Selection: the thing that made it tractableThe rule that unlocked this for me: on an RTL row, logical column ≠ visual x, and the cursor must resolve through the shaped layout rather than The second half is that copying must emit logical order, not visual order. A selection that looks right on screen and pastes reversed is the failure mode users report as "selection is broken," and it's really a copy-path bug. Worth treating render-order and clipboard-order as two separate representations from the start. Segmentation edge casesThese are the ones my test suite pins, all of which I got wrong at least once: 1. Arabic-Indic digits are not Arabic letters. 2. Combining marks must travel with their base character. Tashkeel, shadda, tanwin, superscript alef. A plain character-level reverse puts the mark before the letter it belongs to, and the result renders as garbage that's hard to trace back to the reversal step. The fix is to group each base with its trailing combining marks into a cluster first, then reorder clusters. Ranges I cover: 3. Punctuation joins an LTR island only when another LTR character follows it. This is the subtle one. Paths, filenames and version strings inside an Arabic sentence have to survive intact — 4. Base direction decides which algorithm runs at all. A predominantly-Arabic line and an English line quoting an Arabic phrase need different treatment — whole-line handling versus per-run handling. I classify by comparing Arabic-letter count against Latin-letter count on the line. Mixed lines that sit near the boundary are where bugs hide. On the "which programs need this" questionNot directly your issue, but adjacent and it bit me: the program running inside the terminal matters. Ink-based TUIs pre-apply BiDi on Windows; vim, less and htop emit logical order and must be left alone. If a terminal ever ships a compatibility mode for the pre-reordered case, gating it on the invoked program — basename, extension stripped, runner prefixes like Happy to go deeper on any of these. The implementation is Zig here and Rust on my side so the code doesn't transfer, but the test cases do. A few of mine, as Happy to write out a fuller fixture set with expected outputs if that'd help build a test corpus. (Edited to add links — the repo was private when I first posted this. The transform and all its edge cases live in |
|
Following up on the fixture-set offer — here's the corpus, inline so it's usable without chasing a link. How it was generated, because that's the part that makes it worth trusting: I added a temporary emitter to my BiDi module that constructed each pair using the real implementation functions and asserted the pair round-trips before printing it, then removed the emitter. The codepoint fields were re-derived programmatically from the strings afterwards. Nothing here was transcribed by hand — Arabic string surgery by eye is exactly where an error hides and never gets spotted, especially in a fixture file, because your editor reorders the text on display and you cannot see what it actually says. How to read it. Each case is two representations of the same text: The Codepoints are given explicitly for every case for the display-reordering reason above. {
"$comment": [
"Language-agnostic BiDi test corpus, generated from Bayan's src/bidi.rs and",
"self-validated: every pair below was proven to round-trip by the implementation",
"that produced it, not transcribed by hand.",
"",
"DIRECTION. Each case is a pair of representations of the SAME text:",
" logical - the order the bytes arrive in / sit in memory",
" visual - the order a UAX#9-applying renderer puts on screen",
"A terminal that applies BiDi maps logical -> visual.",
"A terminal undoing a client that already applied BiDi maps visual -> logical.",
"The pairs are useful either direction; only the arrow changes.",
"",
"Codepoints are given explicitly because editors and terminals reorder RTL text",
"on display, which makes eyeballing a fixture file unreliable."
],
"cases": [
{
"name": "arabic_word",
"pins": "Baseline whole-line reversal for a pure-Arabic line.",
"logical": "مرحبا",
"visual": "ابحرم",
"logical_codepoints": "0645 0631 062D 0628 0627",
"visual_codepoints": "0627 0628 062D 0631 0645"
},
{
"name": "arabic_sentence",
"pins": "Word order reverses as well as letter order; the space is preserved between the reordered words.",
"logical": "مرحبا بالعالم",
"visual": "ملاعلاب ابحرم",
"logical_codepoints": "0645 0631 062D 0628 0627 0020 0628 0627 0644 0639 0627 0644 0645",
"visual_codepoints": "0645 0644 0627 0639 0644 0627 0628 0020 0627 0628 062D 0631 0645"
},
{
"name": "combining_tanwin",
"pins": "THE CLUSTER RULE. U+064B (tanwin) must stay immediately after its base U+0627 (alef) in BOTH representations. A codepoint-level reverse emits 064B before 0627 and renders as garbage. Reorder grapheme clusters, not codepoints.",
"logical": "معاً",
"visual": "اًعم",
"logical_codepoints": "0645 0639 0627 064B",
"visual_codepoints": "0627 064B 0639 0645"
},
{
"name": "arabic_indic_digits",
"pins": "THE DIGIT RULE. U+0661 U+0667 U+0665 keep their internal order in both representations - Arabic-Indic digits are weak LTR, not RTL letters. An 'is this Arabic?' test that range-checks the whole 0600-06FF block sweeps digits in and reverses every number on the line.",
"logical": "رقم ١٧٥",
"visual": "١٧٥ مقر",
"logical_codepoints": "0631 0642 0645 0020 0661 0667 0665",
"visual_codepoints": "0661 0667 0665 0020 0645 0642 0631"
},
{
"name": "ltr_island_path",
"pins": "THE ISLAND RULE. 'config.txt' survives intact inside an RTL-base line: the dot is absorbed into the Latin run because a Latin character follows it. Absorb punctuation unconditionally and a TRAILING period flips ('01.' becomes '.01'), so the join needs a lookahead.",
"logical": "افتح ملف config.txt رجاء",
"visual": "ءاجر config.txt فلم حتفا",
"logical_codepoints": "0627 0641 062A 062D 0020 0645 0644 0641 0020 0063 006F 006E 0066 0069 0067 002E 0074 0078 0074 0020 0631 062C 0627 0621",
"visual_codepoints": "0621 0627 062C 0631 0020 0063 006F 006E 0066 0069 0067 002E 0074 0078 0074 0020 0641 0644 0645 0020 062D 062A 0641 0627"
},
{
"name": "ltr_base_arabic_tail",
"pins": "THE BASE-DIRECTION RULE. This line has more Latin than Arabic, so the base is LTR: only the Arabic RUN reverses, in place, and the Latin is untouched. Contrast arabic_sentence, where the whole line reverses. Note U+061F (Arabic question mark) travels with the Arabic run.",
"logical": "What are you working on مساعدتك؟",
"visual": "What are you working on ؟كتدعاسم",
"logical_codepoints": "0057 0068 0061 0074 0020 0061 0072 0065 0020 0079 006F 0075 0020 0077 006F 0072 006B 0069 006E 0067 0020 006F 006E 0020 0645 0633 0627 0639 062F 062A 0643 061F",
"visual_codepoints": "0057 0068 0061 0074 0020 0061 0072 0065 0020 0079 006F 0075 0020 0077 006F 0072 006B 0069 006E 0067 0020 006F 006E 0020 061F 0643 062A 062F 0639 0627 0633 0645"
},
{
"name": "no_arabic_untouched",
"pins": "A line with no Arabic must be returned byte-identical - no reordering, no normalisation. Guards against a transform that runs unconditionally.",
"logical": "pure english",
"visual": "pure english",
"logical_codepoints": "0070 0075 0072 0065 0020 0065 006E 0067 006C 0069 0073 0068",
"visual_codepoints": "0070 0075 0072 0065 0020 0065 006E 0067 006C 0069 0073 0068"
}
],
"multiline_case": {
"$comment": "Selection/copy spans lines with different base directions. Each line classifies independently; a whole-block transform is wrong.",
"visual_block": "ابحرم\nplain english\nWhat are you working on ؟كتدعاسم",
"logical_block": "مرحبا\nplain english\nWhat are you working on مساعدتك؟"
},
"program_gating": {
"$comment": [
"Not BiDi, but adjacent and it caused real bugs. If a terminal ships a mode for",
"clients that pre-apply BiDi, it must key on the INVOKED PROGRAM - basename,",
"extension stripped, runner prefixes unwrapped - not on a substring of the",
"command line. Otherwise 'vim claude.py' silently enables it.",
"true = the client pre-applies BiDi; false = the client emits logical order."
],
"matches": [
"claude",
"claude --continue",
"npx claude",
"\"C:\\tools\\claude.exe\" chat",
"& C:\\Users\\x\\claude.cmd"
],
"does_not_match": [
"vim claude.py",
"less claude.txt",
"git log claude",
"htop",
""
]
}
}```
Happy to add cases if there are shapes you are unsure about — RTL inside RTL, neutrals between two Arabic runs, mixed digits, whatever the implementation is currently guessing at.
*(Edited to add a link — the repo was private when I posted this. The corpus now lives at [`tests/fixtures/bidi_cases.json`](https://github.com/jaqop/Bayan/blob/main/tests/fixtures/bidi_cases.json), with a [README](https://github.com/jaqop/Bayan/blob/main/tests/fixtures/README.md) explaining each case, so you can fetch it rather than copy it out of this comment. The JSON above is identical.)* |




there is already an accepted issue related to RTL (meaning PRs are welcome)
#1442
I think before you open any PRs benchmarks would be appreciated to make sure LTR doesnt have a large performance impact, there are some benchmarks in the repo you can run im not sure how you are supposed to run them though