Arabic isn't properly rendered in Neovim until a newline is inserted

[pro-anon]

The letters here aren't properly rendered

Only after inserting a new line they get fixed

This issue doesn't happen when typing directly into the terminal

[arakiken]

Sorry for replying late.
Thanks for your report, but I couldn't reproduce this problem in my environment. (neovim 0.8.2 with empty ~/.config/nvim)
Will you tell me your neovim version and configuration?

[pro-anon]

v0.8.2
I've tried it without any config, the issue is still the same

[pro-anon]

I'm running it on arch. Installed mlterm from here https://aur.archlinux.org/packages/mlterm

[arakiken]

I tested plain mlterm(3.9.2) and plain neovim(0.8.2) without any configuration, but I haven't reproduced it yet.
I'm not sure, but does ./configure without --enable-optimize-redrawing option in building mlterm improve it?

[pro-anon]

Nope, still the same. Maybe it's an issue with the font? What font do you use?

[pro-anon]

I've noticed that, in addition to inserting a new line, inserting ﻻ re-renders the line properly

[arakiken]

I use "Not Sans Arabic" font to show arabic characters, but I think it doesn't caused by font configuration.
Sorry but I haven't found out the cause yet.

[holytrousers]

I don't think the bug is in mlterm.
Try to run nvim with the --clean option.
Looks like it's happening when you have set cursorline and the cursor is highlighting the arabic text. This can happen if you are using variable width fonts.
Make sure you have monospace fonts with Arabic glyphs installed and you have configured mlterm to use them (Try Dejavu Sans Mono)
if you are using xft, you have to specify which fonts to use for Arabic glyphs. I have tried to set all Arabic Unicode ranges to use Deja Vu Sans Mono and managed to make it work :

DEFAULT = Fantasque Sans Mono, 12
ISO10646_UCS4_1_FULLWIDTH = Noto Sans Mono CJK JP, 10
U+10a60-10a7f=DejaVu Sans Mono, 10
U+10a80-10a9f=DejaVu Sans Mono, 10
U+600-6ff=    DejaVu Sans Mono, 10
U+750-77f=    DejaVu Sans Mono, 10
U+8a0-8ff=    DejaVu Sans Mono, 10
U+1ee00-1eeff=DejaVu Sans Mono, 10 
U+10e60-10e7f=DejaVu Sans Mono, 10
U+fe70-feff=  DejaVu Sans Mono, 10
U+fb50-fdff=  DejaVu Sans Mono, 10

[pro-anon]

@holytrousers
I've tried the --clean flag with neovim and DejaVu Sans Mono, but the issue got worse. Now I have to insert a new line, then delete it so that the line before it would get rendered correctly.

[holytrousers]

have you tried the mlterm-git package ?
the one you mentioned has been flagged as out of date

[holytrousers]

@pro-anon I have reproduced your problem by using the :set arabic option and actually writing the text. Previously i pasted arabic text from yamli.com into vim that's why i wasn't able to reproduce the problem.
Writing in arabic (using the "ara" keymap) inside the shell produces the same behaviour though. لا fixes it.
The same problem happens when writing in arabic inside the shell in other terminals xterm, alacritty.
Looks like Kitty implements bicon properly out of the box.
Running bicon in xterm and alacritty and writing in arabic inside the shell fixes the problem.
Writing arabic with the :set arabic option in vim works fine in xterm and alacritty without running bicon.
Running bicon in mlterm doesn't fix the problem though.

As a temporary workaround try pressing Control+L in normal mode inside vim, that will refresh the screen and render the ligatures properly (inoremap to refresh every time you switch to normal mode )
Another trick i've found fixes ligatures : try to insert a letter, then move backward one letter, so that the cursor is inserting text before that letter. ligatures seem to work then.
Looks like it's a problem with the bicon implementation in mlterm. @arakiken might give an insight on this

[holytrousers]

ps. add inoremap <Esc> <Esc><C-L> to your vim configuration

[holytrousers]

NB. The trick with prepending a letter before the inserted text works only partially though. Baseline Ligatures work then but the letters at the end of the word don't change their shape.

[holytrousers]

I think we have narrowed down the problem, try :

• Start vim
• :set arabic
• Begin typing in latin by pressing ctrl-^ (from left-to-right)
• Switch back to arabic by pressing again ctrl-^ and write in arabic

Ligatures and end of word letters render properly.
Which means that the bug occurs only when the whole line is written from right-to-left, ie. the beginning of the line is on the right side of the window.

[holytrousers]

@pro-anon : your prompt is styled to align the text to the left side of the screen, that's why when you type arabic text in the shell it displays properly.
This confirms the previous conclusion : the problem occurs only when the whole line is RTL.

[holytrousers]

@pro-anon : i think enabling the Variable Width Column option solves the issue completely.
Can you confirm that ?

[holytrousers]

Another issue i have found with arabic script : when using :set arabic in vim, beginnings of lines always stay on the left side, and text is switched to the right side only after some text is written. On a new line, the cursor remains on the right side.
Other terminals don't seem to have this problem.

[pro-anon]

@holytrousers

I think we have narrowed down the problem, try :

* Start vim

* `:set arabic`

* Begin typing in latin by pressing `ctrl-^` (from left-to-right)

* Switch back to arabic by pressing again `ctrl-^` and write in arabic

Ligatures and end of word letters render properly. Which means that the bug occurs only when the whole line is written from right-to-left, ie. the beginning of the line is on the right side of the window.

Yes, that is exactly what I do set arabic. Sorry for not mentioning it before. And you were correct! Starting with Latin fixes it, <C-l> too but only in normal not insert mode.

[pro-anon]

I'm not sure what you mean here. If you mean the mlterm flag --varwidth then yes, it does fix it, but it looks very disorienting (I have used only one white space between every word, and no white space at the beginning of each line):

[holytrousers]

I'm not sure what you mean here. If you mean the mlterm flag --varwidth then yes, it does fix it, but it looks very disorienting (I have used only one white space between every word, and no white space at the beginning of each line):
Yes, either

• --varwidth
• use_variable_column_width = true to your ~/.mlterm/main
• use the configuration dialog by pressing Control + RMB and go to the font tab.

I think your issue is because you have set the varwidth option but forgot to set the font to monospace.
set it in the ~/.mlterm/vaafont
use DejaVu Sans Mono for the moment, and set it for the unicode blocks i've mentioned earlier.

[arakiken]

Thanks very much.
I can reproduce this problem. (enbugged over 5 years ago)
I'll fix it next weekend. Please wait.

[pro-anon]

Yes, either

* `--varwidth`

* `use_variable_column_width = true` to your ~/.mlterm/main

* use the configuration dialog by pressing `Control + RMB` and go to the `font` tab.

I think your issue is because you have set the varwidth option but forgot to set the font to monospace. set it in the ~/.mlterm/vaafont use DejaVu Sans Mono for the moment, and set it for the unicode blocks i've mentioned earlier.

Can you show an example of a vaafont config file? I've added this DEFAULT=DejaVu Sans Mono to my ~/.mlterm/font which fixed the Arabic font issue but messed up the rest:


@holytrousers

[holytrousers]

I believe that arabic text is not in DejaVu Sans Mono, it looks like some variable width font. I guess you are using cairo which switches automatically to the nearest font with necessary glyphs.
I don't use cairo because its rendering quality is ugly and xft looks much crispier, but that's not related to this issue.
Anyway i've set both aafont and vaafont to use these fonts

DEFAULT = Fantasque Sans Mono, 12
ISO10646_UCS4_1_FULLWIDTH = Noto Sans Mono CJK JP, 10
U+10a60-10a7f=DejaVu Sans Mono, 10
U+10a80-10a9f=DejaVu Sans Mono, 10
U+600-6ff=    DejaVu Sans Mono, 10
U+750-77f=    DejaVu Sans Mono, 10
U+8a0-8ff=    DejaVu Sans Mono, 10
U+1ee00-1eeff=DejaVu Sans Mono, 10 
U+10e60-10e7f=DejaVu Sans Mono, 10
U+fe70-feff=  DejaVu Sans Mono, 10
U+fb50-fdff=  DejaVu Sans Mono, 10

First line is for latin scripts, second line is for japanese and chinese, the rest are for arabic.
It works for me, i'm sure not all blocks are necessary, but i've copied all of those i've found in the unicode table.
Please try to set both files or at least the vaafont and report back if it works.

[pro-anon]

Much better now. Thanks @holytrousers

[arakiken]

Sorry for replying late.
I think that this following commit improves it.
f8c6b88

[pro-anon]

Works very well! Thanks!

Note though, it's better to remove use_variable_column_width = true from the user config because it had some issues like this:

variable_width_issue.mp4

[holytrousers]

Sorry for replying late. I think that this following commit improves it. f8c6b88

Thank you for fixing on this, glyphs render properly now !
The only thing that is still conflicting with vim's arabic input is mlterm's right to left implementation :
in vim, :set arabic moves the cursor to the right side on an new empty line, but it looks like mlterm moves it again to the opposite side thinking it's on the left side, then moves back to the right side upon entering arabic text.

[holytrousers]

reporting back from the front with some screenshots:

See how the cursor is positioned on a new line, it should be on the right side (as in other terminals)

[holytrousers]

To reproduce it, just type some arabic text and erase it with backspace and the erased letters will leave those traces. This happens on letters that extend down the baseline like س ي ن

To make this glitch disappear set line_space = 1 or greater.

[holytrousers]

The problem with refreshing positional letters drawing (that has been solved in f8c6b88 ) still persists when using backspace.
See this ب that draws as if stil connected to the next letter after it has been deleted:

[holytrousers]

this is another problem i've found with the automatic switch to the right side:
Here's ncmpcpp displaying some Umm Kulthum tracks, the artist tag being in arabic, in xterm and mlterm for comparison :

[arakiken]

The problem with refreshing positional letters drawing (that has been solved in f8c6b88 ) still persists when using backspace.
See this ب that draws as if stil connected to the next letter after it has been deleted:

Thanks. I fixed it.
3ff4d5c

[arakiken]

this is another problem i've found with the automatic switch to the right side: Here's ncmpcpp displaying some Umm Kulthum tracks, the artist tag being in arabic, in xterm and mlterm for comparison :

It is difficult for a terminal emulator to determine whether to display text right-aligned or left-aligned appropriately.

Changing bidi_mode option manually as follows makes mlterm show text the same as xterm with bicon.
$ mlcc bidi_mode left

[arakiken]

The only thing that is still conflicting with vim's arabic input is mlterm's right to left implementation : in vim, :set arabic moves the cursor to the right side on an new empty line, but it looks like mlterm moves it again to the opposite side thinking it's on the left side, then moves back to the right side upon entering arabic text.

mlterm simply determines right-aligned or left-aligned by a returned value of fribidi_log2vis() in each line.
It is difficult for mlterm to recognize whether the current mode of vim is arabic or noarabic.

YFI, if you want to make vim in mlterm work the same as in xterm, execute following commands which stop mlterm from processing BiDi and which stop vim from using BiDi feature of mlterm.
$ unset MLTERM (or :set notermbidi in vim)
$ mlcc use_ctl false
vimdoc: https://vim-jp.org/vimdoc-en/arabic.html

[pro-anon]

The problem with refreshing positional letters drawing (that has been solved in f8c6b88 ) still persists when using backspace.
See this ب that draws as if stil connected to the next letter after it has been deleted:

Thanks. I fixed it. 3ff4d5c

Are you sure the fix is working?

Does it work on your end, @holytrousers?

[holytrousers]

Thanks. I fixed it. 3ff4d5c

Great it works !
@pro-anon : it fixes glyphs' positional rendering after pressing backspace. From the example you've shown, i understand you are talking about the leftovers that remain after some text has been deleted and that have something to do with line spacing set to 0 ?
In that case no, that bug is still there.

[holytrousers]

this is another problem i've found with the automatic switch to the right side: Here's ncmpcpp displaying some Umm Kulthum tracks, the artist tag being in arabic, in xterm and mlterm for comparison :

It is difficult for a terminal emulator to determine whether to display text right-aligned or left-aligned appropriately.

Changing bidi_mode option manually as follows makes mlterm show text the
same as xterm with bicon. $ mlcc bidi_mode left

Yes it stops from going back to the beginning of the line but something gets messed up : see how the adjacent column with track numbers is drawn in the wrong place ?

[holytrousers]

The only thing that is still conflicting with vim's arabic input is mlterm's right to left implementation : in vim, :set arabic moves the cursor to the right side on an new empty line, but it looks like mlterm moves it again to the opposite side thinking it's on the left side, then moves back to the right side upon entering arabic text.

mlterm simply determines right-aligned or left-aligned by a returned value of fribidi_log2vis() in each line. It is difficult for mlterm to recognize whether the current mode of vim is arabic or noarabic.

YFI, if you want to make vim in mlterm work the same as in xterm, execute following commands which stop mlterm from processing BiDi and which stop vim from using BiDi feature of mlterm. $ unset MLTERM (or :set notermbidi in vim) $ mlcc use_ctl false vimdoc: https://vim-jp.org/vimdoc-en/arabic.html

Yes, unsetting MLTERM and mlcc use_ctl false makes mlterm behave like other terminals. ( :set notermbidi in vim doesn't seem to do anything )

If i understand it correctly, is it then that upon getting the signal from fribidi_log2vis() that there is arabic text, mlterm switches just to the opposite side and not explicitly to the right side?

[pro-anon]

I don't think setting the line_space to one or higher fixes it either.

[holytrousers]

I don't think setting the line_space to one or higher fixes it either.

It doesn't fix it completely, but it alleviates the problem a bit.

I don't think this particular problem is related specifically to arabic script, i have seen these leftovers at other occasions, but arabic glyphs make the problem more pronounced.

[holytrousers]

@pro-anon : i use U+600-6ff = DejaVu Sans Mono, what is your font ?

[mohessaid]

Thank you all for your great efforts. We started using NeoVim as our main editor. It was very disappointing to see the lack of support for Arabic or other RTL languages on the existing terminals. Luckly enough, we found mlterm even though they mentioned it in the :h arabic documentation in NeoVim. I didn't see it until I tried every existing terminal out there. We through the manual of mlterm and it was helpful to set something working for now. We won't claim that we understood everything in the manual since some of them were a little advanced for mere fullstack developers. In the other hand it was refreshing to see someone working on this. We are planning to explore your code and contribute as much as we can. Actually, testing projects in our company will include doing research and working on enhancing support for Arabic and other similar languages for your terminal and others.
I don't know if this helps but currently, we use the following configurations:
both aa and vaa fonts has these lines:

ISO10646_UCS4_1 = MesloLGS NF 16
U+0600-06FF = Droid Arabic Kufi 16
U+0750-77F = Droid Arabic Kufi 16
U+08A0-8FF = Droid Arabic Kufi 16
U+FB50-FDFF = Droid Arabic Kufi 16
U+FE70-FEFF = Droid Arabic Kufi 16
U+1EE00-1EEFF = Droid Arabic Kufi 16
DEFAULT = MesloLGS NF 16

Color file using the newmoon color scheme

black=#2d2d2d
red=#f2777a
green=#92d192
yellow=#ffd479
blue=#6ab0f3
magenta=#e1a6f2
cyan=#76d4d6
white=#b3b9c5
hl_black=#777c85
hl_red=#f2777a
hl_green=#92d192
hl_yellow=#ffeea6
hl_blue=#6ab0f3
hl_magenta=#e1a6f2
hl_cyan=#76d4d6
hl_white=#ffffff

For the main file we use the following:

fg_color = #ffffffffffff
bg_color = #2d2d2d
geometry = 80x20
use_anti_alias = true
use_variable_column_width = true
type_engine = cairo
only_use_unicode_font = false
not_use_unicode_font = false
cursor_fg_color = 
cursor_bg_color = 
receive_string_via_ucs = true
scrollbar_mode = autohide
scrollbar_view_name = next

#colors

vt_color_mode = high
blink_cursor = true

#font
use_aafont = true
box_drawing_font = decsp
use_multi_column_char = false
use_ot_layout = true
use_dynamic_comb = false

The results as you can see

Screencast.from.02-24-2024.10.37.11.AM.webm

For the Xft mode like @holytrousers prefers. It wasn't that good with most fonts.

Screencast.from.02-24-2024.10.44.54.AM.webm

Once again, thank you for the work and for the fruitful discussion.