Complex script like devanagari support

[pranphy]

It would be nice to support complex scripts like devanagari (देवनागरी लिपी)

The left pane is vscode rendering same file as uivonim in the right pane. As you can see the text in the right is pane is barely redable.

[clason]

Thanks for the comment! Non-Latin script is something we haven't really tested. Could you maybe help and explain what exactly makes the right pane "barely readable"? As someone who'd be hard-pressed to tell an abugida from an abjad, let alone read devanagari, it's really difficult to tell what the issue is.

(Also, you seem to be using a different font in each screenshot -- can you make sure that this is not the problem here?)

[pranphy]

Thank you for responding!

In each of the snapshots below the top line is from vscode and bottom one is from uivonim. Both the editors use same font (the size might differ).

1. The most apparant issue is that the portion glyph over the top horizontal line in each character is missing or partially there. You can see how in the dot on top of second to last character is not visible in uivonim. And similarly with 3rd,4th, 7th, 8th and 9th characters.

In the above example the first letter is the consonant and the 11 letters subsequent to those are the form of same letter after being joined by a vowel sound each. After being joined by vowel modifiers, the shape and size of the consonant must remain same as in unmodified form and the vowel modifiers should take extra space if any. Also apparent from this snapshot is that the character with vowel modifiers joined take up the same width as the unmodified consonant, making the subsequent characters somewhat squeezed, its more apparent in the original snapshot I posted because it uses actual sentence with different modifiers in same word.

This image from wikipedia shows the joined vowel modifiers in each of the cases.

2. Joined character are not joined: In devanagari script when particular sequence of character appear, they are replaced by a glyph for example म + ् becomes म् but म + ् + र becomes म्र, similarly क + ् + ष = क्ष. These characters are not joined or replaced by the proper glyphs from the font. (More about these on wikipedia )

I don't have the knowledge of technical part of implementation of these devanagari characters in fonts, some of the terms I've used above might not make sense but hopefully I made two problems clear.

Edit: I had raised similar issue here #2623 on oni editor similar to this. This comment might be something to consider.

[clason]

Thank you for the clear and detailed explanation! I understand the issues now (even if I don't know how to fix them...) You are correct that there are basically two independent problems with the current (GL-based) rendering:

1. The cell size calculation is not smart enough to adapt to glyph sizes. The vertical cutoff is something I actually also observe with certain brackets in Latin (or basic ASCII) script as well; the non-adaptive width is only relevant to non-Latin scripts as well.

2. The rendering is missing advanced glyph-shaping features required for certain non-Latin scripts (and probably emoji modifiers as well). This is related to another missing feature that some may want: ligatures.

It's not quite clear how to proceed -- Uivonim uses a GL-based font rendering (inherited from Veonim) for performance reason instead of using Electron/Chromium (as VS Code does; I think we also use it for rendering the command line and other UI elements, so devanagari should look much better there?). The first point can probably be handled with our rendering (and at least the vertical cutoff needs to be addressed), but the second seems much too complicated; we would instead need to look into including a proper text shaping engine like https://github.com/harfbuzz/harfbuzz for rendering.

[smolck]

Bit of a shot in the dark, but if you wouldn't mind testing, how does it look now on master after #255 @pranphy? I rewrote the font atlas code so it might be better (although I don't think it solved this issue by any means).