Integrate Harfbuzz to properly shape/render symbols and diacritics in many languages

[biodranik]

This meta issue wraps all related non-Latin script rendering issues on the map, that can be fixed by integrating Harfbuzz into our Drape rendering engine.

HarfBuzz implements separate shapers for Indic, Arabic, Thai and Lao, Khmer, Myanmar, Tibetan, Hangul, Hebrew, the Universal Shaping Engine (USE), and a default shaper for scripts with no script-specific shaping model [source].

A detailed list of languages supported by Harfbuzz.

There is an interesting Maplibre discussion about the related complexities.

Should fix:

• Bad fonts for Persian and Thai languages. #460
• Fix malayalam rendering #516
• Diacritics rendering issues - Bangla #1723
• Devanagari font in 2022 May release is still old #2597
• [Bug] Better support Kurdish alphabet by using a better shaping tool (Harfbuzz) #2612
• Incorrect display of parentheses in rtl language #7875

Also may be related:

• Incorrect rendering of Japanese character #3812
• Some issues and feedback from a long time MAPS.ME user  #732

---

This project has been funded through NGI0 Entrust Fund. NGI0 Entrust Fund is established by the NLnet Foundation with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 101069594.

[strump]

Vng disabled Harfbuzz last year in commit #fecab9f4. Should we start from reverting it?

[biodranik]

That is only a small piece of the puzzle, and likely an irrelevant one. HB should be added as a submodule first.

[biodranik]

Here are the results of the research on the current Drape rendering engine text rendering.

To simplify, the current architecture does the following:

1. Initializes fonts including white and black lists.
2. Extracts character regions for each font.
3. Gives weights for each font: character ranges explicitly listed in data/unicode_blocks.txt has the highest priority, and fonts with more chars in each char range have higher priority than those with fewer chars.
4. Each line of text is processed via ICU's log2vis first, and split if necessary into 2 lines.
5. For each char its character range is calculated, a font with a higher weight is selected, and Freetype's glyph metrics and the glyph itself are cached in the texture atlas.
6. Rendering thread uses metrics (x and y offset and advance) and cached glyphs to render a line of text.

The issue here is that the current implementation simply adds offsets and advances for each glyph while ignoring complex shapes and glyph combinations. Integrating Harfbuzz should fix that.

Here is a screenshot with a test prototype:

Related methods are mentioned in the diagram below.

[biodranik]

Useful resources:

1. Caveats with font rendering in Warp terminal https://www.warp.dev/blog/adventures-text-rendering-kerning-glyph-atlases
2. Inkscape text rendering notes: https://wiki.inkscape.org/wiki/Text_Rendering_Notes
3. Details on text layout and rendering process: https://litherum.blogspot.com/2015/02/end-to-end-tour-of-text-rendering.html?m=1