Feature: [Linux] RTL support with modern versions of ICU

[TrueBrain]

Motivation / Problem

With ICU 58+, we no longer have RTL support on Linux. This is becoming a bit annoying, as we can't keep using ICU 50 to build our releases.

Debian, Ubuntu, Flatpak, and many others already have this problem, and their versions do not support RTL. Our official Linux release does, but purely as we use a very old ICU to build against.

So it is about time. Switch to ICU + harfbuzz, meaning we no longer need ICU-lx (which supplied the RTL support).

Fixes #6922. A 5 year old ticket.
Fixes #6666.

Description

This means we have RTL support again with ICU 58+. It makes use of:

• ICU for bidi-itemization
• ICU for script-itemization
• OpenTTD for style-itemization
• harfbuzz for shaping

Work-in-progress

So far the Windows (Uniscribe) and Linux (ICU + harfbuzz) look identical for single-line entries.

• Support multi-line
• IME input (the char-to-glyph mapping returns nullptr atm)
• Crashlog showing harfbuzz version
• Remove ICU from comments where it is no longer ICU-only
• Split fallback layouter to its own file
• Support custom OpenTTD icons (like train-icon, etc)

Limitations

Some translations aren't perfect, which means some things look really odd. So often it seems like the shaping goes wrong, but in reality it is the translation who is to blame. So for now, we are comparing Windows (uniscribe) with this PR, and accept it if both render things the same.

Checklist for review

Some things are not automated, and forgotten often. This list is a reminder for the reviewers.

• The bug fix is important enough to be backported? (label: 'backport requested')
• This PR touches english.txt or translations? Check the guidelines
• This PR affects the save game format? (label 'savegame upgrade')
• This PR affects the GS/AI API? (label 'needs review: Script API')
  • ai_changelog.hpp, gs_changelog.hpp need updating.
  • The compatibility wrappers (compat_*.nut) need updating.
• This PR affects the NewGRF API? (label 'needs review: NewGRF')
  • newgrf_debug_data.h may need updating.
  • PR must be added to API tracker

[michicc]

In the end, nobody here can really verify the layout results, so if it looks alright enough, it is definitely better than nothing at all.

[matthijskooijman]

Seems this PR got omitted from the changelog, but it is quite a relevant change IMHO. Maybe good to add it retroactively?

[TrueBrain]

In the changelog we tend to only mention things that have an impact on users. In this case, for them nothing changed: RTL worked with 13.0 and with 14.0, with little observable difference.

The only change really is for someone like yourself. And we informed you by closing your ticket about this :D we don't really have a good way to "talk" to distro maintainer in that sense.

So personally I don't see a reason to add this to the changelog itself. And as this was the last outdated package that we depended on, I also hope this is the last time it would have been interesting to distro maintainer :D

[matthijskooijman]

In the changelog we tend to only mention things that have an impact on users. In this case, for them nothing changed: RTL worked with 13.0 and with 14.0, with little observable difference.

Ah, that makes sense. Not super-convenient for me and maybe worth reconsidering in the future (mentioning dependency changes or licensing changes/additions of 3rd party code is probably not problematic for regular users, but would help packagers), but no need to retroactively fix anything then.

And we informed you by closing your ticket about this

Yeah, so I put a cryptic note on my own TODO list, but then was a bit confused when it was not mentioned in the changelog. Now I know why :-)

And as this was the last outdated package that we depended on, I also hope this is the last time it would have been interesting to distro maintainer :D

I don't want to burst your bubble, but I'm pretty sure this is not going to be the last dependency change ;-p