Handle hotkeys on non-Latin keyboards via transliteration

[stevecotton]

Prototype: no UI is added for configuring this; at the moment it comes with
a Python script and you need to paste the output of that script into a .cfg
somewhere - for example data/core/hotkeys.cfg. The second commit in this PR
is just adding the script output. Fixes #5421.

Imagine a keyboard with Latin "a" and Greek "α" on the same key, "b" and "β" on
the same key, etc. This commit adds a fallback so that, when the user presses
ctrl+β, the engine first searches for ctrl+β and then falls back to searching
for ctrl+b.

The transliterations are only searched for when the original input doesn't
trigger a hotkey; if the user manually adds a hotkey for ctrl+β then it will be
found first when the keyboard is in Greek mode.

I'm using Greek for the examples because (looking at Wikipedia) there seems to
be consistency in which Greek letters match to which Latin letters. Cyrillic on
the other hand seems to have as many keyboard variants as Latin does (Qwerty,
Qwertz, Azerty, etc),

The Greek layout puts a semicolon on the Latin-q key, which has been omitted
from this conversion. Even if it was included, it would look for the semicolon
and colon based hotkeys before looking for the q hotkeys, that's a design
choice to avoid overriding existing hotkeys.

The syntax for the preferences file is like this:

[hotkey_transliteration]
		[substitute]
				src=α
				dst=a
		[/substitute]
		[substitute]
				src=β
				dst=b
		[/substitute]
		[substitute]
				src=ψ
				dst=c
		[/substitute]
		[substitute]
				src=δ
				dst=d
		[/substitute]
		...
[/hotkey_transliteration]

Many thanks to @renom for investigating a different method of handling keyboard
layouts. Although that method ended up blocked by SDL issues, the work saved me
from trying to read the actual keyboard layouts via SDL for this method.

[Wedge009]

I remember the difficulty of mapping keys in SDL... I vaguely recall there's methods to parse keys on the basis of its position in the keyboard, in addition to the actual value of the keys. But it got complicated very quickly and I think what worked for some people/keyboards/locales would break things for others.

If this improves things for the hot-key implementation we have now, that sounds good.

[renom]

I reported the bugs caused by incorrect keyboard layout detection to the SDL's GitHub. Let's wait for the answer.
libsdl-org/SDL#4187

[stevecotton]

Thanks for reporting it to SDL, and I'm glad that they've fixed it. However I don't think we can rely on the fix for the upcoming stable branch - Wesnoth 1.16 will need to work with the version of SDL that shipped in Ubuntu 20.04.

[Pentarctagon]

Ubuntu 20.04 will never get 1.16, since major version upgrades are never backported since they break things, so the earliest Ubuntu LTS that would have 1.16 would be 22.04.

What would be our responsibility is whether it will work on the SDL we provide ourselves via Flatpak and Steam.

[stevecotton]

Ah, that makes sense. As it's one of our CI targets I assumed that all features would have to work on that baseline, but it makes more sense to limit multilingual input support to Flatpak and Steam.

[Pentarctagon]

The main goal of the CI is to make sure that those platforms can build our code so that people using them can contribute.

However it doesn't make sense to spend limited developer time on implementing a custom fix for a for an upstream bug when the platform distributing Wesnoth won't ever get the version said custom fix would be part of in the first place. The same would apply to almost any fix or feature except perhaps security flaws, since those might actually get backported individually.

[stevecotton]

Looking at the translation statistics, it seems Russian is the only language with a Cyrillic keyboard that has over 40% translated. I've been overthinking the question of whether it would be rude to other Cyrillic-based users to ship only the Russian layout.

I've been wondering about the minimum viable UI needed for this PR to be part of 1.16. I think it would be:

• Ship at least one transliteration in mainline
• Either:
  • ship all transliterations for supported languages, or
  • users can easily install other layouts following instructions on a Wiki page or forum post
• Add a new preferences sub-dialog to enable or disable transliterations

@renom and @Zireael07, does this mock-up have the correct mapping of letters? Should it be to a Qwerty or to a Qwertz layout? Should one of the phonetic-based Russian to Latin mappings be included as well?

Although Greek was useful for prototyping, we don't seem to have an active Greek translation team to ask whether this would be the correct mapping; that also makes be doubt whether a Greek transliteration is useful.

@renom, how do you feel about this method compared with #5598?

[Zireael07]

The Russian layout I'm familiar with is the Windows layout, described at the top of this site: http://xahlee.info/kbd/russian_keyboard_layout.html and at a glance, the mapping looks ok (but I am not a native Russian speaker and still mix up some letters, so you'd best acquire an independent opinion too)

[Pentarctagon]

@stevecotton is this something you're still pursuing, and if yes does it supersede #5598?

[stevecotton]

I'm not sure if I'll take another go at this or not. It has the same problem of assuming that the other keyboard layout is Qwerty, although it puts that assumption in a config file which the user can change instead of hardcoding it (or picking up the hardcoding that's built into SDL).

Edit: ignore the next paragraph, there's several other changes between the two keyboards; and IBus disagrees with Wikipedia about where that extra key goes.
In addition to Russian Cyrillic keyboards, it seems the same transliteration set could handle Ukrainian Cyrillic keyboards - just needs one extra letter to be added (similar to Western European keyboards, Ukrainian ones have a smaller left shift key, allowing an extra physical key to the right of it).

[stevecotton]

The recent push was just to rebase onto the latest master (because it needed some updates to compile), and tidying up comments. Still not sure whether I'm continuing with this or not.

The build failures were because I forgot to retest against the schema, but had added id= keys while thinking about how to record users' preferences to enable or disable specific transliterations. There's also a failure where I'd missed adding the new dialog's test to the list of auto-run tests.

[CelticMinstrel]

I think the main point was about updating the year to 2024…?

[irydacea]

I'm very confused why I'm being tagged about authorship of a file I did not author.

[babaissarkar]

I'm very confused why I'm being tagged about authorship of a file I did not author.

So the marked line earlier is wrong? (I wonder if that's a copy-paste)

[stevecotton]

While I welcome the number of reviewers, it seems to be focused on bikeshedding a copyright statement rather than reviewing the main change. That .hpp file was a copy-paste-find-and-replace of another modal dialog, hence the attribution to irydacea; thanks for pointing it out and I'll update it when I update this branch. The same file will need an update when it gets rebased, because GUI documentation is now on the Wiki instead of in the C++ files.

Does anyone have feedback about whether Qwerty is a reasonable assumption for the other layout? Currently I think the implementation here is okay, but instead of saying that it maps between two unrelated layouts, it maps between pairs of layouts that are so commonly used together that keyboards are sold that have both sets of letters physically printed on them. If we were ever to have a language where the physical keyboards tend to be Qwertz or Azerty then that would be appropriate pairing for that specific mapping.

For anyone wondering why I'm not suggesting a Bengali mapping, despite having @babaissarkar here in the conversation: https://en.wikipedia.org/wiki/Bengali_input_methods

[babaissarkar]

For anyone wondering why I'm not suggesting a Bengali mapping, despite having @babaissarkar here in the conversation: https://en.wikipedia.org/wiki/Bengali_input_methods

Good point, the situation is real chaos there. Though if you want to focus on anything specific in the future, I'd suggest starting with Avro first, since that seems to widely used in Win systems here. The minimum would be that the English (US) shortcuts work while somebody has the Bengali layout active. (That seems to be the case with LibreOffice and Firefox, at least.)

I'm also resolving the earlier conversation about that copyright statement, which @stevecotton said he'd update.

[babaissarkar]

Usually the "Close" button uses the id "cancel", which allows Esc to close the dialog. Since it is using ok, it gets closed when Enter is pressed. Is that intended?