Add i18n support #49

grannypron · 2021-01-26T23:12:15Z

Integrate the gnu gettext library to support translation of hardcoded strings into different languages.

Discussion at http://ua.reonis.com/index.php?topic=4052.0

grannypron · 2021-01-26T23:17:51Z

In terms of development, all strings must be marked with gettext() around them. And the #include <../i18n/libintl.h> must be included at the top of every file that uses that.

Two new dlls will need to be distributed with the .exe: iconv.dll and intl.dll. Maybe it's possible to statically link (I think that's the term) the source from gettext right into the exe which would just make the exe grow a little. Not sure - that would probably be a follow-on.

After all strings are marked with gettext, one would run xgettext on the code to generate a messages.po file that would be filled out by a translator. Then, one runs msgfmt to turn that messages.po into a binary messages.mo file and that is included in a new i18n subdirectory of the dsn directory - actually it would be a i18n/**/LC_MESSAGES/ subdirectory where ** is the local identifier like "en" or "fr".

grannypron · 2021-01-26T23:20:31Z

One problem is that some of the hardcoded strings are in global variables like the MENU_ENTRY StartMenu global variable. This initialization (and now its gettext lookup) happen very early in the initialization of the program - before the config file is loaded. So, at least for an IOC, we will use environment variables or parameters to start the .exe with localization settings.

grannypron · 2021-01-26T23:25:39Z

Another option that would not require "compilation" of po files might be https://github.com/garbageslam/spirit-po While this is non-standard and it doesn't seem to be active, this would allow people to change translation without running any external programs. Speed of execution is not too much of a concern here anyway since we are talking about turn-based games.

grannypron · 2021-01-27T00:14:14Z

The spirit-po project was very attractive, but it relied on including spirit boost which looked to me to be a HUGE library that I would have had to include in the project which was just not something that I think we wanted to do.

grannypron · 2021-01-27T00:22:58Z

This SO question provides a good overview and is left with gettext: https://stackoverflow.com/questions/2185568/alternatives-to-gettext

grannypron · 2021-01-27T03:21:00Z

I dug into trying to use https://github.com/kahrl/gettext-msvc/ to create statically linked libs of libintl and libiconv and I was able to compile them but ended up with unintelligible (to me) linker errors, which stopped me in my tracks:

Severity	Code	Description	Project	File	Line	Suppression State
Error	LNK2001	unresolved external symbol __imp__mbrlen	UAFWin	...\src\UAFWin\libintl.lib(printf.obj)	1	
Error	LNK2001	unresolved external symbol __imp__wcrtomb	UAFWin	...\src\UAFWin\libintl.lib(printf.obj)	1	
Error	LNK2001	unresolved external symbol __imp__mbrtowc	UAFWin	...\src\UAFWin\libintl.lib(printf.obj)	1	
Error	LNK2001	unresolved external symbol __imp__setlocale	UAFWin	...\src\UAFWin\libintl.lib(setlocale.obj)	1	
Error	LNK2001	unresolved external symbol __imp__open	UAFWin	...\src\UAFWin\libintl.lib(loadmsgcat.obj)	1	
Error	LNK2001	unresolved external symbol __imp__open	UAFWin	...\src\UAFWin\OLDNAMES.lib(open.obi)	1	
Error	LNK2001	unresolved external symbol __imp__read	UAFWin	...\src\UAFWin\libintl.lib(loadmsgcat.obj)	1	
Error	LNK2001	unresolved external symbol __imp__read	UAFWin	...\src\UAFWin\OLDNAMES.lib(read.obi)	1	
Error	LNK2001	unresolved external symbol __imp__close	UAFWin	...\src\UAFWin\libintl.lib(loadmsgcat.obj)	1	
Error	LNK2001	unresolved external symbol __imp__close	UAFWin	...\src\UAFWin\OLDNAMES.lib(close.obi)	1	
Error	LNK2001	unresolved external symbol __imp__bsearch	UAFWin	...\src\UAFWin\libintl.lib(localealias.obj)	1	
Error	LNK2001	unresolved external symbol __except_handler4_common	UAFWin	...\src\UAFWin\MSVCRT.lib(chandler4gs.obj)	1

grannypron · 2021-01-27T19:22:32Z

Continue to get linking errors when trying this build as well: https://osdn.net/projects/libintl-msvc10/downloads/63654/libintl-msvc10-0.2.2-gettext-0.19.5.1base.zip/

I am giving up on statically linking at this point.

grannypron · 2021-01-29T02:06:24Z

Initial attempt at a roll-your-own solution using JSON is commit df099d7

grannypron · 2021-01-29T15:35:56Z

DungeonCraft_uaf_00049.zip

Reotor · 2021-01-29T21:30:45Z

DungeonCraft_uaf_00049.zip

The developer suggests using the "messageMap.txt" file in the game "Data" folder to make translations by inserting original and translated strings into it according to a certain scheme..

This seems to work, but there are encoding problems:

When I change the encoding of the file "messageMap.txt", I get an error in the game:

Another problem is that the GENERAL GRAPHICS ERROR message displays in game apparently when the game accesses the "messageMap.txt" file:

GENERAL GRAPHICS ERROR message displays immediately after the first movement of the mouse cursor and does not disappear until you switch to another menu. If there is no translation, everything is fine, if the text is affected by this file, this message appears again after the movement of the mouse.

grannypron · 2021-01-29T22:09:45Z

The text displaying incorrectly is because the json reader only reads in UTF-8, and because the file is UTF-8, the Cyrillic characters are coded in Unicode with 2 bytes instead of one. The font doesn't have all the Unicode glyphs pre-rendered and the CDXBitmapFont::Draw* routines are only designed to write 8-bit (1 byte) characters (see #41) right now. I will have to translate backward to ANSI I think.

grannypron · 2021-01-29T22:50:08Z

Yep, translating back to ANSI with the routine here did the trick:

https://stackoverflow.com/questions/8298081/convert-utf-8-to-ansi-in-c/35272822#35272822

grannypron · 2021-01-29T22:58:28Z

DungeonCraft_uaf_00049_b.zip

Reotor · 2021-01-29T23:07:52Z

DungeonCraft_uaf_00049_b.zip

Huge THANKS! Now everything seems to be working fine! Working on interface translation ...

Reotor · 2021-01-30T18:58:41Z

While translating the interface, I found out that there are errors in the titles of multiclass characters in the character creation menu:

grannypron · 2021-02-01T21:00:40Z

DungeonCraft_uaf_00049_d.zip

Reotor · 2021-02-01T23:21:11Z

Considering that the work may take some more time, In case someone needs to test its functionality now, I will post the current version of the "messageMap.txt" here: messageMap.txt.

Except for in-game items and spells, the work is about 70-90% done (while I may be wrong and underestimate the scale of the game ...)

grannypron · 2021-02-03T01:24:10Z

DungeonCraft_uaf_00049_e.zip

Reotor · 2021-02-03T15:59:48Z

Here is the new file messageMap.txt:

messageMap.txt_002.zip

Some nuances:

Since it takes more space to translate some of the option names into Russian, I had to capitalize the text in most cases. And in order to make everything look symmetrical, I made the rest of the text of the option names the same.
It should be understood that this translation will definitely be supplemented and reworked (including with the help of the Russian-speaking community from our forums). Therefore, it cannot yet be called final.
Only the interface has been translated so far (where possible). Names of items, spells, etc. will be translated later.
As it was said before, I translated only what was possible, but some lines remained in the code and I do not know how it was originally written. Your help is needed here. For this purpose, I have inserted hints into the file.

Reotor · 2021-02-04T17:44:28Z

Here is a new file with new strings and markups added to find an opportunity to translate the unsuitable strings.

messageMap.txt_005.zip

For now, I will gradually add items, spells, etc. to the file.

Reotor · 2021-02-06T23:15:56Z

Here is the new version of the "messageMap.txt" file.

messageMap.txt_008 (all items translated).zip

Added and translated all in-game items (at least those that were in my "items.txt" file), as well as several new strings that I discovered during the game. Then I will translate spells, monsters, etc.

Reotor · 2021-02-10T20:34:25Z

New messageMap.txt here:

messageMap.txt_012 (all items and spells translated).zip

The names of all in-game spells have been translated, as well as new lines and some additional minor adjustments made. Now the translation of the names of the monsters is next...

grannypron · 2021-02-10T20:54:33Z

When you are all set and we are sure it is able to do all you need, lmk and I will cut a 5.28 release. Since you are the primary stakeholder on this atm!

Reotor · 2021-02-13T23:57:10Z

When you are all set and we are sure it is able to do all you need, lmk and I will cut a 5.28 release. Since you are the primary stakeholder on this atm!

So the new file is here!

messageMap.txt_013 (all items, spells, monsters translated).zip

The names of the game monsters have been translated into Russian. Now I need your help please in order to get to the text that is not possible to get with just this file ...

Reotor · 2021-02-14T01:35:29Z

Before the release of version 5.28, we need to deal with the translation of the text that I could not translate using the message.txt file

Reotor · 2021-02-14T01:56:31Z

Problem № 001:

It is necessary to provide the ability to translate without affecting the changing numbers. Is it possible to translate this text and how to do it?

I cannot enter this text with numbers into the "MessageMap.txt" file, as the numbers will constantly change. You can try to enter strings into the file for each new combination of numbers, but then you get just a huge number of combinations.

Reotor · 2021-02-14T02:09:52Z

Problem № 002:

Approximately the same as the previous problem....

Reotor · 2021-02-14T02:22:00Z

Problem № 003:

No matter how hard I tried to enter "attacks" word into the file, I could not get it translated...

Reotor · 2021-02-14T02:24:25Z

Problem № 004:

The same as the previous

Reotor · 2021-02-14T02:26:40Z

Problem № 005/006:

The same (for char stats menu when creating characters). Only "AVAIL" : "Есть:", worked

Reotor · 2021-02-14T02:32:35Z

Problem № 007:

(Chatacter Name) successfully wrote (Spell Name) into the spellbook." : "",

Reotor · 2021-02-14T02:33:33Z

Problem № 008:

Reotor · 2021-02-14T02:34:48Z

Problem № 009:

Reotor · 2021-02-14T02:35:57Z

Problem № 010:

Reotor · 2021-02-14T02:38:01Z

So far, that's all I've noticed. I ask for help, thanks

grannypron · 2021-02-14T02:47:33Z

Yeah I think just about all of these will require minor code changes. The strings are hardcoded in and appended together before they are passed to the DisplayText methods. I think number 6 might be able to be done by putting /GSTR or some number of spaces after STR but the rest will have to be minor code changes.

grannypron · 2021-02-14T03:07:59Z

For number 4, check out [TimeText] in specialAbilities.txt

grannypron · 2021-02-14T03:09:08Z

Number 5 is in Global_Display in specialAbilities.txt

Reotor · 2021-02-15T00:10:46Z

001, 002, 004, 008, 009, 010 ! SOLVED ! Thank you very much!

003, 005, 006, 007 still can't fix...

Reotor · 2021-02-15T18:05:59Z

005/006 SOLVED, TY!

Reotor · 2021-02-15T21:59:23Z

All the rest problems with translation for current moment SOLVED, TY! I need some more time to identify any other similar translation problems...

Reotor · 2021-02-19T18:24:03Z

messageMap.txt_LastBuild (18.02.2021).zip

grannypron · 2021-02-19T18:54:14Z

Closing. Any other i18n tweaks should be filed under separate tickets.

grannypron added the enhancement New feature or request label Jan 26, 2021

grannypron added a commit that referenced this issue Jan 29, 2021

Initial attempt at a home-rolled localization solution for #49.

df099d7

grannypron added the Waiting for review label Jan 29, 2021

grannypron added a commit that referenced this issue Jan 29, 2021

Removing unnecessary code that I accidentally left in for #49

afec0f0

grannypron added a commit that referenced this issue Jan 29, 2021

Adding points of integration for translation / message mapping. #49

edfb236

grannypron added a commit that referenced this issue Jan 29, 2021

Rearranging integration points for string substitution - #49

d7b92c5

Reotor mentioned this issue Jan 30, 2021

Incorrect titles of multiclass characters in the character creation menu #50

Open

grannypron added a commit that referenced this issue Feb 3, 2021

Including game menus in translation - #49

a47100a

grannypron added a commit that referenced this issue Feb 15, 2021

Work on message mapping hardcoded strings - #49

916b964

grannypron added a commit that referenced this issue Feb 15, 2021

Allowing translations of more hard-coded strings - #49

02044ed

grannypron mentioned this issue Feb 19, 2021

Uaf 00049 #51

Merged

grannypron removed the Waiting for review label Feb 19, 2021

grannypron added a commit that referenced this issue Feb 19, 2021

Adding VS new project new file references for #49

0ee3b3b

grannypron added a commit that referenced this issue Feb 19, 2021

Adding new files to VS project for #49

bc1fcb2

grannypron closed this as completed Feb 19, 2021

grannypron added this to the 5.28 milestone Feb 19, 2021

Add i18n support #49

Add i18n support #49

Comments

grannypron commented Jan 26, 2021

grannypron commented Jan 26, 2021

grannypron commented Jan 26, 2021

grannypron commented Jan 26, 2021

grannypron commented Jan 27, 2021

grannypron commented Jan 27, 2021

grannypron commented Jan 27, 2021

grannypron commented Jan 27, 2021

grannypron commented Jan 29, 2021

grannypron commented Jan 29, 2021

Reotor commented Jan 29, 2021

grannypron commented Jan 29, 2021

grannypron commented Jan 29, 2021

grannypron commented Jan 29, 2021

Reotor commented Jan 29, 2021

Reotor commented Jan 30, 2021 • edited

grannypron commented Feb 1, 2021 • edited

Reotor commented Feb 1, 2021 • edited

grannypron commented Feb 3, 2021

Reotor commented Feb 3, 2021 • edited

Reotor commented Feb 4, 2021

Reotor commented Feb 6, 2021 • edited

Reotor commented Feb 10, 2021 • edited

grannypron commented Feb 10, 2021

Reotor commented Feb 13, 2021

Reotor commented Feb 14, 2021

Reotor commented Feb 14, 2021 • edited

Reotor commented Feb 14, 2021 • edited

Reotor commented Feb 14, 2021 • edited

Reotor commented Feb 14, 2021

Reotor commented Feb 14, 2021 • edited

Reotor commented Feb 14, 2021

Reotor commented Feb 14, 2021 • edited

Reotor commented Feb 14, 2021 • edited

Reotor commented Feb 14, 2021 • edited

Reotor commented Feb 14, 2021

grannypron commented Feb 14, 2021

grannypron commented Feb 14, 2021

grannypron commented Feb 14, 2021

Reotor commented Feb 15, 2021 • edited

Reotor commented Feb 15, 2021 • edited

Reotor commented Feb 15, 2021 • edited

Reotor commented Feb 19, 2021

grannypron commented Feb 19, 2021

Reotor commented Jan 30, 2021 •

edited

grannypron commented Feb 1, 2021 •

edited

Reotor commented Feb 1, 2021 •

edited

Reotor commented Feb 3, 2021 •

edited

Reotor commented Feb 6, 2021 •

edited

Reotor commented Feb 10, 2021 •

edited

Reotor commented Feb 14, 2021 •

edited

Reotor commented Feb 14, 2021 •

edited

Reotor commented Feb 14, 2021 •

edited

Reotor commented Feb 14, 2021 •

edited

Reotor commented Feb 14, 2021 •

edited

Reotor commented Feb 14, 2021 •

edited

Reotor commented Feb 14, 2021 •

edited

Reotor commented Feb 15, 2021 •

edited

Reotor commented Feb 15, 2021 •

edited

Reotor commented Feb 15, 2021 •

edited