Add tool: LcfTrans

[Ghabry]

This is a tool I'm working on since months (with big breaks in between).

It goes through the LDB file and all LMUs and extracts strings out of it and writes them in a po file.
Strings with same content and context are merged into one (Required by the format)

Writing back is NOT supported. Rendering this tool semi-useless currently because you can't do anything with the translations :(
A super simple po reader exists (fromPO) which can be used as a base.
Not sure if writing back makes sense at all. The ultimate goal would be adding "loadTranslation" to liblcf and support on-the-fly language switching ingame this way.

The tool lacks some polishing:

• RPG2k (English version) is not properly handled (has these %-placeholder)
• Many terms (for battle) could be skipped for RPG2k3 games
• Encoding only read from RPG_RT.ini (same problem as all tools)

[BlisterB]

It's a really good idea ^^. That sure will be usefull for the editor!

A french guy did a similar program called DreaMaker. I used it to translate a game, it was usefull but had some problems.
http://rpgmaker.net/forums/topics/2988/

[Ghabry]

Yeah I know that tool. Well "Not invented here". The tool has some Problems: No sourcecode, not Multi platform, translation file format is non-standard.
But it also Supports RPG Maker XP, so it has some advantage over my tool.

Usually I prefer reusing existing stuff but here I made an exception to gain extra flexibility.

And my motivation was not to support writing the strings back into the binary files but to support multi-language in Player, and this is the first step to it.

[BlisterB]

Oh yeah of course I was not saying "you should use it" but just "extract string is a really good idea". I have no doubt that your implementation will be more suitable to your goals ^^.

[Ghabry]

Looking forward to the first multilanguage versions of Yume Nikki and Ib :P

[BlisterB]

The Player will be abble to switch from a trad file to another :p ?

[Ghabry]

before 2017 :P. So you can start translating your favourite game to French already :P

[BlisterB]

Omg great Ghabry!
I was thinking to translate The Way ^^.

[Zegeri]

The database file is LDB, not LSD. Same mistake in lcftrans/README.md.

[Zegeri]

It produces erroneous output when using two consecutive line breaks.

msgctxt "event"
msgid ""
"foo\n"
\n"
"bar"
msgstr ""

The ultimate goal would be adding "loadTranslation" to liblcf and support on-the-fly language switching ingame this way.

+1 to that.

Besides DreaMaker, there's also RPGMaker Trans, GPL and supports from 2k to VX Ace.

[carstene1ns]

This is an oversight and needs to be changed in the whole file ;-)
i.e. sed 's/lmu2png/lcftrans/g' -i lcftrans/Makefile.am

[Ghabry]

Thx for your feedback. The issues reported by zegeri and carstene1ns got fixed

[Ghabry]

@sorlok
This is mostly finished now (there can be bugs ;))

It generates now the following files:

• RPG_RT.ldb.po - The database with all strings except event related strings
• RPG_RT.ldb.common.po - Strings found in messages of common events
• RPG_RT.ldb.battle.po - Strings found in messages of battle events
• RPG_RT.lmt.po - Map names (translating them is only required when the game uses the "Teleport" event command)
• MapXXXX.po - Strings found in message of map events

How message strings are constructed:

A new message starts at a ShowMessage event command. When it is a multiline message (event cmd ShowMessage_2) they are merged together. There is no length limit (but everything >4 needs a hacked editor)

Lonely ShowMessage_2 is reported as an error.

A Choice is merged with the previous message when it fits in the window (message lines + choices <= 4)
When it doesn't fit it is a lonely message (this is the same logic RPG_RT uses).

---

Except for the first two commits I killed now all further commits. For historical reasons they can be found here:
https://github.com/Ghabry/easyrpg-tools/tree/lcftrans-legacy

The code is now much cleaner, I can see that my C++ expertise increased alot in the last 4 years. The old code was terrible.

---

The only open research point is providing Translation comments and Line information (both useful for translators to find the string origin).

[Ghabry]

I'm emitting now the following human readable info lines (also powered by ForEach, very nice API :)).

• For DB entries that are not Term: ID X
• For common events: ID X, Line Y
• For battle events: ID X, Page Y, Line Z
• For map events: ID X, Page Y, Line Z, Pos (A,B) (Pos info is useful for translators so they can find the event quickly)

When there is a Choice the following is emitted:

• When the choice is part of the message: Choice starting at line X (Y options)
• Where the Choice is lonely: Choice (Y options)

---

I consider adding a Has Face info as this alters the message width but this will only work for very basic cases and has a high false positive rate.

[Ghabry]

Aaand here is the Update feature. It is very naive:

I just loads an existing Po file and then checks if

1. The Po entry is translated
2. The combination of "context+original" exists.

When 2 is true: Merge
When 2 is false: Delete the entry. Write it in a FILENAME.stale.po file.

[Ghabry]

@fmatthew5876
Can you tell me if some of my template foo is bad? At least my brain is exhausted now ;)

[Ghabry]

I use the ParentType to type filter because I must do a distinction between common event and battle event (they both have the same root lcf::Data::data (the database).

Is this a good way to do the filtering? My old check was a string comparison but imo this is more robust.

[mateofio]

You should consider taking this whole lambda and dispatching to another function with a null base case and an implemented EventCommand case.

The problem here is that because this lambda is compiled for every possible type, you need all these wrappers to make it compile. So for example rpg::Actorm the compiler will run through this code and compile all of it and complain if any of it doesn't compile.

By dispatching early, you get out of "generic" world and into "EventCommand" world. As soon as you do that, you can just directly get the code, indent, etc.. without all the wrappers, making your code cleaner and more readable.

For the parent filtering, the way it's written yes you need to do that if statement.

However, another way if you already have the ParentCtx object, that has the struct, so instead of calling ForEachString on the database root, call ForEachString on the CommonEvent / Troop parent directly. Then you need don't need the is_same stuff.

If you call ForEachString on a child node, it would be useful to have the full context if possible, so it matches the same context type you would get if you called with db.

[Ghabry]

because I needed lots of state I created a small class with two parse methods: One for evt commands, one "catch-all".
Helped to simplify the code alot 👍

[sorlok]

Found a small bug. Previously, running on The Blue Contestant, for skills, I get:

#. Skill 13: Description
msgctxt "skill.description"
msgid "A spell of dust that causes poison"
msgstr ""

On the latest commits, I get:

#. ID 13
msgctxt ""<0x00><0x00><0x00><0x00><0x00><0x00>.description"
msgid "A spell of dust that causes poison"
msgstr ""

The context is garbled (should be "skill.description").

[Ghabry]

@sorlok
I can't reproduce this, also the one on the PR builder is fine.

Try a clean rebuild of liblcf again, then compile lcftrans again (or fetch the binary from the PR builder - just expand the "All checks have passed" message)

[sorlok]

Just wanted to check in and see what the status is on this.

[Ghabry]

Interestingly the editor doesn't allow an empty line on line 4. Which confirms that this is for aligning num input or choices :)

@sorlok should work now. For some unrelated reason Windows PR build fails though.

Choices are splitted.

Editor input:

@> Text: A
 :         : B
 :         : 
@> Text: C
 :         : D
@> Text: E
 :         : 
 :         : 
@> Text: 
 :         : 
 :         : F
@> Text: 
 :         : 
 :         : 
@> Text: XXX
@> Text: X
 :         : Y
 :         : Z

Tool output:

#. ID 2, Page 1, Line 1, Pos (2,5)
msgid ""
"A\n"
"B\n"
""
msgstr ""

#. ID 2, Page 1, Line 4, Pos (2,5)
msgid ""
"C\n"
"D"
msgstr ""

#. ID 2, Page 1, Line 6, Pos (2,5)
msgid ""
"E\n"
"\n"
""
msgstr ""

#. ID 2, Page 1, Line 9, Pos (2,5)
msgid ""
"\n"
"\n"
"F"
msgstr ""

#. ID 2, Page 1, Line 15, Pos (2,5)
msgid "XXX"
msgstr ""

#. ID 2, Page 1, Line 16, Pos (2,5)
msgid ""
"X\n"
"Y\n"
"Z"
msgstr ""

[Ghabry]

Imo @sorlok should have the final word here. When we get an agree from you that the format is fine now this can be merged :)

[sorlok]

Thanks @Ghabry. The sample text you provided look good.

I'm running through and updating the Aurora's Tear translation and checking, just to make sure nothing else stands out. Once that's done I'll let you know!

[sorlok]

It looks good. I actually used the "-u" flag on my current Aurora's Tear translation, and it showed me exactly what changed. This can be merged, as far as I'm concerned.

[Ghabry]

Thanks for also independently testing the "-u" flag, I always only used artificial translations to test it.