Add support for generating comments from tr() and tr_n() functions for gettext (pot) translation system

[Vovkiv]

Describe the project you are working on

Godot (starting from 4 version) introduced support for gettext translation system, which is better suited in most cases then using csv or json, since it was specifically designed for translating software/sites/games/etc.
It has comments system (including, ability to adding comments into translated po files for future translators to explain their translating choices, wording, etc), context hints, po and pot considered widely used and thus has many software and services that supports it.
And comments is great way to making long explanations/documentation for strings. For example, explain to translator that string shouldn't be longer then n symbols or that some specific word shouldn't be translated or in which case this string even will be visible in-game/program (which is useful for debugging, so translator know what they need to do to trigger appearing of string in right context).

As of now, I working on game that uses gettext system, instead of csv files, and I want to make sure that there will be detailed documentation for each string to eliminate problems for translator, which in return might guaranty better translated text for my game.

(Kind of off topic: from my own translating experience (specifically, with software that uses gettext and if be MORE specifically, GTK4 software) I find it harder to translate something when there zero context and documentation. I need to guess in which context some strings appears or if there some limits that developer didn't mention (like, characters limits) Likely, gettext provides tools for this and some developers makes use of it. This makes translation debugging/testing much slower and harder, which might result in more misstranslations, wrong wording, wrong context, and (that, in my opinion) much worse - giving up on translation, if it hard or confusing enough.)

Describe the problem or limitation you are having in your project

gettext file format has some options for providing comments. The simple one is:

# Comment.

Or from gettext manual, #. considered to be extracted comment that extracted directly from source code and they should be targeted by developer to translators.

But as of now, Godot can't provide comments through tr() tr_n() functions.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

If Godot will be able to take advantage of already existing feature (comments generating) in already supported system (gettext), it will allow game/software developers to improve quality of their source text )or, in this case, pot file) without using additional tools outside of Godot.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I propose adding 3rd optional (same as 2nd argument: context) argument for tr() (And 4rd argument for tr_n(), since it requires more arguments) functions that will be used by pot generation system as comment.

var translated_string = tr("Untranslated string", "Context", "Long (or short) comment that will provide some explanations of limitations of broader context for this string.")

After generating pot from that script, we will get something like that:

# Long (or short) comment that will provide some explanations of limitations of broader context for this string.
#: some_script.gd
msgctxt "Context"
msgid "Untranslated string"
msgstr ""

And translators will see something like that in their po editing software of choice:

Since this is optional arguments, this addition shouldn't break any existing projects compatability.

If this enhancement will not be used often, can it be worked around with a few lines of script?

It should be possible to configure gettext or simple edit pot file by hand to include comments or use translation context for that, but there several limitations with that:

1. Some po editing software doesn't even handle long context messages properly (in this example, POEdit):
   

Here we can't even see original string because context is taking all space. To see original, we need to highlight desired string:

If source context string is long enough, it will make translator's navigating experience worse then it should be.

2. Context string wasn't designed for that purposes which will result in another problem:
   Changing context in pot or source code will make strings "fuzzy" for translated po files since context string used to determine where translated string should be used. So, lets say, I would create translation:

var string = tr("Untranslated string", "This stirng will be used in foo situation.")

But later (when translations for some languages were already done) I noticed that I made typo in word "stirng". I fixed it, generated new pot file and uploaded. Because I did that, "Untranslated string" become not translated in game for all languages, because gettext form source code (and as result, also pot file) expects for context to be "This string will be used in foo situation." but in po files there still "This stirng will be used in foo situation.". That means, that I should ask all translators to fix that typo in their translations or I manually (or with help of some scripts, tools) edit every translation to fix that typo, which might become especially cumbersome if there more then 2-3 translation available for my game.

3. Manually added comments in pot will be removed on next pot generation, which means you will need to maintain comments separately, which is not something that you, as developer, should do.

Is there a reason why this should be core and not an add-on in the asset library?

Because Godot already can generate pot file (Project -> Project Settings... -> Localization -> POT Generation) and provide context for strings (2nd argument for tr() and 3rd argument for tr_n() functions), there no reason to not include comments generation as core feature.

[dalexeev]

As an alternative, we could add extraction of GDScript comments with a specific prefix (e.g. TRANSLATORS:) to the POT generator, rather than adding a new argument to tr() and tr_n().

[Vovkiv]

@dalexeev

As an alternative, we could add extraction of GDScript comments with a specific prefix (e.g. TRANSLATORS:) to the POT generator, rather than adding a new argument to tr() and tr_n().

While it's how gettext utilities usually collects comments (since you can specify what considered to be comment), I do think that this syntax feels somewhat disconnected from actual string that need to be translated and might conflict with some comment styles.

Does comment should be strictly 1 line before tr()? Or there can be several? Can it be inserted in same line after tr()? (e.g: tr("String", "Context") # Some comment.). After line? If yes, how determine which comment correspond with which tr() function if there several of them?
e.g:

# Comment a.
var a = tr("String a", "Context a")
# Comment b.
var a = tr("String b", "Context b")

Also, what about typos? I think using "TRANSLATORS: " is very prone to making typos (especially for non-native English speaker who more likely to misspell words) especially if there no word completion system for comments. And then wonder why there no comments in pot file.

I also suspect, that writing "TRANSLATORS:" near each line that need to be translated and commented will be quite cumbersome in situations where there tons of strings.

Also this addition might require adding syntax highlighting for "TRANSLATORS" which might require additional work for theme creators.

And what about documentation and discoverability? Using additional argument will mean, that it will be much easier for user to discover this feature simple opening documentation for tr() function (with ctrl + click on function in-editor or via online docs) + argument completion system will remind that this feature exists:

If it will be implemented as keyword, this information then should be implemented somewhere else and thus completion system will tell nothing. That will mean, that less people (who doesn't already familiar with gettext syntax) will know about this feature and will result in more undocumented translations.

While using additional argument will probably mean, that gettext utilities won't work with gdscript files (meaning you can run gettext on your Godot project), I don't think this is problem since I doubt that gettext will be able to parse and work with scene/resource files where you could specify "auto-translate" property to true., so there little to no reason to generate pot files with something other then Godot itself.

And, IMHO, implementing and maintaining additional argument will be much easier as opposed to keyword.

[dalexeev]

I'm not insisting, I'm just mentioning it as an alternative because:

1. This duplicates the purpose of the context argument.
2. This breaks GDExtension and C# binary compatibility (requires a little boilerplate).
3. In Godot source code, this is also implemented using TRANSLATORS: comments.
4. The argument will be passed at runtime, while comments are ignored at runtime.

Does comment should be strictly 1 line before tr()? Or there can be several? Can it be inserted in same line after tr()? (e.g: tr("String", "Context") # Some comment.). After line?

This has already been implemented for doc comments. Two types are supported: block comments before a line and inline comments.

If yes, how determine which comment correspond with which tr() function if there several of them?

Also, what about typos? I think using "TRANSLATORS: " is very prone to making typos (especially for non-native English speaker who more likely to misspell words) especially if there no word completion system for comments. And then wonder why there no comments in pot file.

I also suspect, that writing "TRANSLATORS:" near each line that need to be translated and commented will be quite cumbersome in situations where there tons of strings.

I think a comment is a rare occurrence when you want to describe the context of the string in detail (possibly even across multiple lines, and string literals are not very convenient for this). Usually the context argument should be sufficient.

Also this addition might require adding syntax highlighting for "TRANSLATORS" which might require additional work for theme creators.

This is already implemented in the Godot editor, we could just add a new default comment marker.

there no word completion system for comments.

And what about documentation and discoverability? Using additional argument will mean, that it will be much easier for user to discover this feature simple opening documentation for tr() function

I think autocompletion in comments would help here; we already have a list of keywords in comments and doc comments (comment markers, @tutorial, @deprecated, @experimental). There is a basis for such autocompletion, although it is not currently used. There are also 2 draft PRs (one, two) for tooltip documentation.

[Vovkiv]

Welp, in that case TRANSLATORS: makes more sense then.
(I also forget about arguments not being useful during runtime, huh.)
Then it would be important to add mention about usage of TRANSLATORS: in documentation (like tr()/tr_n() functions and related areas) and this should works just fine.

[AdriaandeJongh]

C# support for this feature would also be appreciated. I don't know if that should be a separate issue?

[Calinou]

C# support for this feature would also be appreciated. I don't know if that should be a separate issue?

POT generation isn't implemented at all for C# scripts yet. This should be tracked in a separate proposal 🙂

See the implementation of its GDScript counterpart for reference: https://github.com/godotengine/godot/blob/aa5b6ed13e4644633baf2a8a1384c82e91c533a1/modules/gdscript/editor/gdscript_translation_parser_plugin.cpp