Localization

[alexmojaki]

Hi @aroberge,

@martinvilu has offered to translate futurecoder to Spanish:

• https://www.reddit.com/r/learnprogramming/comments/qjs3wh/i_built_futurecoder_a_100_free_and_interactive/hjv4j3u/ -
• Translations to other languages alexmojaki/futurecoder#223 (posted as I write this, perfect timing!)

Apparently they've even started translating friendly traceback!

There's also been a few requests for a French translation as well and one person has emailed saying they're considering creating a French fork of the codebase.

I've never had to implement localization before so I wanted to ask both of you for advice and hear about your experiences.

How does the system in friendly-traceback work? Are you happy with it? Does it automatically generate and update things nicely, so you only have to write the actual translations? Does it work well when you change an English string?

Each exercise in futurecoder has a solution which in the codebase is a regular Python method (it needs to be executable and I want syntax highlighting and other editor assistance) from which the source code is extracted so that the user can see the solution. Here's an example (the function header is not shown to the user):

        def solution(self, strings: List[List[str]], word: str):
            for sublist in strings:
                for string in sublist:
                    if word in string:
                        print(string)

Ideally this kind of code would also be translated, specifically the variable names. Do you think it would make sense to treat each variable name as an individual string for translation?

@martinvilu can you explain a bit more how this would work in futurecoder? You would do the work in https://poeditor.com/ and it would magically sync with GitHub? What happens if I edit an English string that already has a translation? How easily will this work with the combination of Python, JS, and HTML?

@aroberge if the infrastructure was in place and futurecoder had po/pot/mo files, would you be interested in helping with a French translation?

[aroberge]

Just acknowledging that I read this wnd would be interested in helping with a French translation for futurecoder.

I use Poedit (local install; I paid for a Pro version) and was not aware of the online editor. I am a bit hesitant to use a non-free service for my projects.

I will try to write a more comprehensive reply later (possibly only tomorrow).

[martinvilu]

Hello from the distant 🇦🇷

I've mentioned POEditor online, as I have already contributed translations in the Thonny proyect with very little resistance. And from what I've seen, the tool syncs with the PO files in the repository. (I'll try to connect my fork and report later my findings)

For now, I'm working offline with a classic fork/pull request as it might be a lot easier for everyone.

[alexmojaki]

Oh wow, that's a lot of languages in Thonny! @aivarannamaa I'd be keen to hear any wisdom/experiences you can share.

[aivarannamaa]

POEditor doesn't really sync changes to Thonny's repository -- I'll export changed languages before each release myself.

It is very little work for me, but actually I'm not very comfortable with this. Every time I update mo/po files in Thonny's repo, I just hope that the updates doesn't contain any offences, which are hidden to me.

Maybe I should require a specific person to become responsible for a language, who reviews all changes in POEditor and submits to repo via a PR.

[alexmojaki]

What do you do when you create new strings, e.g. when you add a new feature? Do you wait until they get translated into every language before releasing? Do you have separate code paths that check whether a translation exists?

[aroberge]

Hi @alexmojaki,

I have had some success in adding translations and having people contribute to programs such as RUR-PLE [1] and its successor, Reeborg's World [2], although not as much as Thonny. For these two projects, I used a simple dict-based approach where I would have something like en["some string"] = "some string"; fr["some string'] = "une chaîne quelconque". This worked reasonably well for me, especially for a mixed environment (translating some html, javascript and Python content), and seemed to work well enough for people that contributed translations.

How does the system in friendly-traceback work? Are you happy with it? Does it automatically generate and update things nicely, so you only have to write the actual translations? Does it work well when you change an English string?

For friendly-traceback, and a few other projects, I used the standard gettext approach to create .pot files and started using the free version of Poedit, a desktop program designed for this task. After a while, I decided to pay for the pro version, to support its creator and found that the available suggestions were much better, making the translation task so much easier; I wish I had used the pro version from the start. If you do not do translations yourself, you do not need an editor like Poedit.

In my code, I use the English version of the strings, surrounded by the standard _(...) function call; this enables gettext to identify them. For long strings where values need to be inserted, I use automatic string concatenation (instead of triple quoted strings; see [4]) together with the .format() method. For example,

            cause = _(
                "Perhaps you meant `from {name} import {attr}`.\n"
                "`{attr}` is a name similar to `{rest}` and is an object that\n"
                "can be imported from `{name}`.\n"
            ).format(name=name, attr=attr, rest=rest)

When I update or add new strings to be translated, I run a script [3] that scans my files, looking for all strings to be translated. I then open the existing ".po" French file, update it from the generated ".pot" file, and add the missing translations.

gettext supports many features, such as handling of plural based on the number of objects. If I recall correctly, English uses plural for zero objects while French uses singular in this case. And, according to the web:

Chinese and Japanese have a single form for plurals. German, English, Spanish have two forms. On the other hand, some Slavic languages have 3 or more forms for plural nouns. And then there are languages like Arabic and Welsh that have 6 forms.

I decided not to support differend plural forms, which is also what Google's Blockly does. [5] (I encourage you to consult that link, which may help you in deciding what standards to adopt in your project).

I have my own method to install gettext, allowing to switch languages in a single session, as you can find in https://github.com/friendly-traceback/friendly-traceback/blob/main/friendly_traceback/ft_gettext.py

Ideally this kind of code would also be translated, specifically the variable names. Do you think it would make sense to treat each variable name as an individual string for translation?

It might be helpful to do so, but there is a "danger" in translating individual words. A single word in English may be translated differently based in the context. For example, the English word "to" can be translated in French as "à" or "pour" depending in the context. The French word "à" in turn can be translated in English as "to", "at", or "in" depending in the context. (I found such a mistake in the original translations for Blockly, where the word "to" appeared as a single word, in two different contexts, but both having been translated the same way - which was incorrect. I was able to fix this in my application as Blockly does not use original English strings, but unique names; see [6]), something you could choose to do, but would require you to maintain an English ".po" file, instead of just using the strings as they appear in your code.

The context dependency of translations is also why I translate chunks of text as large as possible: in some languages, it might make sense to invert the order in which full sentences appear, compared with the English version.

= = =
Currently, friendly-traceback has 629 translated strings [7] . This makes it too costly to use the site https://poeditor.com/ as the number of original strings + translations makes a total greater than 1000 [8] which is their limit for the free version.

So, like @martinvilu, I stick to the approach of local translations with the fork/pull request approach ... although it is not a pull request for me, obviously.

@aivarannamaa mentions having a specific person responsible for a given language, to prevent inappropriate translations. I found something similar (but not as bad) in another project (I forget which one) where someone with apparently poor writing skills had provided translation in a language I did not understand. An other person, with obvious qualifications in that language from their profile, provided corrections.

[1] Rurple has support for English, French, German, Simplified Chinese, Turkish, and Welsh.

[2] Reeborg's World has support for French, English, Chinese, and partial support for Korean and Polish.

[3] py -3.7 c:\users\andre\appdata\local\programs\python\python37-32\tools\i18n\pygettext.py -v -p locales -d friendly *.py */*.py

[4] If I were to use triple-quoted strings, in order to get the right indentation of the string content, the strings on consecutive lines would not respect the code indentation.

[5] https://translatewiki.net/wiki/Translating:Blockly

[6] Blockly.Msg["PROCEDURES_DEFNORETURN_TITLE"] = "to";

[7] With an additional 28 strings for friendly, kept as a separate project.

[8] https://poeditor.com/help/faq#What-are-strings

[alexmojaki]

Thanks for the detailed reply!

For translating code, since I'm thinking of translating variable names, I'm assuming there won't be words like "to", but rather more 'concrete' nouns and verbs. If I require translating entire blocks of code, it's more work for the translator and I have to worry about ensuring that the translated code is equivalent, which probably means ensuring that the only thing they've done is translate the variable names.

https://poeditor.com/pricing/ claims that it's free for open source projects, so hopefully I can use that.

[aroberge]

Thanks, I had missed that ... and just created a project which was apparently approved!

[aroberge]

https://poeditor.com/join/project?hash=yC6iMTgLn5

[alexmojaki]

Nice! Hope you can find someone to help you with the English translation lol

[martinvilu]

I haven't seen the 'import' option, so suddenly, the project is 52% complete. :-D

[aroberge]

@martinvilu Yes, I just saw that. I need to set up a webhook for that file. Could you do a pull request so that I can link it via a webook?

[martinvilu]

Yes, I'll do it as soon as possible.

[aroberge]

@martinvilu No need to do a pull request: I've already added the existing Spanish .po file and tested it.

[aroberge]

Blog post: https://aroberge.blogspot.com/2021/11/friendly-traceback-en-espanol.html with links to the two pages that have been added to the documentation for the Spanish translation.

[alexmojaki]

Last link in the post is broken, remove the . from the end of the URL.

[aroberge]

Fixed; thanks!