Setup a web-based translation service for Mudlet

[vadi2]

We need make it super easy for people to contribute to translating Mudlet - that means something web-based where the only barrier is just logging in. There might be an existing web-based service that could offer us an open-source license to get setup.

Available alternatives:
Launchpad - it's future is questionable however
Transifex - demo project here: https://www.transifex.com/mudlet/mudlet/dashboard/

[keneanung]

There is POEditor as integration with github, that allows translating XLIFF files (which are supposedly supported by qt linguist). And they have an OS license. Another option is Qordoba, which doesn't seem to cost anything and natively support ts files. Never used any translation service/app, so I have no preferences. Vadim Peretokin <notifications@github.com> schrieb am So., 9. Apr. 2017, 13:14:

…

We need make it super easy for people to contribute to translating Mudlet - that means something web-based where the only barrier is just logging in. There might be an existing web-based service that could offer us an open-source license to get setup. Available alternatives: Launchpad <https://translations.launchpad.net/> - it's future is questionable however — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#856>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABAeDUJoAsg-zvnOq1YUvuIqn-fbRwcCks5ruL2WgaJpZM4M3_u1> .

[SlySven]

Re: XLIFF the Qt Linguist Manual: Translators Qt Doc notes:

Translating Strings
You open translation source (TS) files in Qt Linguist for translation. TS files are human-readable XML files containing source phrases and their translations. TS files are usually created and updated by lupdate. If you do not have a TS file, see Release Manager to learn how to generate one.
You can use Qt Linguist also to translate files in the international XML Localization Interchange File Format (XLIFF) that are generated by other programs. However, for standard Qt projects, only the TS file format is used. The minimum supported version for XLIFF format files is 1.1.

[vadi2]

There's web-based translation platforms out there now that have massive advantages over desktop applications: no need to install any software, suggests translations for already translated text, and etc.

[keneanung]

@SlySven I've read this, which is why I said what I said. It reads like not fully supported or an afterthought, which is why I added supposedly (I never tested it).

@vadi2 The products I mentioned are web-based platforms. But you will still need to use qt linguist to make the output of those platforms usable by the Qt translation framework. Which also binds us to the supported formats TS files and XLIFF.

[SlySven]

I've played with Linguist and the UI is quite good as far as I can tell (I'm a fan 😍), it does replicate UI dialogues {and show the source code where the text string originates for C++ originating QStrings} so you can see a good idea at what they will look like in the different languages and you can work on multiple translations simultaneously. I just hope that Qt is more rigid in not reordering the contents of the TS files unnecessarily to keep the git noise down when a file gets updated by a contributor - unlike, say, QGridLayouts in .ui files. *sigh*

As for needing to use Qt Linguist - I think as we are using Qt Libraries generally we are likely to be committed to it (as in Pig rather than Chicken levels of commitment in a bacon-and-egg sarnie) besides going native gettext would be harder in not a good way I believe.

[vadi2]

Sounds like I'm interested in seeing Qt Linguist export to something web-based then, with the option for people to install and preview their translations realtime if they'd like. I'll look at the services you mentioned @keneanung, they look nice.

I also had Transifex on my mind but that doesn't seem to be doing so well anymore.

[vadi2]

Another web translation platform: https://hosted.weblate.org/projects/tilix/translations/

[Kebap]

Here is how Mozilla project handles localization and translation: https://developer.mozilla.org/en-US/docs/Mozilla/Localization/Quick_start_guide/Translation_phase

[Kebap]

Another great in-depth look at several alternatives and the process in general in this master study thesis about “Translations in libre software”: https://larjona.wordpress.com/translations-in-libre-software/

[vadi2]

Thanks @Kebap, I've read through the paper you've linked. I'm interested in Weblate and Transifex - will be checking them out.

I'd like this to have the lowest barrier of entry possible and that rules out desktop-based translation where the person has to download and install software, then hunt around for the right files before they have a chance to start work.

[vadi2]

Weblate seems nice and open-source, but their hosted version is also down right now - and so are all hosted translations like for the Tilix project. Not good. I'm also wary of hosting any more things on our server because it's running at the limit already.

Going to be trying out the Transifex demo. They support open-source and they seem to be a much bigger shop, so far less likely to go down, unlike Weblate.

[vadi2]

With that said, if Weblates Git integration does actually work, it'll be a lot better to use since it should auto-update new source text from Git, and maybe commit it back. Transifex on the other hand is requiring me to manually upload the files to translate.

[vadi2]

While Transifex supports .ts files natively, it seems to choke on upload. Using ts2po and uploading .po files worked though.

[vadi2]

It looks like you can have automatic two-way updates between Transifex and Github with some middleware: https://docs.transifex.com/integrations/github-txgh

[vadi2]

I was able to download a .po with the translation intact from Transifex, but po2ts chokes on the file with a Python encoding error:

po2ts: WARNING: Error processing: input ./for_use_mudlet_rupo_1_ru.po, output ./for_use_mudlet_rupo_1_ru.ts, template None: 'ascii' codec can't encode characters in position 1994-1999: ordinal not in range(128)

I'll leave the Transifex project up at https://www.transifex.com/mudlet/mudlet/dashboard if anyone else would like to it out for translating @Kebap @keneanung. Will give Weblate a try next.

[vadi2]

Weblate is nice - it has a ton of third-party login options, so joining it is really friction-free.

[SlySven]

@ vadi2 wrote:

I was able to download a .po with the translation intact from Transifex, but po2ts chokes on the file with a Python encoding error:

I think I read somewhere that their version of po2ts only works to an old specification of .ts file and needed updating (at their end I guess) - I note that it comes from the Translation Toolkit from Translate House Github source code and the latest version is 2.2.5 (I dunno what they are using) at the time of writing. Found a web-page from another project (not sure that it was GitHub based but the overall process for them seemed like the sort of things we will have to do: mixxx internationalization page.

Actually that "encoding" error sound like it is trying to produce a .ts file where the encoding is ASCII and NOT UTF-8 so is barfing at the point where it needs to use some characters (in a translated string) that are not ASCII...

[vadi2]

That page is last updated four years ago! The process for developers is out of date, I think. That's some pretty impressive translation documentation though!

…

On Tue, 19 Sep 2017, 6:18 pm Stephen Lyons ***@***.***> wrote: @ vadi2 wrote: I was able to download a .po with the translation intact from Transifex, but po2ts chokes on the file with a Python encoding error: I think I read somewhere that their version of po2ts only works to an old specification of .ts file and needed updating (at their end I guess) - I note that it comes from the Translation Toolkit <http://toolkit.translatehouse.org/> from Translate House Github source code <https://github.com/translate/translate> and the latest version is 2.2.5 (I dunno what they are using) at the time of writing. Found a web-page from another project (not sure that it was GitHub based but the overall process for them seemed like the sort of things we will have to do: mixxx internationalization page <https://mixxx.org/wiki/doku.php/internationalization>. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#856 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGxjEdeBPs-7fDkUwfoLjUt48eJJhl0ks5sj-lagaJpZM4M3_u1> .

[vadi2]

Weblate does not seem to be in a good position. I've yet to receive a response for my hosted application, and the project doesn't get much by the way of updates. Back to Transifex it is.

Tips on how to get Transifex setup with Qt: https://forum.qt.io/topic/36750/qt5-ts-files-and-transifex-continuous-translation-localization

[vadi2]

https://bugreports.qt.io/browse/QTBUG-1136:

finally, linguist is essentially on life support - we ceded the ground to ISVs which specialize in this area.

Qt Linguist is a dead-end technology - it will not be getting any features going forward, so we have to use a web-based platform here.

I'll look at getting Transifex translations up and running now that #1334 is available.

[SlySven]

Just to reiterate what I mentioned in Kebap's cross-reference that crowdin also looked like a valid choice for an Open Source Project like ours - and they can handle Qt .ts files and also has Github integration - the pseudolocalizaton feature should also help us to prevent problems in the GUI because we have not structured the text properly (I think) and will definitely help us to ensure we leave enough leeway for languages with strings longer that en-US ...

[vadi2]

crowdin looks nice - let's see how it compares to Transifex, I've got a test project up here.

[vadi2]

@SlySven that code mangling to make html nicer for translators was premature - transifex for example handles html markup really nice already:

[vadi2]

I figured it out - crowdin has that option as well. Just select "hide" in editor tags settings:

This means that we do not have to do the code mangling @SlySven has been doing anymore - we can use HTML as-is in the source strings, and both neither coders and translators are inconvenienced.

I'm happy with @SlySven's suggestion of using @crowdin for translations. @Kebap, @keneanung, @SlySven your thoughts?

Once we agree I'll brush up the project on crowdin to be official, apply for an open-source license and we start getting the word out for translators.

[Andrulko]

Hey,

I'm a part of Crowdin team and will gladly assist with further setup if necessary! Feel free to mention me if any questions arise

[keneanung]

I'm fine with crowdin as well. Its UI has been confusing at first, but I'm sure we (I) will get used to it.

[SlySven]

I will confess I haven't looked closely at the Transifex one as I switched off mentally considering it as I understood that it didn't handle the current Qt .ts format well/directly - has that changed or was I incorrect in that belief?

One thing about the Crowdin one that I am seeing in the recent git commits is that you were uploading multiple .ts files which had been configured locally to be for a specific language. You've sort-of fixed it now by only sending a single file a mudlet_en.ts file but that still seems not quite right to me. You may want to remove or comment out the TRANSLATIONS = lines in the mudlet.pro file and have a shell script that runs (in the base of the source):

lupdate -recursive ./src/ -ts ./translations/mudlet.ts

to generate a single, non-langauge specific file to pass into the Crowdin translation process and get that to produce the mudlet_xx_YY.ts files we want (I've already changed one setting on Crowdin to specify the _xx_YY language+locale suffixes where it originally seemed to be set to only a _xx language one). That will still result in a set of files to then be copied back into the same directory ready for lrelease.

BTW Leaving the plain un-suffixed file around is helpful because it means that minority language interested parties can copy and rename it to work privately if they so wish for a different, uncovered, language with the provided Qt Linguist. {Maybe a piratical one for 19th September for instance!!!}

[SlySven]

I think the HTML anti-obfustication steps I have been taken do seem to have become a bit less necessary as the Qt Designer plugin does seem to have become a little less eager to mangle the HTML in the last couple of Qt versions. I'm not yet totally convinced that it won't mess things around and change things by, say, forcing the use of the particular fonts that the last person to edit it has in their systems rather than leaving it to be a generic one that can work on all platforms. I'll have another look and report that here because it will affect how we record such HTML in our work.

[vadi2]

I will confess I haven't looked closely at the Transifex one as I switched off mentally considering it as I understood that it didn't handle the current Qt .ts format well/directly - has that changed or was I incorrect in that belief?

I was as well, it seems it has changed: https://docs.transifex.com/formats/qt-ts

I don't plan on using the Qt Designer plugin so whatever mangling it does is not relevant - crowdin handles the translations quite well enough on its own, and it's HTML rendering with the "hide" style is tolerable.

[SlySven]

Oh, it is still doing it - inserting tables and a DTD - so, which is easier to read and parse by humans, this:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd">
<html><head><meta name="qrichtext" content="1" /><style type="text/css">
p, li { white-space: pre-wrap; }
</style></head><body style=" font-family:'FreeSans'; font-size:9pt; font-weight:400; font-style:normal;">
<p align="center" style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><span style=" font-size:large; font-weight:600;">Welcome to Mudlet!</span></p>
<p align="center" style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">Click on one of the MUDs on the list to play.</p>
<p align="center" style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">To play a game not in the list, click on <img src=":/icons/list-add_small.png" /><span style=" color:#555753;">New</span>, fill in the <span style=" font-style:italic;">Profile Name</span>, <span style=" font-style:italic;">Server address</span>, and <span style=" font-style:italic;">Port</span> fields in the <span style=" font-style:italic;">Required</span> area.</p>
<p align="center" style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">After that, click <img src=":/icons/dialog-ok-apply_small.png" /><span style=" color:#555753;">Connect</span> to play.</p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">Have fun!</p>
<p align="right" style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">The Mudlet Team <img src=":/icons/mudlet_main_16px.png" /></p></body></html>

or this, which does the same as far as we are concerned:

<html><head/><body><center><big><b>Welcome to Mudlet!</b></big></center></p>
<p><center>Click on one of the MUDs on the list to play.<center></p>
<p><center>To play a game not in the list, click on <img src=":/icons/list-add_small.png"/> <span style="color:#555753;">New</span>, fill in the <i>Profile Name</i>, <i>Server address</i>, and <i>Port</i> fields in the <i>Required</i> area.<center></p>
<p><center>After that, click <img src=":/icons/dialog-ok-apply_small.png"/> <span style=" color:#555753;">Connect</span> to play.</center></p>
<p>Have fun!</p>
<p align="right">The Mudlet Team <img src=":/icons/mudlet_main_16px.png"/></p></body></html>

Technically this is all Qt rich-text which is only a sub-set of HTML (so why force the inclusion of the strict HTML 4.0 DTD?)

Even showing < and > here instead of < and > in the raw file that the translators will see, I think it is clear which is going to be easier to work with in the absence of a HTML/Qt Rich Text display widget...

I think I read somewhere that CJK languages really should not use Bold or Italics font-effects (instead I think they use the equivalent of a dot accent at one edge of the glyph) - so in those cases the mark-up could be involved in the translation process...

[vadi2]

Running shell scripts isn't portable (for Windows), so I'd rather not do that. The current mudlet_en.ts file seems to be working out fine?

It's Friday so let's reach a conclusion on this in the next couple of days so we can get started on the next steps, which I think would be:

1. process for translators to report confusing strings - if a translator can't understand it, chances are our users don't either
2. actual translation process and how would it work with our workfow

[Kebap]

I would recommend first thinking more about the details of the processes we need, and only then decide which tool fits them best.

As far as I can see right now, we have a few different main sources of text:

• Mudlet client itself
• Mudlet website
• Mudlet wiki
• "other" spontaneus texts need to be included manually

Then the processes for each are much the same, but need to be reviewed seperately:

• Find out a way to tell outdated texts from relevant current texts which need translation.
• Get texts out of sources (see above). Can this be automated with any tool? Who should be responsible?
• Get texts in to translation tool.
• Translating inside tool (ok this is to be expected, but also needs good interface and workflows)
• Translators report confusing strings back to Mudlet team with information how they need to be adjusted.
• Mudlet team updates text in source. Either because of reports, or because of natural evolution.
• Updated texts from source need updates in translation tool. Export and import again, plus diff.
• Translated texts will be finished. Export them from translation tool. Automation again?
• Import translated text to source.
• Display of translations in source. Here we will need some sort of language toggle switch in each source.

I noticed some plugins for Wordpress and Mediawiki, as well as integration in Crowdin and Transifex, but did not have much time to research all this in more detail.

I would love a most automated process for all steps above, and choose whichever tool can support best.

[SlySven]

As far as source code/dialogue(forms) go:

• Find out a way to tell outdated texts from relevant current texts which need translation.
• Get texts out of sources (see above). Can this be automated with any tool? Who should be responsible?
  • Both are done with lupdate command which extracts source texts from, guess what, the source code and adds/updates the records of them in the - what I think should be language neutral mudlet.ts file (the current mudlet_en.ts signals to some parties that it specifically holds English language translations - whereas - with how we will be using it as the source code ==> Crowdin date transfer medium it will be used for all but the English(American) case. If not invoked with the --no-obsolete option then existing but now gone strings will not be removed - which is what we will want at least during the development stages. The reason I propose to generate a separate one for the en-US locale because that will be a very short one which only contains strings for plurals i.e. places where we would have source strings such as "deleting %n room(s)" so that, for that case either "deleting 1 room" or "deleting 2 rooms" are used.
• Get texts in to translation tool.
  • The process of updating the development branch should cause the above to happen and the updated .ts to be uploaded from (I suggest) the linux CI build (because it is easiest for us to script for)
• Translating inside tool (ok this is to be expected, but also needs good interface and workflows)
• Translators report confusing strings back to Mudlet team with information how they need to be adjusted.
  • It looks like crowdin allows anyone to make a comment or raise an issue about any source string which will be at least visible to manager (but may be open to more parties)
• Mudlet team updates text in source. Either because of reports, or because of natural evolution.
  • Including places where the source text is confusing or misspelt from the expected en_US form - e.g. I may have used licence (en-GB noun, c.f. license as en-GB verb) but it is always license in en-US for both forms.
• Updated texts from source need updates in translation tool. Export and import again, plus diff.
  • re-run lupdate - but this should happen naturally via CI scripting on 'development' branch updates
• Translated texts will be finished. Export them from translation tool. Automation again?
  • One of us can "Download translations" as a full set of mudlet_xx_YY.ts files manually from Crowdin and git commit them into the /translations directory - remember that these files are not updated by lupdate except, perhaps, for the plurals-only mudlet_en_US.ts one - instead they are generated at Crowdin from the mudlet.ts file.
• Import translated text to source.
  • Not quite what is required, the translation texts are not read by the application from the .ts files. Instead, once I get the code revised and out in public, the mudlet application will read the translations at run-time by loading the selected mudlet_xx_YY.qm binary file (along with the appropriate Qt one - which I note we are already distributing with the installer versions!) So initially I imagine one of us will run lrelease to convert the .ts files into the matching .qm files and commit them to the git repository as well. Eventually Vadim has suggested we will ship the release versions with a set of .qm files included as a resource file. This is appropriate for installer versions but Linux packagers will instead expect to store them in, say a read-only /usr/share/mudlet/translations (alongside the external Mudlet lua files). I have prototype code that will allow both of these arrangements to be handled but by allowing the user to load translation files from elsewhere (possibly from the /translations directory in the source code) it would permit stand-alone development of translations for other minority languages by interested parties with the original Qt tool Linguist - it would also allow developers to work on private/test cases.
• Display of translations in source. Here we will need some sort of language toggle switch in each source.
  • From experimentation using the QLangaugeEvent is more complex than we need as it fires every time QQTranslator::load(...) is called - I've found it is simpler to have the mudlet class emit a (void) signal_translatorChangeCompleted(const QString&, const QString&) where the arguments of current and previous xx_YY codes (language_COUNTRY/REGION) are largely unused except for, IIRC the Host which generates a lua event for script/package event handlers to work with. That signal is sent to every class with persistent GUI texts to a slot_guiLanguageChange() where it calls the required retranslateUi(this) that is defined in every class that also uses setupUi(this) provided you set up the option in Qt IDE correctly (I dunno how this works when using qmake directly):
    
    that call will change the translations on every translatable string in the forms/dialogue and then we have to regenerate all the GUI texts that we create at run-time, taking into account what ever conditions may have caused different texts to have been put together. This is not as bad as it sound because we only need to alter persistent - i.e. is around for a long time - string because the transient texts will automatically use the new translation whenever tr(...) is used to create them.

This is not all I have on this topic but I can see how most of the steps should go for application GUI texts...

[Kebap]

Thanks for those details SlySven! It seems to me, maybe you have done this before? For another application? With Corwdin? That would make translating the Mudlet application itself much easier.

On the other hand, we still have the same questions for the other text sources in Mudlet universe, like wiki documentation, Mudlet website, and additional texts which may need to be incorporated somewhat manually, as explained in my last post above.

[vadi2]

I haven't gone through the answer above fully but Crowdin has Github integration which it does not account for - #1692 is an auto-generated PR, we will not need to do any manual uploading or downloading; simply merge the PRs. Thus the process will be fair bit simpler than described!

[vadi2]

Guys crowdin trial will expire in 4 days so I need a yay/nay on it - we seem to be happy with it so far

[keneanung]

There was a yay from me!

[Joker-ITA]

Hi! ;)

I've tried https://crowdin.com/ … for translate it's ok, i don't know if is good for quick feedback, requests and so on… like "contestualization" for a translation or example...

[vadi2]

What about the comments section in crowdin?

[Andrulko]

Hi @Joker-ITA,

There are actually several ways on how you can add extra context for translators:

1. In the editor, click "Edit Context" under the string to provide more info to translators;
2. You may leave a comment for any string in the editor (on the right panel, type your comment);
3. Add context for any string via the project settings -> Strings tab;
4. I bet you have some specific terms which regular translators may not understand, thus it makes sense to create a glossary and upload it to Crowdin. More info. It can be even Excel spreadsheet with 2 columns: Term & Description;
5. If you have comments inside files (i.e. Android XML, iOS strings, .properties, .yml, Google Chrome JSON), they will be imported automatically along with the file;
6. You can upload the screenshot to Crowdin and tag any string on it: https://support.crowdin.com/adding-screenshots/
7. If we talk about webapp l10n, it makes sense to try our In-Context feature to allow translators to translate directly on your website.

It's only major ways how you can contextualize texts for translation 😄

[SlySven]

@Kebap wrote:

Thanks for those details SlySven! It seems to me, maybe you have done this before? For another application? With Corwdin? That would make translating the Mudlet application itself much easier.

Only manually using Qt's own tools (I spent a couple of weeks in the South of France in the last couple of years working out the details). We will still use two of them - lupdate & lrelease - but we are replacing Qt Linguist (being the tool that adds the translations to the source texts present in a number of .ts files generated by lupdate for each translation, before lrelease turns them into the language/location specific mudlet_xx_YY.qm files needed - again one per language/location translation offered) with Crowdin which will take a single mudlet.ts† file and generate the mudlet_xx_YY.ts files instead.

One important difference between using linguist and what we will be doing using Crowid is that we can get away without a:

TRANSLATIONS =

line in the project file for Crowid.

† Actually I think we will need two files - mudlet.ts for all locales and which will go through Crowdin process and mudlet_en_US.ts which will be extracted with the -pluralonly option and will only have a few translations to convert messages like tr("deleted %n room(s)", "", roomCount) to "deleted 1 room" or "deleted 2 rooms" depending on the value of roomCount and can probably be done by yours truly...

[vadi2]

Great - we're going with Crowdin :)

I will adjust the filename like @SlySven suggests and setup the project in Crowdin proper so people can start translating. Any existing test translations we've done will be saved in the Transaction Memory so re-applying them will be pretty easy.

[vadi2]

Answering @Kebap's questions, taking into account @SlySven's answer:

• Find out a way to tell outdated texts from relevant current texts which need translation.
  • lupdate tool does this
• Get texts out of sources (see above). Can this be automated with any tool? Who should be responsible?
• Get texts in to translation tool.
  • we manually run lupdate and commit them to development - this part could be automated on a daily basis by Travis
  • crowdin automatically picks them up due to the github integration
• Translating inside tool (ok this is to be expected, but also needs good interface and workflows)
  • crowdin's translation interface
• Translators report confusing strings back to Mudlet team with information how they need to be adjusted.
  • write an issue down in crowdin
• Mudlet team updates text in source. Either because of reports, or because of natural evolution.
• Updated texts from source need updates in translation tool. Export and import again, plus diff.
  • crowdin picks it up
• Translated texts will be finished. Export them from translation tool. Automation again?
  • yep, crowdin auto-opens a PR: New Crowdin translations #1717
• Import translated text to source.
• Display of translations in source. Here we will need some sort of language toggle switch in each source.
  • @SlySven will do magic here :)

[vadi2]

What is the need for having a separate translations file with only the plurals?

[vadi2]

Translation instructions are available at https://wiki.mudlet.org/w/Translating_Mudlet