Automate push/pull translations from Crowdin

[luzpaz]

ref: https://tracker.freecadweb.org/view.php?id=3715 and https://forum.freecadweb.org/viewtopic.php?f=10&t=30340&p=272480#p272480

• Create a script that further automates the current scripts that we use to push/pull translations from. Perhaps on a weekly basis? Make it intelligent enough to operate in certain parameters:
• push only languages/strings that have been translated during the period between the previous pull and the current pull
• push only languages/strings that have been proofread (this may present an issue for languages that don't have proofreaders)
• determine the rate at which we push new translations to crowdin and pull from it.
• code a friendly way to disable script efficiently if needed.

[chennes]

Technologically this is straightforward: in fact, anyone with the appropriate level of access could do it right now with just a few lines of bash shell script. I don't have the right CrowdIn rights, so it can't be me. I guess that leaves it to @yorikvanhavre ? We can also automate the git end of things, though it might be best to create a new Git account similar to what I did for the Mantis bug import, rather than having the commits tied to an individual developer. If we really wanted to we could run this sort of thing as a cloud process someplace, but that's probably overkill.

[kaktusus]

Crowdin level scripts can only be executed by project managers. Currently there are 6 chosen ones.

[ronnystandtke]

Yes, please automate this process. A very short round trip time is essential for efficient development.

A script that makes it possible to update and test translations locally before committing any changes to Github or Crowdin would be a very welcome addition.

See FreeCAD/FreeCAD#9275 for some additional comments.

[chennes]

Yes, please automate this process. A very short round trip time is essential for efficient development.

I don't think the system envisioned here is of that much use to translators in terms of making a shorter round-trip: we're talking here about something that is run on the order of weekly and serves to add predictability to when translations are available to end users.

I think there are two other groups of people with different automation needs. First, developers who are modifying the source code to enable/fix translation problems. For them, you really want to bypass the CrowdIn upload/download entirely: all a developer cares about is, given a change to the source code:

1. is that string available for translation (e.g. is it in the *.ts file)?
2. if a translated string is provided, does that string appear in the UI?

This doesn't need CrowdIn at all, it needs a script that runs lupdate locally, verifies that some expected context/string is present, generates some kind of dummy language file, injects a fake translation string into it, and processes to a *.qm file. Then the UI can be examined for the presence of this string.

I suspect translators have a different need: given a (real) translation string, what does it look like in the UI? So pull down new CrowdIn data, process it, apply it, and make it available to FreeCAD at runtime.

Does this fit with your understanding?

[ronnystandtke]

Hi Chris

Thank you very much for your additional comment. You are right, there are at least two different groups of contributors to i18n/L10n:

• Translators (who most probably just run the binaries of the stable or development versions of FreeCAD, see some missing or incorrect L10n and contribute these back to Crowdin)
• Developers (who cloned the FreeCAD sources from github and are able to change and compile the program locally on their machines)

As you already mentioned, translators could profit from a helper script that downloads and applies all translations that are provided in Crowdin at this very moment. This way they could more or less instantly check how a new or updated translation looks on screen and improve things right away.

Developers already have a very short round trip time because they can change, compile and run everything on their local machines. The "interesting" group is now the intersection: developers who work on i18n/L10N. For this group the situation is still a bit awkward and this is the group I ended up in very shortly after my first steps contributing to FreeCAD. :-)
I live in Switzerland, where we have four official languages (German, French, Italian and Rhaeto-Romance) and I wanted to check how changes in the source code or in a locale specific *.ts file look like in the running program before committing them back to github or Crowdin.

A typical workflow would look like this:

1. Make changes to the source code, including changes to some user visible strings.
2. Run helper script A, that runs lupdate for updating all relevant *.ts files of all languages.
3. Update or add translations to the *.ts files of the languages you master (using Qt Linguist or a simple text editor).
4. Run helper script B, that runs lrelease to update all *.qm files for all changed *.ts files.
5. Compile, run and verify the changes on screen, jump back to (1) or (3) until everything looks great.
6. Commit your changes to github and/or Crowdin.

As long as Crowdin is the single source of truth wrt L10N, the workflow above most probably must make sure to ignore all local changes to the *.ts files.

What do you think? Does that sound reasonable?

[chennes]

Your proposed workflow is completely possible right now: we have a helper script to run lupdate on the entire package (updatets.py), and we have a script that runs lrelease (updatecrowdin.py). Everything else is just normal compilation workflow up to step 6b (update your translations on CrowdIn -- I don't know how to do that from the local translated ts files).

[ronnystandtke]

I don't see how I can use both scripts right now to get to the proposed workflow.

As far as I can see, updatets.py so far only updates the locale agnostic *.ts files, these only contain unfinished translations. To be able to update or add translations, the script for step (2) must also update all the locale specific *.ts files, e.g. FreeCAD_de.ts, FreeCAD_fr.ts, FreeCAD_it.ts, ...

The script updatecrowdin.py seems to work only for zip files downloaded from Crowdin. This seems to work only for Crowdin managers. Even with my Crowdin token set up, I only get a "HTTP Error 403: Forbidden" when I try to download the zip file.

[chennes]

Doesn't step 3 in your process above create/modify the locale-specific TS files? I normally do it by hand, not with QtLinguist, but I imagine it's much the same process either way. Then I manually run lrelease on the file I am working on to generate the QM, then I recompile FreeCAD.

[ronnystandtke]

While I can modify locale-specific *.ts files with Qt Linguist or by hand, the strings that got changed or added in the source files in step (1) are still missing. The locale-specific *.ts files are completely untouched after running the current version of updatets.py. Automatically running lupdate also for all locale-specific *.ts files to merge all source code changes into them would probably help here. Are there any reasons against this?

[chennes]

@yorikvanhavre Take a look at this GitHub CI integration workflow: https://github.com/crowdin/github-action

What do you think about setting it up to run once per week? Or maybe even more?

[chennes]

Are there any reasons against this?

I don't know anything about what happens when you run lupdate on those files: we always generate them from CrowdIn. Does lupdate simply add the missing strings?

[yorikvanhavre]

@yorikvanhavre Take a look at this GitHub CI integration workflow: https://github.com/crowdin/github-action
What do you think about setting it up to run once per week? Or maybe even more?

Yes that would be fine.

For me before doing this, there is one preliminar step we should achieve: Remove the qm files entirely from the source, and build them at build time.

When that is working, then we can set up a crowdin integration. That would indeed offload parts of the work and make merges more predictable. But indeed that still leaves us with one manual part, that is to regenerate the base ts files. For this I don't see much what we can do to automate.

[kaktusus]

QM file building can also be done in a script.
See how translators do it in a local environment for testing.

I wonder how I could have missed such an important discussion. 🤨

[yorikvanhavre]

Building qm files automatically at build time is almost dome now, there are still a few modules to adapt. @chennes what was the reason those were left out? I don't remember...

[kaktusus]

all in all, I see it as optimizing the process.

[chennes]

@wwmayer did all of the C++ modules, I think it's the Python Mods that remain, their resource system works a little bit differently. Someone just needs to do it, I think (my cMake skills are pretty weak so I'm not the ideal candidate...).

[yorikvanhavre]

I'll have a look at it, but I suffer from the same sickness 😅

[kaktusus]

Such an important topic stuck or abandoned?

[yorikvanhavre]

Not at all, we all want it of course

[kaktusus]

We have a working solution developed.

https://github.com/OPGL/FreeCAD/blob/main/.github/workflows/update_translations.yml

I suggest that:

• do not remove context data from the import.
• scheduler performed operations no less than 3 times a month (at first, test, even weekly)

@OPGL I want to thank you for your contribution and commitment.
This is a long-awaited functionality.

[yorikvanhavre]

Nice work! It still needs an additional step to build the zip file on crowdin before downloading it (updatecrowdin.py build), and the problem is we don't know how long it takes... also I'm not sure where the crowdin token should be stored?

[kaktusus]

You can watch a functional test here:
OPGL/FreeCAD@3cb0c2e

😉

[OPGL]

Greetings,

Thanks for you answer, i'll fix building zip today. Why length of building that zip is problem?
Also CROWDIN_TOKEN in stored in repository secrets, it's automatically encrypted and decrypted when needed in github actions.

[yorikvanhavre]

Excellent!!
the length of building is uknown, crowdin does not tell when it's done. With our script we use updatecrowdin.py build-status to check when the build is done (only one build left with 100% done). Maybe we need to add one more command to that script, something like "build-and-download" (which would continuously poll until the build is done)?