Language Packs

[zcwilt]

UPDATE: THE ZEN CART CROWDIN PROJECT IS LOCATED AT: https://crowdin.zen-cart.com
... which links to https://crowdin.com/project/zen-cart

---

Most language packs available in https://www.zen-cart.com/downloads.php?do=cat&id=6 are seriously out of date and generally only support the old DEFINE type language files and not the new array type language files.

They are also generally not updated regularly, when we get new Zen Cart versions and new language strings are added.

To try and help with this, I have created a Laravel console application that automates translation using the Google translate API.

The repository is https://github.com/zencart/translate

The documentation is a bit light at the moment and I will be updating that shortly. You need to have tokens for Google Translate and Github.

While Google Translate does a good job, there are times that it fails and the code allows for adding overrides for specific translations.

The plan is to update the current Plugin manager code, so that translation packs can use the zc_plugins directory to manage translations.

[torvista]

After many years keeping the spanish language pack up-to-the-minute for no apparent reason, I would attest that a language pack takes a very long time to reach a state that could be considered "done". So should be treated as something that is never finished/subject to constant change and not a static plugin.
Hence this should be borne in mind in how it is managed, to both encourage that constant change/improvement, and the updating/including of those improvements in sites that use those packs.

[zcwilt]

Thanks @torvista
I agree that manually supporting a language pack can be hard, especially against ever changing core code.
Hence why we are looking at using automated tools to help with this.

There will be challenges, especially around situations where the automation produces incorrect translations. This especially for
grammar, idioms and expressions.

I'm still working on some ideas to help with this, and will post more here about that.

[zcwilt]

At some point I'll try the automation on Spanish and hopefully get your feedback on how bad it is :)

[zcwilt]

One thing I'm looking at is how contributors can create a language pack using the laravel app.

Currently this means that they have to install code locally, create a token for the google translate API and have a github token to allow access to a repository. This is a bit onerous, so the idea may be to add a UI interface to a hosted version of the laravel app
which would produce a downloadable zip.

[syntaxerror]

As a long time translator of ZC and other software, for small Nordic languages;

While the idea is praiseworthy, I don't see it as much realistic that Google Translate (or any other AI translator) will produce anything useful (as of today) for most of the languages, except for maybe the top 5 western languages (Spanish, French etc.) - of ap. 7000 languages in the world (195 countries).

Including or providing bad translations would be equally bad as non translations, especially for the stores as people would avoid them, as they looks fishy with bad translations...

Using resources on getting t.ex. Crowdin up running would probably have been better, building an active translator community, you can also integrate it with Google Translate and others to get proposals and automations.

The reason for most languages being outdated is simply because it's a huge manual task migrating (from define to array) and keeping the translations up to date. It requires old fashioned tools like notepad and diffs, side by side comparisons - and it's mostly not language team friendly and thus becomes a one man task.

Just my 5 cents. ;)

[drbyte]

Just my 5 cents. ;)

You raise good important points. Much appreciated!

[drbyte]

Including or providing bad translations would be equally bad as non translations, especially for the stores as people would avoid them, as they looks fishy with bad translations...

Thanks for putting words to something I've often felt but never articulate.

[drbyte]

I don't see it as much realistic that Google Translate (or any other AI translator) will produce anything useful (as of today) for most of the languages, except for maybe the top 5 western languages (Spanish, French etc.) - of ap. 7000 languages in the world (195 countries).

Yes, good point. Much AI is geared around a few very limited scopes.

it's a huge manual task migrating and keeping the translations up to date.

Yes. Unfortunately there are a lot of necessary manual tasks involved in keeping things in sync across upgrades.

[drbyte]

getting Crowdin up running

This is definitely the easiest way to crowd-source contributed content.

Incorporating that content back into core is quite challenging, and that's mostly what stalled that (still open) project.

[syntaxerror]

Unless you plan to incorporate the translations for various languages into the core, there is no need to incorporate any content back into the core. Auto syncing the core language files from GitHub to Crowdin is the easiest way, they don't need any syncing back into GitHub.

The main language files for admin and catalog requires tons of rearranging of the lines, so I created my own project at Crowdin where I uploaded those files, and then uploaded my converted Norwegian files as translations, built the project and downloaded them ready to go. Such a enormous timesaver: norskzencart/norwegian-zencart-translation@7938db4

[syntaxerror]

Here's a complete Norwegian language pack exported from my Crowdin test project, being temporary available at https://crowdin.com/project/zen-cart-demo

First I converted the old language pack from pre-1.5.8 using the tool found on your site, then uploaded it to the Crowdin project (which is synced from GH master), resulting in the Norwegian translation to be close to 60% done. Then I ran pre-translation using the Translation Memory in the Crowdin project, before running also the Machine Translation, resulting in 95% translation - and it's ready for proofreading ...

Zen Cart Demo (translations).zip

[syntaxerror]

Note! Since Zen Cart is using lowercased language names instead of iso codes etc., you might also need to add language mapping in Crowdin to get the correct names in the language exports, since this isn't supported by default in Crowdin as I can recall.

So in the Crowdin project, visit 'Settings -> Languages' and scroll down to the bottom, selecting the 'Language Mapping' button. Then add the custom code for each language, or maybe consider switching from using 'english' to instead using 'en-US'/'en-GB' (locale).

[torvista]

As always, I wonder why not start from an endpoint and work out how to get there.

New language packs could be AI-generated as a starting point: it's easy for users to modify but not create.

My imagining of an ideal would be the admin user doing a one-click install (download) of a translation (which would have an indication of its readiness/how much human-checked it is).

I see two modification mechanisms necessary:

1. be able to correct the generic texts to pass back to the source for review and inclusion.
2. be able to override the generic texts with site-specific: stored in the site.
   Both should be Admin-user accessible. It's crucial to make feedback/modification as easy as possible for non-technicals to move away from language-pack maintenance being a one-person/thankless job.

Seems a huge amount of work...how important is this in the priorities/roadmap for Zen Cart???

[syntaxerror]

You can do that also by using Crowdin, as there's a feature to pre-translate all added languages using both Translation Memory and Machine Translation (AI) from any chosen translation tool.

And as many already have a language pack for several languages, they can also be uploaded as a starting point, before running the pre-translate feature for the missing strings, as I just tested successfully.

Both the import and pre-translate should be run without setting the strings as approved, and by that giving full control on the proofreading to the translator / translation team knowing the language. Crowdin also have this voting feature on translations, letting people contribute by voting on the best translation proposal.

Instead of spending days rearranging the lines in the files, I spent an hour setting up the project, and uploading the existing translations to let Crowdin do the needed rearranging. Then by using the translation memory, they are ready to be proofread already...

[syntaxerror]

be able to correct the generic texts to pass back to the source for review and inclusion.

Why? What's the need to pass anything (and what) back to source???
*Source = English core language

[torvista]

By source I meant the translated language....since when I see a mistake I'd want to flag it/offer an improvement easily in that moment, somehow.

[syntaxerror]

And that you can do easily directly in Crowdin ... :)

[torvista]

In any case, congratulations on the great progress you have made, I hope it leads to some concrete decisions from the maintainers/some procedural documentation and allow this aspect of Zen Cart to progress under its own steam and encourage contribution by the less-technical community.

[zcwilt]

@syntaxerror Thanks for the update and explanations around Crowdin.

However I think we have already run into an issue.

As some languages have been auto translated, we have now hit a limit on the number of translation words we have.

I think the free limit is 60,000

Email from crowdin is

As a result of a recent source files update, you’ve exceeded the source strings limit for your Crowdin account (username: zcwilt). Please upgrade your plan to keep making more сontent multilingual. Otherwise, your account will be suspended within 10 days.

Upgrade your Crowdin account: https://crowdin.com/pricing

If you have any questions or need help, please contact us at support@crowdin.com

Kind regards,
Crowdin team

[zcwilt]

I might try reaching out to crowdin to see if they can offer some support for open source, given they have this
https://crowdin.com/page/open-source-project-setup-request

[syntaxerror]

Glad you figured it out, it's indeed the way to go, and I see no reason why they shouldn't accept Zen Cart as a Open Source project. :)

[zcwilt]

Just got an email to say that they have, and the error banner that suggested the account would be disabled has been removed

[torvista]

Crowdin
I've had a look, how do I update the Crowdin Spanish with my 100% complete spanish from Github? The only import is Xliff.
Or even better, can someone else more au fait with this, do it ? I feel like I've done enough with this pack over the last 13 years.

[syntaxerror]

It's possible for admins to import packages, from zip or by using CLI, but the translator credit for the strings will be set to the person importing.

zc_install isn't added yet. And I don't know if the team will go for Crowdin, it's still displaying the text for testing, so let's wait and see before doing this work importing. ;)

Edit! A user (at least proofreader) can also upload files one by one.

Docs: https://support.crowdin.com/uploading-translations/

[torvista]

ok!

[syntaxerror]

I've tested uploading your Spanish package using CLI, haven't done it since 2015 but got it working.
Uploaded it as approved, just to test that also: https://crowdin.com/project/zen-cart-demo/es-ES

[torvista]

Awesome! Thanks for that.
As this seems to still be in proof-of-concept, I'll not finish it off for now, until there is a definite decision that it'll be used.
Easier for me to maintain my Z200 branch for the moment on GitHub.

[syntaxerror]

Sure that's anyway just my test copy.

[drbyte]

Given it's relatively easy for people to extract/download their own language packs from Crowdin after (whatever amount of) translating/proofreading is done, then I see no reason not to make this the preferred approach, and remove the "testing" text from the description.

A few things probably need to be sorted out first ...

[syntaxerror]

Best and easiest would probably be to use locals (without any lower-casing) like en-US (American English), en-CA (Canadian English), en-GB (British English) etc., because the languages have variations between the countries where they are used, that's also the default in Crowdin etc. (also for CLI commands etc). If there actually is enough interest and translators for maintaining several dialects in ZC is another mater though, anyhow it wouldn't require any language code matching per language then, but ofc also upper-cased language names should in theory work. Drawback is that store admins would anyhow need to change their language config for folder names, if any change is made to the naming, but that they would need anyway if the language code mapping is missing for their language in the Crowdin project - and that's easy to forget when adding new languages.

[syntaxerror]

Note! I also recommend creating a cron job on your servers, to automatically build the downloads (update zips) on Crowdin at least every 24 hours (or more frequently), else I think it only happens automatically when a language reach 100% - or whenever it's being built by a project manager pushing the build button.

https://developer.crowdin.com/api/v2/#tag/Translations

Ideally you also create a page on zen-cart.com displaying the progress per language, and a download button for each languages zip (which requires a cron to download the zips to your server first), plus info about contributing and a button/link to the language on Crowdin.

[syntaxerror]

html_includes: include in translation packs, or not?

I'd vote for including.

Do more things come to mind?

Incorporate the flags in core:
https://github.com/joomla/joomla-cms/tree/5.0-dev/build/media_source/mod_languages/images

[torvista]

How would you handle a pack for the current release version and a pack for the ongoing development version, which becomes the release at some point?
I include the development additions as they are occur, as that is also my production site.
This I do easily with two GitHub branches.

[syntaxerror]

https://support.crowdin.com/versions-management/

[scottcwilson]

@syntaxerror Would you be willing to write a blurb on CrowdIn for https://docs.zen-cart.com/dev/languages/creating_a_language_pack/ ? If you can submit a PR that would be great, but if not just pasting in some instructions would work too. Thanks for your contribution to Zen Cart!

[syntaxerror]

Not sure what you're asking, but if you're thinking about some docs this could probably help: https://docs.joomla.org/Crowdin

[torvista]

So, I can download the entire structure en-bloc as Zen Cart (es-ES).
This I can compare with and merge from my Github translation as a one-off to get this in sync from now on,....but then the only upload I can find is as Xliff.

So think an admin needs to upload this for me.

Aside are we agreed that this is the place for translations from now on?

[torvista]

Just checking one file and it has the text prior to what I supplied yesterday.

I also see there is no option to mark a translation as "fuzzy"/requires review, as I am used to seeing in other translation apps.

If a string/pack has been machine-translated as an initial step, how is it marked as needing human-review?

[syntaxerror]

Just checking one file and it has the text prior to what I supplied yesterday.

Again, a build is required to update zip's. Until the team set up anything to run it automatically some admin has to push the button.

I also see there is no option to mark a translation as "fuzzy"/requires review, as I am used to seeing in other translation apps.

Anyone else than WP/Poedit? ;) A string is fuzzy/requires review until it's proofread and marked as so (indicated by green).

Some docs, also posted previously:
https://docs.joomla.org/Crowdin
https://support.crowdin.com/
-> https://support.crowdin.com/crowdin-intro/
-> https://support.crowdin.com/online-editor/

[torvista]

Again, a build is required to update zip's

Sure, but I'm now talking about the online file being previous.

Fuzzy/Not Approved/Needs Editing (Weblate)...there's a pretty clear indication on a string of it's state.
In Crowdin I see don't see anything on the string edit page to indicate it's state.

[syntaxerror]

Sure, but I'm now talking about the online file being previous.

Sure, but you're being unclear. So better specify more what you actually are talking about, and include some more details to verify your issues. ;)

According to the activity log, your language strings was uploaded yesterday by Chris, which selection he did, e.g. to overwrite existing and auto approve (proofread) we can not see:
https://crowdin.zen-cart.com/project/zen-cart/activity-stream

Chris Brown (drbyte) uploaded 2,947 translations into Spanish

In Crowdin I see don't see anything on the string edit page to indicate it's state.

Please look at the docs, especially about the translation and proofreading flow.

The amount of proofread strings are indicated with green colour in the folder/file view:

In the editor they have a green mark indicating they are proofread, both in the list and proposed translations. If not approved (proofread), the have a blue square in the list and a grey V in the proposals. Just hit the grey V on the proposed translation you want to accept, or add your new translation and delete the faulty ones.

[drbyte]

I mistakenly didn't run a build after importing yesterday.
But, done now.