Migrate .json language catalogs to .po

[bram-atmire]

Given the support for .po as the message format for the catalogs in #306, this issue can be used to track the migration.

[bram-atmire]

First experiments:
bram-atmire@40efacf

http://bram-atmire.github.io/100daysofcode/2018/11/27/100-days-of-code-day-91-My-little-po-file.html

[bram-atmire]

Progress:
bram-atmire@4b4ef4e

http://bram-atmire.github.io/100daysofcode/2018/11/29/100-days-of-code-day-92-Message-extract.html

[bram-atmire]

Progress:
bram-atmire@b3a4bc6

The Dutch language file is now more or less complete in .po, also worked on the German file and did a few attempts at automating this conversion.

http://bram-atmire.github.io/100daysofcode/2018/12/01/100-days-of-code-day-93-Po-file-editing.html

[helix84]

Hi Bram, I happened upon your work and skimmed your dev blog. Thank you for taking this up.

There's a conceptual difference between how gettext approaches translation and how Angular seems to do it that I wanted to explain.

When you use gettext in C, Python, PHP or what have you, you replace hardcoded messages in your source code with calls to the gettext() function, typically aliased as _() because it's used often. So you would see _("English original") in the source code rather than "English original".
When you want to create/update your .pot, you would call xgettext, point it at your source code and it extracts all the original strings. This yields a .pot with msgid "English original" and msgstr "".
When you want to update all the .po files to match the new .pot, you call "msgmerge -U -c nl_BE-old.po nl_BE-new.po new.pot" for each .po file. Or you do it from your desktop .po editor ("Catalogue > Update from POT file" in poedit) or your web UI like Pootle would do it in the background without bothering the translator.

If the same message is used in two (or more) different places (contexts), it would be automatically deduplicated and the same translation would be displayed both contexts where the original is used. You can imagine that an original string like "Display" could mean "Monitor screen" in one place and "Draw on screen" in another and require two different translations.
That's why you have the option to distinguish between the two contexts where the message is used using msgctxt, which is an arbitrary key, typically the module name. In the .po and .pot, the pair of msgctxt and msgid is then treated as unique (rather than just msgid).

I'm still not familiar with Angular, but it seems to make things harder for the developer (just like Cocoon did) by pushing out the English messages out of the source code, store them in a json file and refer to them from the source code only by a key.
I hope you can see the advantage of seeing the English strings right in the source code and letting the tooling deal with the extraction. I don't expect it would be easy to change how Angular does things. Perhaps look if someone has already done it (I haven't really looked at it, but https://www.npmjs.com/package/gettext-extract might be the equivalent of xgettext for JS in general). So what you have done - using keys as msgids is the best workaround.

[bram-atmire]

Hi Ivan, just saw your comment now at the end of my work today. Indeed, introducing the msgctxt and the original English translation into the code, instead of working with those artificial message ids might make it more standard & better, but this would require us to change the code in every single place where keys are being used today. What do you think? Should we do this? Manually? Would it be worth the effort?

Progress: bram-atmire@f0fd05d

Report: http://bram-atmire.github.io/100daysofcode/2018/12/03/100-days-of-code-day-95-Po-file-editing.html

[helix84]

Msgctxt is a side-issue. The main point is whether it's possible and advisable to introduce English messages into the source code. Please, mention it to your JS programmers. I expect them to have strongly negative first reaction to the idea as it's the opposite of what they're used to. Let them "sleep on it" for a few days. They'll figure out whether it's a good idea to deviate from the Angular convention of working with messages, which might be baked into its code. Don't worry about work in reformatting po/json/js files, that's just grunt work and I'd be happy to help after we figure out if it should be done.

[bram-atmire]

we discussed it. There were a few concerns but i'm building demonstrator code to alleviate them.

http://bram-atmire.github.io/100daysofcode/2018/12/07/100-days-of-code-day-97-Po-concerns.html

[bram-atmire]

@helix84 tl;dr Based on what I could find, there is no support for msgctxt right now, but it might not be too far off, since there is already a pull request. More details:
http://bram-atmire.github.io/100daysofcode/2018/12/09/100-days-of-code-day-98-Po-concerns.html

So unless I'm missing something, if we give up the use of the unique message id's, we currently can't allow different translations (at least for the time being) anymore for keys that have exactly the same original value.

I'm not sure how big of a problem that is, but we already have a few keys out there that have exactly the same message today. When I see the current examples, I would personally conclude this is a non issue, and that these particular keys will very very likely be translated the same in other languages. And it would actually make the message catalog cleaner and leaner if they were NOT to show up as duplicates.

loading.default
loading.objects
form.loading
"Loading..."

filters.applied.f.subject
filters.filter.subject.placeholder
filters.filter.subject.head
"Subject"

@cjuergen what's your opinion in this discussion, since you contributed the German language before?

[bram-atmire]

Just to make sure I keep some track record of this: Samir Hachani from Algiers University has volunteered to assist with getting the Arabic translations in place, once we are at a point where we have settled on the .pot (template) file.

[bram-atmire]

Hope I can get the batch substitution of message keys with the English messages done with Repren http://bram-atmire.github.io/dspace7/2018/12/17/DSpace-7-Po-Transition.html

[bram-atmire]

@helix84 and others interested: last progress with the batch renaming using repren:

master...bram-atmire:issue-331-migrate-json-language-catalogs-to-po

[bram-atmire]

Massoud M. AlShareef from Knowledgeware has also kindly offered assistance with the Arabic translations and specific RTL (right to left) challenges.

I will try to pick up this work and finalize the transition to .po in the course of february, as this is currently a blocker for translators to start their work.

[bram-atmire]

Worked more on this today:
https://github.com/DSpace/dspace-angular/compare/master...bram-atmire:issue-331-migrate-json-language-catalogs-to-po?expand=1

In short, I think we're now very close to being able to merge this and to retire the json message catalogs. Would be handy if devs stop adding new keys to the json catalogs soon, so we can make the cutover.

[bram-atmire]

Issued a PR for this for reviewing purposes, to see if there are any remaining blockers to go forward with this #366

[cjuergen]

Hi,

actually the same string might be translated differently and there are language specific things like declensions etc.
Just remembered after having translated the messages.properties once, I ended up with the page aborting a workflow with two button both labelled "abbrechen", because cancel and abort were translated this way. This is the other way direction, just the first example, I remembered.

[helix84]

Yes, Claudia that's what message context is for. The typical case would be where the original string can be used both as a noun in one context (message) and as a verb in another, e.g. "type" (type of object vs. to type on a keyboard). If strings are aggregated (into a single message catalog for translation) from across the whole application and multiple uses are merged into one source string, there's no way to translate these correctly in all contexts.

[bram-atmire]

As also summarized in #306, the message format will be JSON5 as we ran into to many big roadblocks when attempting the po migration.

Even though the message format, as part of the codebase, is not .po based, I stay very committed to develop conversion tooling that makes it easy to work with existing translation tools. For that purpose, we can keep the following open:

#85