Added script for updating messages and basic usage help for other message scripts

[cyplas]

Main change: new script update-message-xml-translations.py, which may be handy on forks with other languages during upgrades.

[cyplas]

Aha, also made the existing scripts independent of dspace-l10n-check.py, which is still python2 (and perhaps not worth updating?).

[TomazErjavec]

Well, if this goes to UFAL, then it would be nice to have them all in Python 3, methinks.

Just to clarify: the scripts are now in Python 3. That's part of the reason why I've made them independent of the pre-existing dspace-l10n-check.py, which is still in python2.

[kosarko]

Does this deprecate dspace-l10n-check.py, can it be removed?
Does https://github.com/ufal/clarin-dspace/wiki/Language-localisation need updating?

[cyplas]

Does this deprecate dspace-l10n-check.py, can it be removed?

I would say so. But that sort of assumes that users everywhere prefer my script. :)

Does https://github.com/ufal/clarin-dspace/wiki/Language-localisation need updating?

It doesn't make anything on that page obsolete, but it does offer an extra (and I hope preferable) script for the last section: https://github.com/ufal/clarin-dspace/wiki/Language-localisation#checking-for-message-key-changes-when-upgrading. If you like, I can edit that section myself, but I'd ask you to test out my new script first yourself.

[kosarko]

I would say so. But that sort of assumes that users everywhere prefer my script. :)
@cyplas seems (https://github.com/ufal/clarin-dspace/blob/4e892cfe0b2d53525d671941a67afdcd7b6a48f8/utilities/project_helpers/scripts/dspace-l10n-check.py) that you are the only contributor to that file. Maybe it was grabbed from some other source; I'm not sure (maybe https://wiki.lyrasis.org/pages/viewpage.action?pageId=19006307). However, if it's not mentioned in the wiki and if non of the documented (in the wiki) scripts actually uses it, it should be save to remove.

Feel free to draft the wiki update here in the comments. I'll let you know if I got the desired output based on the draft. Does that sound ok?

[cyplas]

@cyplas seems (https://github.com/ufal/clarin-dspace/blob/4e892cfe0b2d53525d671941a67afdcd7b6a48f8/utilities/project_helpers/scripts/dspace-l10n-check.py) that you are the only contributor to that file. Maybe it was grabbed from some other source; I'm not sure (maybe https://wiki.lyrasis.org/pages/viewpage.action?pageId=19006307). However, if it's not mentioned in the wiki and if non of the documented (in the wiki) scripts actually uses it, it should be save to remove.

Right, I think you or Jozef pointed me to this script when I took this up. I added it and used it, but did not change it. So I agree and removed it.

Feel free to draft the wiki update here in the comments. I'll let you know if I got the desired output based on the draft. Does that sound ok?

Ok, I suggest this:

When upgrading your clarin-dspace repository to a new release, there may be new or changed message keys (usually XML, but potentially also JS) needing translation. You can check for missing translations with check-message-translations.py:

$ python utilities/project_helpers/scripts/check-message-translations.py de

For the XML messages, you can also automatically add new message keys based on the English ones with:

$ python utilities/project_helpers/scripts/update-message-xml-translations.py de

This will add any new keys from the English messages.xml files to the German messages_de.xml. The content will be copied from the English, and the <message> will be given a TODO attribute (TODO="translate"). Note that the script will also purge messages_de.xml of any XML comments and reorder the messages to match the order in the English files.

There is also a script which checks for missing and superfluous keys relative to what is referenced in the code (although this is not totally reliable since some keys are generated dynamically):

$ python utilities/project_helpers/scripts/check-message-usages.py de

[kosarko]

@cyplas can you have a look at my review comments?

[kosarko]

What do these lines do? Seems that you are looking for } in the tag name?

moreover

sources/clarin-dspace$ (gh-merge/1000)> python utilities/project_helpers/scripts/update-message-xml-translations.py sl

Constructing temporary joint xml /tmp/messages-en.xml from all English messages.xml:
  /mnt/c/Users/ko_ok/sources/clarin-dspace/dspace-xmlui/src/main/webapp/i18n/messages.xml
  /mnt/c/Users/ko_ok/sources/clarin-dspace/dspace-xmlui/src/main/resources/aspects/SwordClient/i18n/messages.xml
  /mnt/c/Users/ko_ok/sources/clarin-dspace/dspace-xmlui/src/main/resources/aspects/Discovery/i18n/messages.xml
  /mnt/c/Users/ko_ok/sources/clarin-dspace/dspace-xmlui/src/main/resources/aspects/XMLWorkflow/i18n/messages.xml
Traceback (most recent call last):
  File "utilities/project_helpers/scripts/update-message-xml-translations.py", line 42, in <module>
    element.tag = element.tag[element.tag.index('}')+1:]
ValueError: substring not found

But I must say python utilities/project_helpers/scripts/update-message-xml-translations.py sl finished on one of four runs, which is even more confusing

[cyplas]

Yeah, this is hacky, and, as your test runs indicate, not robust. Curly braces are used to wrap a namespace in tag names (e.g., "{http://apache.org/cocoon/i18n/2.1}message"). In constructing the temporary joint English messages.xml (which is in check_message_lib.py, already from before), I had to deal with the fact that for some reason, some of the English messages.xml files have a default namespace (xmlns attribute) and others don't. But it looks like the order in which I construct the joint xml isn't deterministic (oops, set instead of list!), and that the namespace in the joint xml is not guaranteed. If and when this joint xml doesn't have a default namespace, your ValueError must be triggered.

Ok, I can fix the joint xml construction, and maybe it's best to make sure it does not have a namespace (since messages_cs.xml and messages_sl.xml don't). And then these lines can and must be removed.

[kosarko]

I'll be a bit nitpicky here, but do we need lxml?

[cyplas]

Aha, probably not. I'll try without.