JSON format breaks i18next files due to automatic restructuring [$120 awarded]

[wichert]

I just had an interesting experience: I configured Weblate to manage translations for an Aurelia project, which uses i18next to handle translations. This worked wonderfully well, until I started testing with updated translations and found that half the translations had stopped working.

The problem turned out to be due to handling JSON structure: Weblate (via Translate Toolkit I guess?) always uses a flat structure for the data. So where my translation.json used to look like this:

{
    "navbar": {
        "reward": "Rewards",
        "about": "About"
    }
}

it got transformed into this:

{
    "navbar.reward": "Rewards",
    "navbar.about": "About"
}

This breaks the translation lookup.

[nijel]

Indeed these files are not properly supported by translate-toolkit so far. There already is issue #650 on same topic, but given that the file mentioned there no longer exists, I will close it in favour of this issue.

[wichert]

Is there some way I can help to improve support for that format?

[nijel]

Improving translate-toolkit support. Currently JsonFile._extract_translatables is able to load these, but the JsonFile.str method used for serialization does not recognize these specially formatted keys. So saves them as they are without parsing.

[phw]

Just for completeness: The old ticket #650 also included another issue, since Chrome web store localization files include fields that are not supposed to be translated (the description which is a description of the translation itself similar to the comments found in some other formats). Not sure how to handle this special case in translate toolkit, though. I think that would be a special file format based on the existing JSON implementation.

[nijel]

Indeed, these are different problems, just happen to be based on same file format.

[nazar-pc]

I have a bit similar use case.
Sometimes translations are in form of the tree:
https://github.com/nazar-pc/CleverStyle-CMS/blob/master/components/modules/Shop/languages/English.json

Each key of parent structure is used as prefix for inner structures,
for instance:

"shop_" : {
  "type_" : {
    "int_set" : "Integer set",

effectively results in key "shop_type_int_set" : "Integer set".
So it is similar to what is described in this issue, but actually without any additional separator in form of dot.

[saily]

@nijel any news on this? Do you need some help?

[saily]

@nijel i just debugged translate-toolkit, it seems it can handle nested json files.

[nijel]

It can parse them, but it will write them differently (see first comment in this issue).

See https://github.com/translate/translate/blob/master/translate/storage/jsonl10n.py#L157

[saily]

Ok, after a lot of discussions i stopped working on translate-toolkit and wrote a small pre-commit hook python script which converts this back. @nijel do you like to ship this within your examples?

[nijel]

Shipping that with Weblate is certainly okay.

[saily]

here's a gist how to fix this with a pre-commit hook: https://gist.github.com/saily/f8a9f72c6b064d51f09040d66e981240

[nijel]

Thanks, I've included the above script in examples.

[wichert]

@nijel what kind of bounty would you want to have to get this added natively to weblate (either through translate-toolkit or directly - as a user I don't care how it is implemented)?

[saily]

@wichert we could just ship the script with weblate by default.

[wichert]

That does not work, since the pre-commit hook is global and would break badly if you also use other formats. You would need a flag per repository that indicates if that hook needs to be run, so more integration is necessary as far as I can see.

[nijel]

I think that adding support for this to translate-toolkit (and then Weblate) should be quite easy, I just haven't found time to do that (there are always more important things to do). Having similar bounty as on the WebExtension JSON format (#1380) would probably make me look into both of them rather sooner than later though ;-).

[wichert]

@nijel I added a $120 bounty.

[wichert]

@denim2x Did you really start working on this? I noticed you claimed the bounty with a completion date of last Monday, but I don't see any activity on your github account indicating you did anything.

[nijel]

Unless there will be some progress here, I can look at this last week in June.

[nijel]

Support for translate-toolkit has been submitted for review: translate/translate#3662