i18n: Add compatibility for Transifex (industry standard provider for open source translations)

[gcushen]

Problem

Popular themes for Hugo, such as Academic, offer language packs to enable users to easily change the interface language to their native language.

Currently, the format used for Hugo's YAML language packs is incompatible with leading open source translation providers, such as Transifex.

Thus, Hugo language packs are currently very difficult to manage (Academic has over 18 languages), difficult to keep in sync with the master language file, and without the ability to perform translations via the key open source translation providers.

Proposal

Restructure the YAML language file into a standardized format in order to align it with the standardized form used by other major open source projects (e.g. django-rest-framework).

This would make Hugo and its themes compatible with existing language pack translation management tool such as Transifex, the industry standard provider of language translations for open source projects.

[bep]

"Transifex is a proprietary, web-based translation platform;"

https://en.wikipedia.org/wiki/Transifex

I see very little "open source" in that project/product?

[gcushen]

It's the leading provider of free translations for major open source projects.

[bep]

major open source projects.

Name a few.

[gcushen]

I named one major one, I know many other major OS projects using it, will list them in a bit :)

[gcushen]

Here's a comprehensive list (43 million references to Transifex in GitHub code :D ): https://github.com/search?q=transifex&type=Code

[gcushen]

To summarize, this includes edX, Kodi, and pretty much all the other big open source projects that offer some kind of text interface.

[bep]

OK, thanks.

I have thought about this in the past -- and here is my thought about how we could add support for other formats:

• Add adapter that converts (internally in Hugo) to go-i18n's data format.
• It should be possible to detect what adapter to use either by filename or by looking at the content
• This way different formats can live side-by-side

A nice addition to the above (but harder to implement) is a converter the other way.

[gcushen]

The v2 beta of go-i18n has just been released and supports a new flatter YAML language file format which at first glance appears to offer compatibility with translation providers like Transifex - see nicksnyder/go-i18n#85

Perhaps we could consider just moving to this significantly more popular format which is now supported by go-i18n. My other recent proposal (#5342) would help prevent breaking user's sites.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. The resources of the Hugo team are limited, and so we are asking for your help.
If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.
If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.
This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

[eighthave]

Ideally, Hugo would just use one of the plethora of existing, standard, full-featured formats that are supported by lots of translation platforms. For example, Weblate is actually free software and many software packages are translated on it. Here are the formats it supports: https://docs.weblate.org/en/latest/formats.html

Using something like Chrome/WebExtensions JSON seems like a good bet:

• https://docs.weblate.org/en/latest/formats.html#webextension-json
• https://developer.chrome.com/extensions/i18n
• https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Internationalization

[eighthave]

I just looked at go-i18n, it supports JSON/YAML formats that have has some decent support (basically, key/value pairs with one or two levels of nesting):

weblate:
  hello: ""
  orangutan": ""
  try": ""
  thanks": ""

[eighthave]

Mozilla is switching Firefox and many other of its projects to its Fluent 1.0 format, which extends on what gettext and XLIFF offer. They already have parsers in Javascript, Python, and Rust, so writing a Go parser shouldn't be too hard. They are pushing this format for websites also, so it could make sense for Hugo also. I haven't used it myself though.

https://hacks.mozilla.org/2019/04/fluent-1-0-a-localization-system-for-natural-sounding-translations/