i18n, proposed tooling and workflow

[morficus]

This issue is intended to act as a catalyst for getting the Internationalization ball rolling, discussions going and decisions made.
Below are a few things I'm proposing is based on a mix of my own research and previous conversations in the #i18n Slack channel.
I obviously do not have all the answers (and there is a chance that the ones I have could be improved on) so please fill in what ever gaps you see or propose changes where ever you see fit.

Glossary of terms

• i18n: internationalization
• BLOS: big list of strings
• admin: the admin section that lives under the /ghost url
• frontend: non-admin panel section, where articles are published and publicly visible

Proposed Tooling

Transifex, for managing translations:

• Transifex has a pretty comprehensive API (http://docs.transifex.com/api/) for interacting with their resources as well as a few grunt plugins in case we need to update the dev workflow (like this, this, and this)

• It also has a mechanism for keeping the source language file in-sync with a file on GitHub , which would make it really easy to keep things updated (full details).

  As long as your source file is accessible from a public URL, such as this raw file on GitHub, Transifex will periodically check the URL for updates. You can also use a file from a private repo in Github as long as the URL has the respective token that Github provides. If there are updates in the file, Transifex automatically fetches the file and updates your content.

ember-intl, for client/Ember translations

• Part of the Format.js library collection
• I compared it to ember-i18n and ember-cli-i18n, Format.js provides a more complete feature set
• It does not officially support sideloading BLOS in the current, but it is implemented in the 2.0 branch (see issue ref)
• It goes beyond just strings, but also includes template helpers for formatting dates, numbers and time per locale
• Includes template helpers for providing custom formatting for dates, numbers and time
• Follows well-known ICU message syntax for pluralization, select and variable interpolation
• Leverages the standards-compliant window.Intl object and provides a polyfill for browsers that do not have support for it as yet.

intl-messageformat, for server-side translations

• I also looked at messageformat.js and i18next-node but they seemed for fall short in some aspects.
• Also part of the Format.js library collection
• Biggest sticking point is that it follows ICU message formats and Unicode CLDR standards
• Leverages native ECMAScript i18n API functionality for date and currency formatting (polyfill if no available in engine)
• Usage is kind of clunky, but that can be hidden behind a facade for a better developer experience

Proposed Rollout

We should be doing a staged-rollout of the i18n functionality. The ideal end-state is for Ghost to have the concept of "language packs" that the user can install on demand when ever they want. But this long-term vision is some-what dependent on (or at they ver least related to) Ghost Apps being flushed out. The phases detailed out below are intended to lead up to this ideal state.

Phase 1 - harvesting strings

Regardless of what process or workflow we end up following to distribute translations, all the hard coded strings (both in the Ember and Node app) will need to be moved in to a "source language file". So phase 1 would be to do a basic implementation of the i18n libs mentioned above. Ember sample, Node sample
At the end of phase 1 English would still be the only language supported in Ghost and there would be no noticeable change to the end-user.

Phase 2 - start translating strings

(this can be done in parallel with Phase 1 once there is a decent amount of strings are in Transifex)
Once we have our base "source language file" (most likely en.json), the next step would be to setup Transifex and let the translators commence the translating in to the initial target languages.
This process will be on-going as the "source langue file" will continue to be updated and new languages may be introduced as well.

Phase 3 - develop locale-setting functionality

Both the Node and Ember app need enhancements to (a) ask the user for their locale during setup, (b) modify locale in "settings" and (c) allow for the locale value to be programmatically retrieved. This might go through a few iterations depending on how we want to allow users to control locales (blog-wide? per-user basis? only during setup?). At this point, i18n could be enabled under Labs with limited locale support (depending on progress of phase 2).
Locale files will still need to be bundled with the release (not a big deal YET, because there would be 2 or 3 locales at most)

Proposed Development Process / Workflow

The source language file (en.json for both client and server) is the only language file that will be in the repo. This file will be kept in-sync with Transfex using this process. All other language files will be downloaded as needed.
Long-term, the only language file that should be in the downloadable ghost-*.zip file should also be en.json. But short-term, until apps are ready, any other supported locale will need to be included as well (hopefully this will only be necessary for 1 or 2 locales).

Proposed Workflows and organizational structure

See the "Workflows and organizational structure" section in this gist for information related to this topic.

[JohnONolan]

This sounds really good.

[jamesarosen]

I'm not a Ghost contributor, but I've worked on a few i18n/l10n projects. (I maintain ember-i18n, though I think ember-intl is a great choice for this flow.) I suggest two very small additions to this flow:

1. At the end of phase 1, build a glossary. There are going to be a handful of really core terms (like "post," "user," "editor," and more) that have nuanced meaning. You'll get better, more consistent translations if you compile this list and write a sentence or two about each one.
2. As you start each localization project, translate the glossary first. Make sure that translators use the established core translations when adding new strings.

[alexbagirov]

Why can't we use localize npm module? It seems to be easy to implement.

[ErisDS]

@jamesarosen Thanks for the input, really appreciate that idea and it dispels a key concern I had about translations - keeping the meaning of key words true to the original.

@alexbagirov if you'd like to suggest an alternative approach that would be very welcome, however you would need to explain what your suggestion is and why it is better than what is in the extensive OP.

It seems to me the approach outlined by @morficus covers all bases, all the way from how to implement it to how to manage the process once it's up and running. The next steps from my perspective would be to get the implementation steps broken down into issues that people can work on - I understand @morficus started looking at string harvesting, but I would not recommend working away on this without having a set of issues and checking in progress - it's just far too big for one person to do alone IMO.

I'm really keen to get this split into issues if we can, and I realise that converting this sort of thing into individual issues isn't something a lot of people have experience with, so I'm very happy to help out figure the right way to break it down etc if needs be.

[JohnONolan]

Some very relevant reference material: http://responsivenews.co.uk/post/123104512468/13-tips-for-making-responsive-web-design

[alexbagirov]

I would like to suggest LingoHub as a translation tool. It suits more not just because it is free for open source projects. It also works like CI becuase it can push translations and all changes right to the GitHub. In my opinion, we should look more into it instead of Transifex.

[morficus]

@alexbagirov Transifex is also free for open source projects. As for LingoHub having the ability to push translations IN TO the repo, I'm not sure there is a need for that because the goal is to offer translations as "language packs" vs having all languages be part of the core project.
With that being said, I will give LingoHub a run with it's 30 day trial to see how it stacks up.

[alexbagirov]

@morficus, you don't have to push it to core repository.

[RubenOlsen]

I would like to partake in the i18n work AND do the Norwegian translation (full no_NO, partly nn_NO but not se_NO as it's out of my competency level).

I agree on keeping the translations out of the core, and use packs instead. I propose maybe having one GH repo per language-pack.

It is also important that we use tools that facilitates anyone (i.e. non-technical people) to help out with translations. Learning non-technical people to use git is complex at is is - learning non-technical people to use git with pull/push is just painful. A setup where translations can be pulled on request into language-pack/-repo would be really nice.

[weipah]

I found two resources which could be usefull:

• Handlebars helper for i18n
• Strategy for i18n in node.js projects

I'm fairly new to node.js and Ghost, but here are my two cents as well:
I think, timezones/shifts/conversions should be handled programatically on the backend (node.js) and therefore be completely transparent for the templates for example.
Template translations could be easily handled with handlebars, but a new handler is needed here I guess.
It could maybe help to look into other projects with a working i18n ecosystem like WordPress or OwnCloud which is also hosted here on GitHub.
Last but not least I'd vote for Transifex as well, because it allows for crowd-based translations and a lot of resources are already available.
I'm not such a fan of the mo-files strategy like WP does, because generating and extending the files is not so easy (reading: you need some technical experience and not only a person who can translate some texts).

As I see @morficus already started with the process which is great!

[djensen47]

One thing I would like to see is support for non-standard locales especially for pseudolocalization. Another use case could be modifying the helper text in the UI. An enterprise might want to deploy Ghost internally and use slightly different instructions/terms.

[Yannovitch]

Is this the thread where I can suggest that not only should our Ghost blog be localized, but it should be able to support multilanguage blog as well ?
What I would like to see is an option where I can write some content in one language (for example my mothertongue, french), and have a little flag on the blog post which link to the same blog post in an other language ( for example, english, as it's a must-have if you want to be read globally).
The same should apply for the whole blog, with a prefix in the URL ( blog.com/fr/ and blog.com/en/ )
Y

[Yannovitch]

And how do you plan to support the URL, actually ?
With a prefix added to the content ? like blog.com/fr/content , blog.com/en/content ?
Or directly with the translated URL ? like blog.com/contenu for the french version and blog.com/content for the english one ?

[ErisDS]

Hi @Yannovitch this issue is only to do with translating the admin panel. You can vote on or suggest other features on our wishlist.

[ErisDS]

The server side strings have all been harvested, but I'm left wondering if that's the right approach?

Should API errors really be translated?
Also, there's a bunch of server console output we spit out during migrations - should this really be translated?

Just wondering where the line is - is it really normal for output from things like APIs to be translated?

There was a discussion in #6075 which talks about using an 'Accepts Language' header when making requests to the API, so the response can suitably return the correct language. This makes sense to me, but makes me think we need to be much clearer on the server side as to what constitutes: 1) server side messaging (stuff to do with rendering blogs) 2) the API and 3) console/developer output?

cc @kevinkucharczyk @morficus @kevinansfield @AileenCGN and anyone else interested in i18n.

[kevinkucharczyk]

I've been thinking about the same thing, @ErisDS, and we do need to work on a solid plan.

I think all console output should remain english. Otherwise debugging errors returned to the console would become crazy hard and we would not be able to easily help out Ghost users - we'd always have to search for the right translation.

Anything appearing in the UI should probably get translated. I guess this would include some API errors, where a message could be displayed directly in the admin panel. @kevinansfield raised a question on this (at #6075): should the server return the translated strings or simply keys and let the client find the translation?

There's a lot of thinking to be done here. It would be nice to have a solid plan and clear path for this one.

[ErisDS]

So having thought about this a bit more, I think the rules should be as follows:

• All UI messages should be translated.
• All console messages should not be translated - this should be though of as debug info or logging, and should always be in English.
• The API messages/errors should get translated, only if the 'accepts language' header or similar is provided.

The last rule means it's up to the client to tell the API what language it should serve, this might seem like unnecessary overhead, but I think it makes sense. E.g. a user selecting to see their admin panel in, say, German, doesn't necessarily want their theme to appear in German. If they make API calls from the theme, any messages there being translated wouldn't be correct.

This means all the migration output should not be translated. In future, we may wish to provide an interface that shows this process, at which time it would need to be translated, however for now I think it should all be treated as debug info and kept to English for easy debugging purposes.

[halfdan]

E.g. a user selecting to see their admin panel in, say, German, doesn't necessarily want their theme to appear in German.

Yes! Thank you, that is absolutely correct.

[ErisDS]

Looking further at what's on the server side, there's a lot of places where there's an error thrown, and the message is translated:

In this case, we have very little idea where that error will end up being shown. It should almost always end up on the server console, but it may also end up in an API response, or a UI somewhere.

This leads me to think we need to work on standardising how errors are done on the server side, before we can really do translations 100% correctly. This issue talks about how errors should be standardised for the API: #6050

If all errors are defined in a single file (nothing to do with translations) along with a code which identifies that error, then we can be sure we pass around the English version until such time as the error hits somewhere it should be translated, and handle translating at that point. E.g. it hits the API and there's a header asking for a translation.

We probably need a separate issue to spec out how this should be done - I've put writing it on my todo list.

[Bugs5382]

So new to the party here after being not he sidelines for a while. Very impressed by what I have read and the code. While I can't commit to this project on a full-time basis, I feel that I am going to take a stab at this and see what I can do.

[juan-g]

ErisDS commented on Oct 12, 2015:

Hi @Yannovitch this issue is only to do with translating the admin panel. You can vote on or suggest other features on our wishlist.

Does this mean that editing the theme is enough to translate the publicly visible frontend? If that's the case, that's really great and we can use Ghost. For example, for a current project the admin side would be fine in English, but the frontend is needed in Spanish.

Please, are the English language strings for site visitors (frontend) in the theme, or in an editable configuration file (needed for months in dates, etc.), or hardcoded in core?

If there is a language configuration file for the frontend, we would be happy to contribute Spanish string translations to Ghost.

[juan-g]

I've just seen there is already a Transifex project to translate Ghost to several languages at https://www.transifex.com/ghost/ghost/ and I've applied to help with Spanish.

Waiting for a reply, but that project seems stalled for now, with 0% translated. However, it would be really easy to do it quickly today, it's just 63 strings (367 words) to translate.

[juan-g]

We are now discussing some related tasks at the i18n channel of Ghost Slack.

[kevinansfield]

I've just seen there is already a Transifex project to translate Ghost to several languages at https://www.transifex.com/ghost/ghost/ and I've applied to help with Spanish.

To be clear for anyone else finding this issue, the referenced Transifex project is not for translating Ghost - it's an unused project that contains strings for Ghost.org.

[juan-g]

Sorry if that's so. Although the description for the ghost translation project of course contains a link to the Ghost's site, it says: "Just a blogging platform", and "You can help translate "Ghost" by requesting to join the project's team." Hence the confusion.

There is another Transifex translation project ghost-org, with the descriptions "The website for the Ghost publishing platform", and "You can help translate "Ghost.org" by requesting to join the project's team."

[juan-g]

Just submitted two complementary pull requests that work together, for Ghost and the default Casper theme, to make Ghost's frontend for site visitors translatable:

[i18n] Translatable Ghost's frontend: Themes, dates, pagination, subscribe forms
#8437

[i18n] Translatable Casper default theme
TryGhost/Casper#306

[ErisDS]

I'm going to close this issue in favour of #8437, which is in active development with support from the Ghost team.

In order to pursue what's outlined in this particular issue, we'd first need to reopen and address the problems described in #6526.

[heyakyra]

I wonder if it would make sense to use Project Fluent in the future. It's a next-gen translation format by Mozilla which improves over GNU gettext and ICU MessageFormat and is supported by i18next-fluent, which is a standard way to implement this in JavaScript. This would allow using a slick open source translation tool like Pontoon or using a common commercial tool like Locize to outsource translations.