Refactor i18n

[tdreyno]

This is a proposal for refactoring how i18n works in Middleman v4.

First, we remove the magical localizable folder.

When i18n is activated, all html files in source are localized by default into as many languages as the user specifies. These files are output into their own folders (en, es, fr) as usual.

To be more specific, there are additional options:

Naming a file with the locale will target only that specific output folder. For example: about.fr.html.erb will only output to fr.

Adding localizable: false to the front matter will not move the file during build and will have the default locale set.

Adding a glob or a proc to the activation can mass skip localization:

activate :i18n do |i|
  i.skip = "/legal/*"
end

Questions:

• Do non-localized files output to the root or to every single locale folder, but they are the same?
• Does any of this seem natural to Rails users?
• Do non-html files need to be localized? Strings in JS?
• Does this help or hinder localizing blogs?

Tracking:

• Remove localizable folder
• Add localized front matter
• Add filter to extension to specify which files in source are localized.
• Make URL helpers (link_to, etc) aware of i18n

[pomartel]

👍

• Do non-localized files output to the root or to every single locale folder, but they are the same? I would only output to the root and not every locale folder. If you want the file to copy to all locale folders, you just have to let it as is without marking it as localized: false. It will then fallback to the default locale for all other locales.
• Does any of this seem natural to Rails users? Absolutely!
• Do non-html files need to be localized? Strings in JS? I would not go as far as localizing JS files. It's probably a better pattern to organize your JS to pull localized strings from html data attributes or server-side JSON.

[lukasnagl]

My following question may result from a lack of Ruby and or Rails knowledge, but is there an easy way to create different content structures for different languages? Basically dynamic content depending on the users language.

Something like:

if i18n.locale?(:russian)
   # insert russian specific content
end

If not, a template helper would be great, if you're about to refactor i18n.

[tdreyno]

@j4zz This should work currently. We'll add a better named current_locale helper as well.

if lang == :en

[lukasnagl]

Great, thanks!

[Aupajo]

👍 this sounds great. We have a large (~400 page) site, which needs to support several locales.

We ended up writing our own plugin to handle localisation. Our structure looks something like:

source/
    locales/
        en_US/
        en_GB/
        ...

(Files in source are global, files in locales are locale-specific.)

The biggest issue for us is that we want to have the ability to specify one page as being present in several (but not necessarily all) locales. A way to handle that would be nice. Perhaps, the ability to alternatively specify a list in the localizable attribute:

localizable: en,fr

Having have two alternative ways to localize a file (locale in extension or frontmatter) might be confusing, though.

Do non-localized files output to the root or to every single locale folder, but they are the same?

We deploy each locale as a self-contained directory, so the latter option suits us. I'm not sure that suits the majority of people, though. What about making this an option, or something that can be overridden in a plugin without too much effort?

Do non-html files need to be localized? Strings in JS?

I can't speak for anyone else, but this isn't an issue for us, at least. You could always append .erb to a JS file to access I18n through ERB if you needed to.

Really like the sound of this. Happy to help contribute here.

[davidwkeith]

FYI @Aupajo #848 is a request for a simular implementation of the same feature. With the proposed localizable: false frontmatter support it seems like a more reasonable request now.

---

Do non-localized files output to the root or to every single locale folder, but they are the same?

Outputting non-localized files to each locale is never what we would want. For the case when we need to output a file to multiple locales that it has not been translated for we use a fallback scheme. Basically setup a fallback map:

set :locale_map, { :en_us => nil, :en_ca => :en_us }

Then we override the translate helper to walk the chain looking for a valid string. (ie no "translation missing" when calling super)

Do non-html files need to be localized? Strings in JS?

Non-HTML files should be localized, we have a bit of hack going to localize some JSON strings files for an Ember.js app. Also, we sometimes build things other than websites, like a large number of email templates, where we need both .html and .txt output. Being able to configure what file extensions get localized from the activate call would be ideal.

[tdreyno]

I think localizable: en,fr is a great idea.

The reason for allowing both frontmatter and config.rb to configure which files are localized is one of scope. Changing an entire folder of files' frontmatter is annoying, but something like the following might be nice:

activate :i18n do |locales|
  locales.filter "about/*", :locales => [:en_us, :en_ca]
end

Now sure about the syntax or naming, but the idea seems nice.

@pulletsforever If we make the entire project localizable, then it would result in a copy of every JS file in every locale... which is probably not what you want. Maybe we have something like this:

activate :i18n do |locales|
  locales.files_glob = ["**/*.html"] # default
  locales.files_glob << "javascripts/i18n/*.json"
end

[davidwkeith]

@tdreyno yes, that looks good.

[Aupajo]

@tdreyno 👍 locales.filter would be handy.

[davidwkeith]

Another possible use case for non-HTML files is images and fonts.

Our product has a localized UI by country, so images are often localized by country. It would be nice to specify:

<%= image_tag "product.jpg" %>

And have it pick up product.fr.jpg if present when the locale is :fr. We have overridden the image_tag helper to do something simular in our project, which saves us a bunch of conditionals in the templates.

The image_tag helper seems like a good fit for this stated purpose as it matches the syntax well.

---

In the case of background-images we use a locale wrapper in SASS that picks up the lang attribute on the HTML element, i.e.:

@mixin locale($locale) {
  html[lang=#{$locale}] & {
    @content;
  }
}

A SASS mixin was written that automatically wrapped declarations when the appropriate file names were found would also be a good fit. It could even be extended to fonts, where you would likely want a different font for non-Latin based languages.

[davidwkeith]

Just brought to my attention: <video> & <sound>:

Again the *_tag helper should be overridden to look for localized videos/sounds or whatever it is looking for. (Not sure how many other media types there are, can't imagine many.)

[sandstrom]

Answers to the questions as well as my own thoughts (further below):

Do non-localized files output to the root or to every single locale folder, but they are the same?
To every single folder (and in the case of mount_at_root they would also go into the root).

Does any of this seem natural to Rails users?
Yes, it seems natural.

Do non-html files need to be localized? Strings in JS?
Probably, but I think that can be handled manually, perhaps adding a line or two in the docs. We do this in our translations.js.erb:

var translations = {
  <% I18n.available_locales.each do |locale| %>
    <% I18n.with_locale(locale) do %>
      "<%= locale.to_s %>": <%= I18n.t('root_key_name').to_json %>,
    <% end %>
  <% end %>
};

Additional thoughts:

1. It would be neat if one could set the load_path via the I18n extension. We currently do this manually:

# Use a nested structure for locale files, so we can keep multiple files per locale
I18n.load_path = Dir[Middleman::Application.root_path.join('locales', '**', '*.{yml}')]

2. Our proxy pages don't get the correct locale even though they have a locale specific in their filename (which we solved by setting it again inside the block, shown below). Though this could be a misunderstanding on our part too.

proxy "/de/foo/index.html", "/foo/index.de.html", :ignore => true do
  I18n.locale = :de
end

3. We couldn't get links to be locale_aware. Again, this can us doing it plain wrong. Anyway, here is how we solved it:

def i18n_link_to(*args, &block)
  url_arg_index = block_given? ? 0 : 1
  if url = args[url_arg_index]
    new_url = "/#{I18n.locale}#{url}"
    args[url_arg_index] = new_url
  end
  link_to(*args, &block)
end

4. A current_locale helper would be nice, with output equivalent to I18n.locale. Again, simple to setup using a custom helper (we did), but still neat.

5. Access to the value of @mount_at_root, e.g. a root_locale helper/property on application would be nice, when one needs to extend upon the capabilities of i18n.

[paulcpederson]

In terms of using translated strings in Javascript, that's something I've done a lot of recently. A lot like @sandstrom I ended up just making a file called translations.json.erb in the localizable folder that has one line of code in it:

<%= t("javascript").to_json %>

And in en.yml I have a javascript key that contains all the strings in javascript that need to be translated.

[joallard]

Something else that just came up is referring to the pages in a language-agnostic way. For example, it's currently hard for me to ask Middleman for "the french version of page how-it-works".

I've been prying and digging for a way to do this simply, but I'm not finding. There is simply no map of which pages got translated into what.

Let's come back to my example of 'how-it-works'. I have page 'how-it-works.html.haml', localized for both fr+en. I'm making a nav bar, and I want to say "go to your version of how-it-works". Now my English pages are unprefixed ('/how-it-works') and French are ('/fr/fonctionnement'). Without a helper, I will have to test for english to see whether I should add a prefix, and then go through the i18n file for paths. And I have to do this for every occurence in my views.

Linking to stuff right now is a bit hard.

---

Edit: I just saw @sandstrom's comment now, which talks about links. Sorry about that.

---

Related concern: the lang switch link. Right now, there's no obvious way to say "link to this same page, but in that other language." Again, a locale mapping would solve that.

[tdreyno]

@joallard Very good idea and should be simple to add that metadata to each localized resource.

[Aupajo]

👍 @joallard Yeah. Someone ran into this issue on the forums a little while back. I gave him a solution, but it relied on making assumptions on how URLs are formed. http://forum.middlemanapp.com/t/i18n-list-of-language-siblings-and-links-to-them/978/2

[lolmaus]

I've tried leveraging the current implementation of i18n and was discouraged with the fact that it's implemented very differently for basic Middleman vs Middleman Blog.

When i thought about it i came to a conclusion that this can't be considered a fault of the i18n module. The very fact of distinction between Middleman and Middleman Blog brings a lot of trouble.

My question is: have you considered integrating Blog into basic Middleman and introducing a unified implementation for both basic pages and blog articles?

[bhollis]

What specific functionality do you think should be brought over to Middleman from middleman-blog? What specifically do you see as differences between the two? I don't use the i18n functionality, so beyond fixing bugs in it, I don't know much.

[joallard]

Without the troubles of bringing blog and core together, I think that just uniformizing the two with the same config parameters would be nice. I have yet to try out blog in multiple languages, but that's what I'm getting from Andrey's comment.

@lolmaus, would you bring us up to speed with (a) the differences between configuring i18n in core vs. blog, and (b) what kind of syntax/config you'd ideally expect with M-blog?

[lolmaus]

Hi! Sorry for not replying in time.

i18n-related inconsistencies between middleman and middleman-blog

1. middleman-blog lacks the mount_at_root option;
2. middleman lacks the ability to set content language by page source path;
3. middleman lacks the ability to set content language by frontmatter;
4. middleman requires content to be placed under /source/localized/, middleman-blog breaks if content is placed there.
5. middleman-blog lists content in different languages together when using pagination.
6. middleman-blog does not allow to set default language per blog in configuration.

The story of my i18n woes

I decided to build a language switcher partial for my Middleman website. For every page, the switcher should point to the same page in other languages, not just to the title page. If the page is not available, it should produce a greyed out language item.

I like the mount_at_root option and i wanted all URLs on the website to follow a single scheme, so i had to configure separate blogs for each language, so that i could mimic mount_at_root by manually configuring blog.permalink. On the good side, this saved me from middleman-blog mixing content in different languages into a single listing when using pagination.

To create the paginator, i needed to be able to do two things:

• retrieve current language prefix (which is an empty string for the default language);
• probe whether the given page is available in a different language.

The only way to make it happen that i managed to find was to work with pages' source paths. I had no option but to make an assumption that all URLs on my site start with a two-character language prefix and a slash (except for the default language). So to generate links to other pages and checking that they exist i remove first three characters from the current page's source path (except for the default language) and attach the prefix of a different language.

This works, but it enforces a terrible directory scheme. Static pages go to /source/localizable/, pages in different languages are adjacent and differ by a suffix in filename. Blog pages go to /source/blogname/ for the default language and /source/??/blogname/ for every other language, pages in different languages are separated and are forced to duplicate language definition in their frontmatter.

Why i think middleman and middleman-blog should be merged

The matter is that middleman-blog is not just for blogs. It fills in the functionality that lacks in default middleman implementation.

Here's what middleman-blog does for me:

• Allows defining different content types. For smaller ones i use YAML data but larger content types middleman-blog is a life-saver.

  Imagine that i'm building a static prototype for a company website. It has the following content types:

  • services;
  • goods;
  • retail shops;
  • news;
  • portfolio;
  • employees.

• Provides a very convenient way to configure per-content type settings.

• Allows for tokens in pages' source paths, that are made available (thanks, @bhollis!) for use in templates and helpers.

• Tags.

• Summaries.

• Listings (no need to waste time programming custom filters for sitemap.resources.select, yay!).

• Pagination.

• Unpublishing pages (without moving their files or messing with their filenames).

• Per-page assets.

• Command line content stub generator.

Neither of those features is limited to the "blog" concept. All of them are universal for "content", and all websites are about content. If i were to decide, i would rename middleman-blog to middleman-content.

When you master using middleman-blog, it becomes almost pointless to use the vanilla middleman pages functionality. Basically, i still use vanilla pages only because middleman-blog forces me to manually provide dates for each page.

That's why i suggest for middleman maintainers to consider merging middleman-blog into middleman 4.0, producing a revamped API.

[rriemann]

Is there currently been worked on? Nice to see some progress in the meanwhile. 👍

[tdreyno]

v4.0 is now being worked on. This feature hasn't started being refactored yet, but hopefully it will within a month or so as we finish a lot of housecleaning in the core codebase.

[rensverschuren]

Completely agree with @lolmaus about merging middleman-blog into middleman. My thoughts on this:

• Actually there a 2 types of content:
  • Content structured as YAML (in /data)
  • Content structured as Markdown files (now middleman-blog). In my opinion this is the same as the content in /data. The only difference is some publishing/blog functionality and the ‘body’ field as Markdown.

I think it would be more logical to move the .md blog posts out of the /source folder and put them in a ‘unified’ /content folder, together with the YAML files. This way there is a better separation between content and presentation.

I would also like suggest a more unified approach on multi-language sites. For example:

• Fully localized paths for blog articles, for example:
  • /en/journal/2014-07-14-title
  • /nl/blog/2014-07-14-title
• Localized pagination is not possible without using the proxy. Is seems that pagination is also not available in custom_collections templates.
• Is it possible to make the proxy language aware? So you don’t have to call proxy for each language?

These are my thoughts on (localized) content. Please check if they would fit in technically/strategically.

Keep up the great work!

[tdreyno]

@rensverschuren I'm not really following all that. Would you be able to make an example repo with the file structure and optimal config.rb you're suggesting?