Extended language codes (#1371)

* Do not restrict language code in URL to 2 characters

* Allow dash in URL, such as pt-br

* Re-order help text in Add Language dialog

* Replace attribute 'language' with 'name'

Use ‘language’ for base language code

* Consistent naming: LanguageCode

* Reload after language change: fix situation when no language is shown in the url

* Use new language data

* Update templates

* Use "add/remove" instead of "activate/deactivate"

* Add some more languages

* Fix territory/script naming arabic and chinese

* Identify main language when it has a script; refactor; more languages

* Consistent naming

* Refactor set_language

* Let filter is_rtl use language data

* Refactor template

* Some tests

* Some more languages

* Make POT buttons work

* Add script for japanese

* Show disabled state in table

* Improve long dialog behavior

* Update documentation

* On first time use ensure a fallback language

* On first time use set the default language

* admin: read _html_head.tpl from other modules too

* Fix language switch template

* Style language switch template

* Change language codes to atom when used in code; refactoring

* Let templates use translation model

* Update doc, spec

* Move toggle code to main module so that it can be called from other modules

* Fix toggling rewrite_url with immediate feedback on page

* In templates sort languages by name

* Resolve conflicts from merge

* Consistent naming

* Rename languages.erl to z_language

Move z_trans functions to z_language
Rename model keys
Change tests to rebar tests

* Handle homepage with language switch

* Make language data more future proof

Keep language data private
Provide public data through properties/1 function
Make z_language single point of language functions (default_language,
is_valid, to_language_atom, fallback_language)
Nest sub-languages into main languages for easier listings

* Split z_language tests

* Update templates

Add help text, move styles out, reflect model changes

* Update status page layout

* Move styling to module less file

* Only append slash when rewrite_url is true

* Update right-to-left documentation

* Fix r_language variable

* Show English names for languages in admin

* Fix conversion to atom

* Add language filter

* add autofocus to filter

* More languages

* Fix edocs build failure

* Merge from master

* Fix upgrade script

* Complete renaming territory to region

* Fix tests

* Update po files

* Fix a typo from the conflict resolution

* Reverted rebar.lock

* Layout fixes to language menu

* Missed updated function call

doc/developer-guide/translation.rst

@@ -30,13 +30,13 @@ Before you can use multiple languages, you first need to enable
 :ref:`mod_translation`.
 
 Next, you need to tell Zotonic which languages you are going to use.
-You can do this in the /admin, following the `Translation` menu item
-in the `Structure` submenu.
+You can do this in admin: Structure > Translation.
 
-On this page you can add and remove languages, enable/disable
-languages and more. Note that a language is always identified by its
-two letter `ISO639-1 language code
-<http://nl.wikipedia.org/wiki/Lijst_van_ISO_639-1-codes>`_.
+On this page you can add and remove languages, enable/disable languages and more.
+
+The selected language can optionally get displayed in the URL of the page - useful if language versions of the page should be accessible as permanent URLs, for instance for search engine indexing.
+
+The language code in the URL is either a 2-letter code (e.g. ``en``), a 2-2 language-territory combination (e.g. ``en-gb``), a 2-4 language-script combination (e.g. ``zh-hant``), or a language-territory-territory combination (e.g. ``zh-hant-hk``).
 
 
 Translation sources
@@ -60,10 +60,18 @@ Texts in ``erl`` files use the ``?__()`` syntax::
 
     ?__("Translate me", Context)
 
-and using string concatenation::
+and with binary strings::
+
+    ?__(<<"Translate me">>, Context)
+
+using string concatenation::
 
     [?__("Edit:", Context), " " ++ Title]
 
+and with binary strings::
+
+    ?__(<<"Edit:"/binary, " ", Title/binary>>, Context)
+
 
 Generation of translations
 --------------------------
@@ -79,28 +87,22 @@ static translations live inside the ``translations/`` subdirectory of
 a module (remember: a Zotonic site is just a module!).
 
 In the ``translations/`` directory of the modules you can find the ``.po``
-files containing the translations. They are marked with the their two
-letter language code.  (Optionally you can name your file like:
+files containing the translations. They are marked with the their language code.  (Optionally you can name your file like:
 nl.foobar.po as Zotonic will only look at the part till the first '.'
 for the language code)::
 
-  translations/
-  ├── nl.po
-  ├── template
-  │   └── mod_foo.pot
-  └── tr.po
+    mod_foo
+    └── translations/
+    ├── nl.po
+    ├── template
+    │   └── mod_foo.pot
+    └── tr.po
+    └── zh-hant.po
 
-Here you see that this module, ``mod_foo``, has been translated into
-Dutch (`nl`) and Turkish (`tr`).
+This shows that module ``mod_foo`` has been translated into
+Dutch (`nl`), Turkish (`tr`) and Chinese traditional script (`zh-hant`).
 
-The translations/ directory contains a subdirectory which contains
-``mod_foo.pot``, which is the translation `template`, on which new
-translations will be based.
-
-The basis for these files (the translation `template`), is the ``.pot``
-file which is located in the ``template/`` subdirectory of the translations
-directory. This ``.pot`` file is regenerated when you click on the ‘Generate
-.pot files’ button on the Translation page in the admin. Alternatively,
+The template for translation strings is located in a ``.pot`` file, in ``mod_foo/translations/template/mod_foo.pot``. This pot file is regenerated when you click on the ‘Generate .pot files’ button on the Translation page in the admin. Alternatively,
 from your Zotonic shell:
 
 .. code-block:: erlang
@@ -113,10 +115,10 @@ strings. These strings are then added to the .pot file.
 Creating a new translation for a module
 .......................................
 
-First, add a language in the admin with the 2-letter code for that language.
+First, add a language in the admin with the language code for that language. See the Translation page (or the code in ``src/i18n/languages.erl``) for possible languages.
 
 Say, we're adding Polish, ``pl``. Now copy the ``.pot`` template file
-to the 2-letter code ``.po`` file::
+to the language code ``.po`` file::
 
   $ cd modules/mod_foo
   $ cp translations/template/mod_foo.pot translations/pl.po
@@ -131,9 +133,7 @@ When templates change, often the translatable strings change: more
 strings are added, strings are changed, or removed. When this happens,
 the translation files need to be kept in sync.
 
-The `Translation status` page in the admin gives an overview, per
-module / language combination, of the amount of strings that are
-translated for each language.
+In admin: Structure > Translation > Translation Status you can find the overview of the amount of strings that are translated for each language.
 
 After pressing the `Generate .pot files` button in the translation
 admin the ``.pot`` files are updated, but the existing ``.po`` files
@@ -165,6 +165,9 @@ To remove duplicates (and make a backup first), use::
 
     $ cat nl.po > nl~.po && msguniq nl.po -o nl.po
 
+To do this for all files, without backup::
+
+    $ find . -name "*.po" -print0 | xargs -0 -I file msguniq file -o file
 
 
 Translated content
@@ -190,7 +193,7 @@ is stemmed according to the site’s configured default language.
 All translations are added to the same full text index. This combined text is
 stemmed using a single stemmer. The selected stemmer depends on the configured
 default language (config key ``i18n.language``). The stemmer can be overruled by
-setting the config key ``i18n.language_stemmer`` to the two letter iso code of
+setting the config key ``i18n.language_stemmer`` to the two letter language code of
 the language matching with the stemmer. You have to make sure that the stemmer
 is configured in PostgreSQL otherwise the pivot process will crash with a SQL error.
 

doc/ref/modules/mod_translation.rst

@@ -17,8 +17,7 @@ of a :term:`resource` gets its own URL, and is as such indexable for
 Google.
 
 This behaviour is enabled by default, but can be switched off in the
-admin, by going to `Structure`, `Translation`. There is a checkbox at
-the bottom of the page.
+admin, by going to `Structure`, `Translation`. There is a checkbox labelled "Show the language in the URL".
 
 Alternatively you can set the config key
 ``mod_translation.rewrite_url`` to ``false``.
@@ -40,14 +39,12 @@ Creates a button which switches to Dutch. And another one for english::
 Supporting right-to-left languages
 ----------------------------------
 
-To support right-to-left languages like Arabic and Hebrew, you need to
-make some changes to your templates. mod_translation adds some support to help you.
+For basic use you don't need to do anything. Zotonic base site adds a ``lang`` attribute to the html tag, and when a right-to-left language is selected (for instance Arabic), the browser will interpret ``lang="ar"`` and automatically adapt the content to right-to-left.
 
-The idea is to give your ``<body/>`` tag the text direction of the main
-item visible on the page. Individual items, for example in menus or
-context lists, need their own text direction.
+Custom right-to-left content
+............................
 
-You can specify the text direction element attributes by including a template::
+If you write your own templates, you can add the ``lang`` tag in the html or body tag, for instance::
 
   <body {% include "_language_attrs.tpl" id=id %} >
 
@@ -59,7 +56,12 @@ When you want to add an extra class added to the rtl or ltr class you can use::
 
   <body {% include "_language_attrs.tpl" id=id class="my-body-class" %} >
 
+
+To create individual right-to-left elements, you can use the same principle::
+
+  <div {% include "_language_attrs.tpl" %}></div>
+
 And when you want to force a specific language::
 
-  <body {% include "_language_attrs.tpl" language=`en` %} >
+  <div {% include "_language_attrs.tpl" language=`en` %} >This is English content</div>
 

modules/mod_admin/lib/css/zotonic-admin.css

@@ -64,6 +64,11 @@ ol,
 p {
   margin: 0 0 15px 0;
 }
+
+table td,
+table th {
+  vertical-align: top;
+}
 iframe,
 video,
 audio {

modules/mod_admin/lib/less/base.less

@@ -29,6 +29,12 @@ p {
     margin: 0 0 @baseMargin 0;
 }
 
+table {
+    td, th {
+        vertical-align: top;
+    }
+}
+
 iframe, video, audio {
     max-width: 100%;
 }

modules/mod_admin/mod_admin.erl

@@ -173,13 +173,13 @@ event(#postback_notify{message= <<"admin-insert-block">>}, Context) ->
                         [];
                     Ls ->
                         Ls1 = string:tokens(Ls, ","),
-                        [ list_to_atom(L) || L <- lists:filter(fun z_trans:is_language/1, Ls1) ]
+                        [ list_to_atom(L) || L <- lists:filter(fun z_language:is_valid/1, Ls1) ]
                end,
     EditLanguage = case z_context:get_q("edit_language", Context) of
                     undefined ->
                         z_context:language(Context);
                     EL ->
-                        case z_trans:is_language(EL) of
+                        case z_language:is_valid(EL) of
                             true -> list_to_atom(EL);
                             false -> z_context:language(Context)
                         end
@@ -429,8 +429,8 @@ context_language(Context) ->
         undefined -> Context;
         [] -> Context;
         Lang ->
-            case z_trans:to_language_atom(Lang) of
-                {ok, IsoCode} -> z_context:set_language(IsoCode, Context);
+            case z_language:to_language_atom(Lang) of
+                {ok, LanguageCode} -> z_context:set_language(LanguageCode, Context);
                 _ -> Context
             end
     end.

modules/mod_admin/templates/_action_dialog_edit_basics.tpl

@@ -1,5 +1,5 @@
 {% with m.config.i18n.language_list.list as languages %}
-    {% with m.rsc[id].language as r_lang %}
+    {% with m.rsc[id].name as r_lang %}
         {% wire id=#form type="submit"
 	        postback={
 	            rsc_edit_basics

modules/mod_admin/templates/_admin_menu.tpl

@@ -46,7 +46,7 @@
 
             <div class="navbar-right">
                 <ul class="nav navbar-nav">
-                    {% all include "_admin_headeritem.tpl" %}
+                    {% all include "_admin_headeritem.tpl" is_nav %}
 
                     {# at the far right is the account menu #}
                     {% with

modules/mod_admin/templates/_admin_translation_tabs.tpl

@@ -3,9 +3,11 @@
 {% with edit_language|member:r_language|if:edit_language:(r_language[1]) as edit_language %}
 <ul class="nav nav-tabs language-tabs">
     {% for code,lang in m.config.i18n.language_list.list|default:[[z_language,[]]] %}
-    <li class="tab-{{ code }} {% if code == edit_language %}active{% endif %}" {% if not code|member:r_language or not lang.is_enabled %}style="display: none"{% endif %} data-index="{{ forloop.counter0 }}" {% include "_language_attrs.tpl" language=code %}><a href="#{{ prefix }}-{{ code }}" data-toggle="tab">{{ lang.language|default:z_language }}</a></li>
+    <li class="tab-{{ code }} {% if code == edit_language %}active{% endif %}" {% if not code|member:r_language or not lang.is_enabled %}style="display: none"{% endif %} data-index="{{ forloop.counter0 }}" {% include "_language_attrs.tpl" language=code %}><a href="#{{ prefix }}-{{ code }}" data-toggle="tab">{{ lang.name_en|default:z_language }}</a></li>
     {% endfor %}
 </ul>
 {% endwith %}
 {% endwith %}
 {% endwith %}
+
+

modules/mod_admin/templates/admin_base.tpl

@@ -25,6 +25,7 @@
             "css/zotonic-admin.css"
         %}
 
+        {% all include "_html_head.tpl" %}
         {% all include "_html_head_admin.tpl" %}
 
         {% include "_js_include_jquery.tpl" %}

modules/mod_admin/templates/admin_edit_widget_i18n.tpl

@@ -1,7 +1,9 @@
-{% with m.modules.active.mod_translation,
-        r_language|default:(m.rsc[id].language)|default:[z_language]
-   as is_i18n,
-      r_language
+{% with
+    m.modules.active.mod_translation,
+    r_language|default:(m.rsc[id].language)|default:[z_language]
+    as
+    is_i18n,
+    r_language
 %}
 {% with edit_language|default:z_language as edit_language %}
 {% with edit_language|member:r_language|if:edit_language:(r_language[1]) as edit_language %}

modules/mod_admin_config/mod_admin_config.erl

@@ -61,4 +61,3 @@ event(#submit{message={config_save, Args}}, Context) ->
         false ->
             z_render:growl_error("Only administrators can delete configurations.", Context)
     end.
-

modules/mod_admin_identity/templates/_admin_edit_sidebar.person.tpl

@@ -30,7 +30,7 @@
 		    <select class="form-control" id="pref_language" name="pref_language">
 				<option></option>
 				{% for code,lang in m.translation.language_list_enabled %}
-					<option {% if id.pref_language == code %}selected{% endif %} value="{{ code }}">{{ lang.language }}</a>
+					<option {% if id.pref_language == code %}selected{% endif %} value="{{ code }}">{{ lang.name }}</a>
 				{% endfor %}
 			</select>
 	    </div>