Please support 'Language' header field of PO files

[jrfern]

First of all my apologies, I'm neither a developer nor a native speaker of English, please excuse all the inaccuracies below.

In our project exelearning.net we use pybabel. I'm the coordinator of the translation teams, and some months ago found out that msgfmt had started producing warnings that a header field was missing in our PO files (see bug ticket https://forja.cenatic.es/tracker/?func=detail&atid=883&aid=1905&group_id=197). I searched the changelog of gettext and found out that in 2010 a new header field had been introduced, Language.

Documentation about the new header (from /usr/share/doc/gettext/NEWS)

Version 0.18 - May 2010

• (...)
• PO file format:
  There is a new field 'Language' in the header entry. It denotes the language
  code (plus optional country code) for the PO file. This field can be used
  by automated tools, such as spell checkers. It is expected to be more
  reliable than looking at the file name or at the 'Language-Team' field in
  the header entry.
  msgmerge, msgcat, msgen have a new option --lang that allows to specify
  this field. Additionally, msgmerge fills in this new field by looking at
  the 'Language-Team' field (if the --lang option is not given).

Does this gettext issue have anything to do with pybabel? Well, yes. Even if I added by hand the field to all our PO files, anytime we did a 'pybabel extract' and 'pybabel update' the field disppeared.

Pedro Peña prepared a patch for babel 1.3 that solved the problem for us

diff --git a/babel/messages/catalog.py b/babel/messages/catalog.py
index 501763b..e26e8f0 100644
--- a/babel/messages/catalog.py
+++ b/babel/messages/catalog.py
@@ -349,7 +349,10 @@ class Catalog(object):
else:
headers.append(('Language-Team', self.language_team))
if self.locale is not None:

•        headers.append(('Language', str(self.locale)))
       headers.append(('Plural-Forms', self.plural_forms))
  

•    else:
  

•        headers.append(('Language', 'LANGUAGE'))
   headers.append(('MIME-Version', '1.0'))
   headers.append(('Content-Type',
                   'text/plain; charset=%s' % self.charset))
  

Please check the patch and add it to your sources, or else find a different way to support the not-so-new header field.

[thanatos]

Yes please? I've been using Babel to see if it can help do translation for us, and it'd be great if read_po, if it encountered a Language MIME header, used that to set the locale attribute on the returned Catalog objects.

Presently, Babel doesn't even expose this header even through mime_headers: if you read in a PO file with that header, it just gets lost.

[jrfern]

Great to see new interest in this quite old bug report (January 2014). During this time I've learnt to create pull requests (clumsily, I'm afraid), and I created one yesterday (#147). Although the fix works locally (make downloads the cldr zip and all the tests pass), travis shows errors in files I haven't touched and the pull was not merged :-(

The reason? I still work in tag 1.3, for the 1.3 version of babel - does travis work in 2.0dev? How can I modify that?

[benselme]

Hi, the tests fail because of a new pytz version. Fixed in #134 but hasn't been merged.

[thanatos]

Oops! I wish I'd read your reply sooner. I tried to add support myself, my work is in my fork just so people are aware of it. I wasn't entirely sure how much our use-cases overlapped, so I just made the modifications I thought I needed. I've not opened a PR yet; I think I'll try to see how much they differ, as well as do some more testing to see if my changes meet my needs.

[jrfern]

As I'm not a programmer my PR is naïf: kept @pedropena's original patch, and just added the new header "Language" after all occurrences of "Language-Team", and gave it the value of the language part of that header. Make didn't complain about my changes.

But if @benselme is right, there's nothing we can do to have the merge accepted until the python-tz issue is fixed, amI right?

[tiborsimko]

Having Language header supported out of the box in Babel would be very good for compatibility with standard GNU gettext set of tools. Currently one gets into the missing Language header very quickly:

$ msgfmt -c -v wtforms.po 
wtforms.po:6: warning: header field 'Language' missing in header
32 translated messages, 3 fuzzy translations.

This makes unnecessary warnings when editing PO files in tools like Emacs's po-mode...

[felixonmars]

Oops, sorry for the reference flood.

With my PR #110 merged, this issue should be resolved now :)

[sils]

@felixonmars that reference "flood" is intended and it's a nice way to track the development of commits and shows we're doing good work so there's no need to be sorry - IMO. Thanks for helping out with this one!

@jrfern A fix has been applied to master it would be great if you (or someone else) could try to reproduce this although it shouldn't happen anymore. Reopen if it does anyway.

Thanks for everyone involved for providing info, reports and commits!

[sils]

@erickwilder this bug has not been backported to the 2.x maintenance branch, can you take care of this?

[sils]

@erickwilder I did the PR, please take a short look, merge and close this bug.

[sils]

I merged the backport, this will be up on pypi with the next release in the following days.