New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Please support 'Language' header field of PO files #76

Closed
jrfern opened this Issue Jan 18, 2014 · 11 comments

Comments

Projects
None yet
7 participants
@jrfern

jrfern commented Jan 18, 2014

First of all my apologies, I'm neither a developer nor a native speaker of English, please excuse all the inaccuracies below.

In our project exelearning.net we use pybabel. I'm the coordinator of the translation teams, and some months ago found out that msgfmt had started producing warnings that a header field was missing in our PO files (see bug ticket https://forja.cenatic.es/tracker/?func=detail&atid=883&aid=1905&group_id=197). I searched the changelog of gettext and found out that in 2010 a new header field had been introduced, Language.

Documentation about the new header (from /usr/share/doc/gettext/NEWS)

Version 0.18 - May 2010

  • (...)
  • PO file format:
    There is a new field 'Language' in the header entry. It denotes the language
    code (plus optional country code) for the PO file. This field can be used
    by automated tools, such as spell checkers. It is expected to be more
    reliable than looking at the file name or at the 'Language-Team' field in
    the header entry.
    msgmerge, msgcat, msgen have a new option --lang that allows to specify
    this field. Additionally, msgmerge fills in this new field by looking at
    the 'Language-Team' field (if the --lang option is not given).

Does this gettext issue have anything to do with pybabel? Well, yes. Even if I added by hand the field to all our PO files, anytime we did a 'pybabel extract' and 'pybabel update' the field disppeared.

Pedro Peña prepared a patch for babel 1.3 that solved the problem for us

diff --git a/babel/messages/catalog.py b/babel/messages/catalog.py
index 501763b..e26e8f0 100644
--- a/babel/messages/catalog.py
+++ b/babel/messages/catalog.py
@@ -349,7 +349,10 @@ class Catalog(object):
else:
headers.append(('Language-Team', self.language_team))
if self.locale is not None:

  •        headers.append(('Language', str(self.locale)))
         headers.append(('Plural-Forms', self.plural_forms))
    
  •    else:
    
  •        headers.append(('Language', 'LANGUAGE'))
     headers.append(('MIME-Version', '1.0'))
     headers.append(('Content-Type',
                     'text/plain; charset=%s' % self.charset))
    

Please check the patch and add it to your sources, or else find a different way to support the not-so-new header field.

@thanatos

This comment has been minimized.

Contributor

thanatos commented Mar 3, 2015

Yes please? I've been using Babel to see if it can help do translation for us, and it'd be great if read_po, if it encountered a Language MIME header, used that to set the locale attribute on the returned Catalog objects.

Presently, Babel doesn't even expose this header even through mime_headers: if you read in a PO file with that header, it just gets lost.

@jrfern

This comment has been minimized.

jrfern commented Mar 6, 2015

Great to see new interest in this quite old bug report (January 2014). During this time I've learnt to create pull requests (clumsily, I'm afraid), and I created one yesterday (#147). Although the fix works locally (make downloads the cldr zip and all the tests pass), travis shows errors in files I haven't touched and the pull was not merged :-(

The reason? I still work in tag 1.3, for the 1.3 version of babel - does travis work in 2.0dev? How can I modify that?

@benselme

This comment has been minimized.

Member

benselme commented Mar 6, 2015

Hi, the tests fail because of a new pytz version. Fixed in #134 but hasn't been merged.

@thanatos

This comment has been minimized.

Contributor

thanatos commented Mar 7, 2015

Oops! I wish I'd read your reply sooner. I tried to add support myself, my work is in my fork just so people are aware of it. I wasn't entirely sure how much our use-cases overlapped, so I just made the modifications I thought I needed. I've not opened a PR yet; I think I'll try to see how much they differ, as well as do some more testing to see if my changes meet my needs.

@jrfern

This comment has been minimized.

jrfern commented Mar 7, 2015

As I'm not a programmer my PR is naïf: kept @pedropena's original patch, and just added the new header "Language" after all occurrences of "Language-Team", and gave it the value of the language part of that header. Make didn't complain about my changes.

But if @benselme is right, there's nothing we can do to have the merge accepted until the python-tz issue is fixed, amI right?

@tiborsimko

This comment has been minimized.

tiborsimko commented Mar 27, 2015

Having Language header supported out of the box in Babel would be very good for compatibility with standard GNU gettext set of tools. Currently one gets into the missing Language header very quickly:

$ msgfmt -c -v wtforms.po 
wtforms.po:6: warning: header field 'Language' missing in header
32 translated messages, 3 fuzzy translations.

This makes unnecessary warnings when editing PO files in tools like Emacs's po-mode...

@sils sils added the PR available label Jul 31, 2015

felixonmars added a commit to felixonmars/babel that referenced this issue Aug 5, 2015

Support 'Language' header field of PO files (python-babel#76)
GNU gettext has support for the 'Language' field in header entry since
version 0.18 (May 2010). This commit adds support for the field and
addresses python-babel#76.

felixonmars added a commit to felixonmars/babel that referenced this issue Aug 5, 2015

Support 'Language' header field of PO files (python-babel#76)
GNU gettext has support for the 'Language' field in header entry since
version 0.18 (May 2010). This commit adds support for the field and
addresses python-babel#76.

felixonmars added a commit to felixonmars/babel that referenced this issue Aug 5, 2015

Support 'Language' header field of PO files (python-babel#76)
GNU gettext has support for the 'Language' field in header entry since
version 0.18 (May 2010). This commit adds support for the field and
addresses python-babel#76.

felixonmars added a commit to felixonmars/babel that referenced this issue Aug 5, 2015

Support 'Language' header field of PO files (python-babel#76)
GNU gettext has support for the 'Language' field in header entry since
version 0.18 (May 2010). This commit adds support for the field and
addresses python-babel#76.

felixonmars added a commit to felixonmars/babel that referenced this issue Aug 5, 2015

Support 'Language' header field of PO files (python-babel#76)
GNU gettext has support for the 'Language' field in header entry since
version 0.18 (May 2010). This commit adds support for the field and
addresses python-babel#76.

felixonmars added a commit to felixonmars/babel that referenced this issue Aug 5, 2015

Support 'Language' header field of PO files (python-babel#76)
GNU gettext has support for the 'Language' field in header entry since
version 0.18 (May 2010). This commit adds support for the field and
addresses python-babel#76.
@felixonmars

This comment has been minimized.

Contributor

felixonmars commented Aug 6, 2015

Oops, sorry for the reference flood.

With my PR #110 merged, this issue should be resolved now :)

@sils

This comment has been minimized.

Member

sils commented Aug 6, 2015

@felixonmars that reference "flood" is intended and it's a nice way to track the development of commits and shows we're doing good work so there's no need to be sorry - IMO. Thanks for helping out with this one!

@jrfern A fix has been applied to master it would be great if you (or someone else) could try to reproduce this although it shouldn't happen anymore. Reopen if it does anyway.

Thanks for everyone involved for providing info, reports and commits!

@sils

This comment has been minimized.

Member

sils commented Sep 17, 2015

@erickwilder this bug has not been backported to the 2.x maintenance branch, can you take care of this?

@sils sils reopened this Sep 17, 2015

@sils sils added the needs backport label Sep 17, 2015

sils added a commit to sils/babel that referenced this issue Sep 17, 2015

Support 'Language' header field of PO files (python-babel#76)
GNU gettext has support for the 'Language' field in header entry since
version 0.18 (May 2010). This commit adds support for the field and
addresses python-babel#76.
@sils

This comment has been minimized.

Member

sils commented Sep 17, 2015

@erickwilder I did the PR, please take a short look, merge and close this bug.

sils added a commit to sils/babel that referenced this issue Sep 21, 2015

Support 'Language' header field of PO files (python-babel#76)
GNU gettext has support for the 'Language' field in header entry since
version 0.18 (May 2010). This commit adds support for the field and
addresses python-babel#76.

sils added a commit that referenced this issue Sep 21, 2015

Merge pull request #225 from sils1297/2.x-maintenance
BACKPORT: Support 'Language' header field of PO files (#76)
@sils

This comment has been minimized.

Member

sils commented Sep 21, 2015

I merged the backport, this will be up on pypi with the next release in the following days.

@sils sils closed this Sep 21, 2015

@pyup-bot pyup-bot referenced this issue Jul 4, 2017

Merged

Initial Update #2

@pyup-bot pyup-bot referenced this issue Oct 12, 2017

Merged

Initial Update #16

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment