Add gettext.pgettext() and variants support

[genepi]

BPO	2504
Nosy	@loewis, @Cito, @merwok, @mitar, @nh2, @nphilipp, @berkerpeksag, @serhiy-storchaka, @ztane, @csabella, @eli-schwartz
PRs	bpo-2504: Add pgettext() and variants to gettext #7253
Files	gettext-pgettext.patch: Patch for gettext module to add pgettext and variants pgettext-3.2.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-11-07.14:32:50.869>
created_at = <Date 2008-03-29.02:10:23.548>
labels = ['3.8', 'type-feature', 'library']
title = 'Add gettext.pgettext() and variants support'
updated_at = <Date 2019-11-24.05:01:50.216>
user = 'https://bugs.python.org/genepi'

bugs.python.org fields:

activity = <Date 2019-11-24.05:01:50.216>
actor = 'eschwartz'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-07.14:32:50.869>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2008-03-29.02:10:23.548>
creator = 'genepi'
dependencies = []
files = ['10036', '19822']
hgrepos = []
issue_num = 2504
keywords = ['patch', 'needs review']
message_count = 33.0
messages = ['64675', '64685', '65526', '79609', '79610', '90338', '90339', '91472', '92796', '97788', '105126', '106121', '106132', '106709', '107560', '122439', '122446', '122482', '202189', '230537', '234853', '277652', '302801', '306594', '306600', '317139', '317300', '318206', '318261', '318282', '328643', '329417', '357389']
nosy_count = 30.0
nosy_names = ['loewis', 'cito', 'bronger', 'wichert', 'genepi', 'franz_glasner', 'eric.araujo', 'Felix Schwarz', 'dwayne', 'sascha_silbe', 'AndrewZiem', 'dns', 'mitar', 'bernie', 'nh2', 'clouserw', 'Flimm', 'olivier-berten', 'nils', 'bersace', 'berker.peksag', 'serhiy.storchaka', 'ztane', 'nhooey', 'Tristan.Fisher', 'Hanno.Zulla', 'Leonid Suprun', 'cheryl.sabella', 'Jonathan Schoonhoven', 'eschwartz']
pr_nums = ['7253']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue2504'
versions = ['Python 3.8']

[genepi]

Please add support for pgettext(msgctxt, msgid) and variants (dpgettext,
dcpgettext...) in the gettext module.

I will not rephrase the justification for these functions and why
contexts are essential for good localization:
http://www.gnu.org/software/gettext/manual/gettext.html#Contexts

[loewis]

Would you like to work on a patch?

[franzglasner]

I have written a patch against the current Python trunk's gettext to
add support for pgettext and friends (upgettext, npgettext, unpgettext,
...).

I have also changed Tools/i18n/msgfmt.py to recognize the "msgctxt" keyword.
Some new unittests for dgettext and lgettext and variants are also included.

If the patch will be accepted then someone should drop a message to the
maintainers of GNU gettext to add the new functions as default keywords
for xgettext. Right now you have to call xgettext with
"--keyword=pgettext:1c,2" to extract messages with context.

Tools/i18n/pygettext.py does currently not handle context variants.

[vstinner]

I don't see any change in Modules/_localemodule.c: you reimplemented
pgettext() in pure Python. Why no reusing existing pgettext() function
(at least when it's present)?

[loewis]

The gettext module is intentionally written in pure Python; it should
stay that way. Whether or not the _locale module should also grow
support for pgettext is a different issue.

[dns]

I came across this ticket while looking for alternatives to Python's
gettext, since I need msgctx support. It seems a patch was supplied for
this a while back. I have never contributed to Python, and am not
familiar with your release process, but is there anything preventing its
inclusion? It would be very convenient to have this in a release version.

[loewis]

Only my lack of time prevents inclusion.

[mitar]

I would just like to add that I am also looking forwards to this feature.

[nh2]

Me too. This makes developing applications with good localizations in
Python really difficult.

[Flimm]

Same here.

[olivier-berten]

Pleeeeeeeaaaaaaaase ;-)

[Bernie]

While we're waiting for this patch to be upstreamed, what's the best way to emulate this functionality with the current gettext module?

I'm looking at the patch and it seems that code similar to this might work?

  def pgettext(ctx, msg):
     return gettext(ctx + "\x04" + msg)

[clouserw]

Yes. You can see an in-production implementation at http://github.com/clouserw/tower/blob/master/tower/__init__.py#L51 (I'm overriding ugettext to support an optional context).

[wichert]

Martin, is there anything we can do to help get this merged? I can really use this as well.

My background here is that currently the complete zope i18n support abuses message ids as a workaround, and the result works but is very painful for translators since the original string is not immediately visible for them.

[merwok]

The patch needs to be updated for the 3.x trunk (py3k branch), since the last 2.x version is already rc.

Everyone, thanks for expressing interest for the development of Python, but posting “me too” messages does not further the discussion, and actually takes time from already busy people. Use cases, patches, reviews and the occasional bump (one is enough) are helpful. :)

[merwok]

I have ported the patch to py3k, without tests errors but with test failures. What is left to do:

1. Carefully assess bytes/str issues. Decide to change the tests or the code.

2. Re-read over the new docs.

3. Exercise the updated msgfmt.py tool, which has no tests.

4. Add support in pygettext.py.

David, Mitar, nh2, David D., Olivier, Bernie, Wil, Wichert: Do you want to help with one of those tasks?

[wichert]

I can help test changes for python 2.x. The python 3.x ecosystem is at least a year away from becoming interesting for me I'm afraid.

[merwok]

New features don’t go into stable branches.

The patch is also reviewable at http://codereview.appspot.com/3340041

[nhooey]

Can someone review the patch and consider its inclusion?

[wichert]

Bump.

Python 3 is still not on my radar, but I'll happily do a backport for Py2 and drop that on PyPI once this gets resolved.

[HannoZulla]

Can we please get pgettext for Python?

[ztane]

*wow* isn't this in Python 3 yet (I've been using Zope translationstrings lately so I didn't know that pgettext is still not supported).

[serhiy-storchaka]

This looks like a nice feature. Éric, could you please create a PR on GitHub?

I think that new l*gettext functions are not needed. They are rather useless in Python 3 (see bpo-29755).

[JonathanSchoonhoven]

Is there anything I can do to help get this into the codebase and out issue purgatory? We're not that far off from the 10 year anniversary of this issue.

[merwok]

The patch would need to be applied on top of the current master to make a pull request. Then, the tasks outlined in that message should be addressed:

https://bugs.python.org/issue2504#msg122439

[csabella]

Éric,

Was your last comment intended as a todo for yourself or an outline of the steps needed for someone else to move this along (maybe it was a reply to Jonathan Schoonhoven's question)?

If the latter, I'd be happy to try to work on the PR.

Thanks!

[merwok]

It was an answer to «Is there anything I can do to help get this into the codebase»

Feel free to take this on! I’ll try to review.

[csabella]

I've created a pull request for this based on Éric's patch in 2010. I attempted to incorporate the changes from bpo-29755 into the new functions and tests. I believe these changes take care of items 1 and 2 from msg122439.

I haven't tried to address item 3 in msg122439 yet.

Since this is already a large change, item 4 in msg122439 might be better as its own issue.

[serhiy-storchaka]

Do you have a use case for lpgettext() and like?

[csabella]

No, I didn't have a use case for the l* functions. I included them just as a result of applying the entire patch from 2010. I also didn't know if further discussion was needed before excluding them.

[serhiy-storchaka]

l*gettext() functions have been just deprecated in bpo-33710. Please update the PR and remove new l*gettext() functions.

[serhiy-storchaka]

New changeset 637a33b by Serhiy Storchaka (Cheryl Sabella) in branch 'master':
bpo-2504: Add pgettext() and variants to gettext. (GH-7253)
637a33b

[eli-schwartz]

Interestingly enough, the final accepted patch (and the 2010 one) also fixes a bug where gettext.install(..., names='ngettext') would incorrectly work, in violation of the documentation.

I think it would also incorrectly install 'gettext', too...

It used to just check if names.__contains__ is a valid attribute, then check if 'foo' in names, so a dumb string "worked" and even matched more things than it should have.

How I discovered this: I fixed a bug in a python project that stopped working on python 3.8, and had a bit of a head-scratch regarding how it ever worked to begin with: linuxmint/cinnamon#8964