Extract file-finding and language-handling code from gettext.find

[jjinux]

BPO	1649329
Nosy	@loewis, @gpshead, @devdanzin, @tarekziade, @chrysn, @merwok
Files	gettext.patch: Patch showing a possible way to refactor the code

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2007-02-01.02:20:07.000>
labels = ['3.7', 'type-feature', 'library']
title = 'Extract file-finding and language-handling code from gettext.find'
updated_at = <Date 2017-09-27.21:43:24.307>
user = 'https://bugs.python.org/jjinux'

bugs.python.org fields:

activity = <Date 2017-09-27.21:43:24.307>
actor = 'eric.araujo'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2007-02-01.02:20:07.000>
creator = 'jjinux'
dependencies = []
files = ['8287']
hgrepos = []
issue_num = 1649329
keywords = ['patch']
message_count = 22.0
messages = ['54994', '54995', '54996', '54997', '54998', '54999', '55000', '55001', '55002', '84641', '119392', '120531', '122390', '122393', '122394', '122395', '122953', '122964', '128692', '128693', '303184', '303185']
nosy_count = 7.0
nosy_names = ['loewis', 'gregory.p.smith', 'ajaksu2', 'jjinux', 'tarek', 'chrysn', 'eric.araujo']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1649329'
versions = ['Python 3.7']

[jjinux]

If you distribute your code in a zipped egg, you can't use translation catalogs stored in that egg. That's because gettext.py only knows how to find the translation catalogs in the actual filesystem.

This wouldn't be so bad if the "find" function didn't serve double duty. On the one hand, it implements the "find" algorithm to pick the right languages and fallbacks. On the other hand, it actually resolves the files in the filesystem. It would be nice if these were separate. It seems like people in projects like Pylons are stuck copying code from the find function in order to work around this problem.

Perhaps gettext can be updated to know about eggs. Perhaps setting localedir to something like "egg://..." would enable this functionality.

[loewis]

I fail to see the bug. gettext.find behaves as specified; if you want something else, don't use that function.

If you want to load a .mo file from a zip file, you should be able to create a GNUTranslation object directly, from a file-like object.

I don't think that gettext should support eggs directly. If you want it to do things other than loading from file systems, you should generalize that appropriately. One appropriate generalization could be the introduction of a directory-like object, where you can do .exists(relpath), and .open(relpath). However, introduction of directory-like objets is PEP material.

Reclassifying this as a feature request.

[jjinux]

Sorry, yes, this is a feature request. The most important thing that I'm requesting is to refactor the code. That "find" function implements a certain algorithm that has nothing to do with the filesystem.

[loewis]

Do you have a proposal on how to do the refactoring? It would be fine to split find into two parts; it would not be acceptable (to me) to teach it egg: URLs.

[jjinux]

File Added: gettext.patch

[jjinux]

Ok, I've uploaded a patch with a possible way to refactor the code. I'm okay if you mark this bug as WONTFIX. I'm raising it as a concern because I know there are a lot of people out there using Web frameworks such as Pylons, Turbo Gears, etc. that are going to have to duplicate the code in "find" in order to use eggs. I hate to have see them do that. :-/

[loewis]

Ok, I see how this fixes find(). What about translation() (which calls find, then also performs open())? Wouldn't this need to be fixed as well?

Also, what about os.path.join? It might depend on the structure of the storage, as well (suppose you have the .mo data stored in a relational database)?

In principle, I'm willing to fix this. I just want to avoid adding a hack.

For example, it might be sensible to come up with a MOOpener interface, that has an .exists method, taking a list of strings, a .join method, taking a list of strings and returning something that can be passed to the .open method, which would take something that .join returned and returning a file-like object. It might be possible to extend the meaning of the localedir parameter to be either a string (understood as a directory name), or a MOOpener object.

In any case, people *will* have to duplicate find for the moment, as any changes we discuss will only appear in 2.6.

[jjinux]

What about translation() (which calls find, then also performs open())? Wouldn't this need to be fixed as well?

Perhaps, but it's probably not as critical. I.e. if the code gets duplicated to deal with eggs, it's not as big a loss. I really wish the distutils guys would weigh in on this one.

Also, what about os.path.join?

I think it's relatively harmless. One can work around that. One can't work around os.path.exists.

In principle, I'm willing to fix this. I just want to avoid adding a hack.

I agree. The patch isn't smelling too pretty. Neither of us wants to commit to a virtual filesystem layer which is how I've handled problems like this in the past. However, doing nothing results in a ton of code being duplicated in any project that uses gettext and zipped eggs. Should we see if Phillip Eby has any ideas since he's Mr. Eggs?

For example, it might be sensible to come up with a MOOpener interface,
that has an .exists method, taking a list of strings, a .join method,
taking a list of strings and returning something that can be passed to the
.open method, which would take something that .join returned and returning
a file-like object. It might be possible to extend the meaning of the
localedir parameter to be either a string (understood as a directory name),
or a MOOpener object.

That's a light virtual filesystem layer ;) If you want that, I have code for that.

In any case, people *will* have to duplicate find for the moment, as any changes we discuss will only appear in 2.6.

True. Alas, this may be a moot point.

Thanks for your time.

[jjinux]

Perhaps instead of committing to a special virtual filesystem layer, perhaps we should just take something that matches some subset of the posix interface. We can document the functions we need.

[devdanzin]

Not really a distutils issue, but fits the current distribution
discussion scope.

[merwok]

The specific problem of gettext+eggs seems to be solved by this project: https://pypi.python.org/pypi/EggTranslations/

Since the egg format is not retained in distutils2, I don’t think any support for it should be added in the stdlib.

FWIW, I like the use of file-like objects to build Translation instances. It’s easier to specify than a VFS and very versatile.

[merwok]

Barry said in http://mail.python.org/pipermail/python-dev/2009-March/087847.html :

The class-based API for gettext takes streams, so resource_stream()
would work just fine. I think i18n plugins for Python do not
necessarily need to use the classic gettext API.

The title as I changed it some time ago is thus invalid, so I’m changing it again. This is still a request for separating the language selection code from the file finding. Anyone feel free to enter a better title, this one is not very clear.

My first vote was to try to fix the code without writing a VFS, but OTOH maybe a lightweight ABC with mixin methods and a concrete implementation of the full gettext logic may be clear and educational here. Shannon, if you’re still getting those emails, what do you think?

[jjinux]

OTOH maybe a lightweight ABC with mixin methods and a concrete implementation of the full gettext logic may be clear and educational here. Shannon, if you’re still getting those emails, what do you think?

Yep, that'd be fine.

[jjinux]

Although, perhaps this bug is going away. It seems like using zip files for eggs is going out of vogue.

[merwok]

Would you like to propose a patch?

[jjinux]

I've never managed to get a patch into Python, but I wouldn't mind trying ;)

[jjinux]

Sorry, I just had a baby on Saturday. Hence, I'm a bit late getting to this. It might take me a couple weeks.

[merwok]

Congratulations. Take the time you’ll need, if this doesn’t go into 3.2 it’ll just be in 3.3.

[jjinux]

The more I think about this, the more I think we should just drop it.

• The easiest way around my original problem was to not install the eggs in zipped format. That's easy. Just mark the egg as not zip safe.
• I don't currently need this functionality.
• No one else is clamoring about it.

I think the increase in complexity just isn't warranted. If someone is really stuck, they can just hack their own version of gettext. It's not that big a deal, and it's fairly unlikely anyway.

Sorry for taking so long to respond; have 6 kids really makes keeping up with extracurricular activities difficult!

[merwok]

Fine, let’s close this then. Don’t hesitate to propose a patch for another bug report or feature request if you get enough free time someday :)

[gpshead]

FYI - We run into this problem getting things using gettext for i18n within a hermetic zipped up Python application at work. I'm not reopening it as I have nothing new to propose right now, but figured I'd drop a note that it does seem like a good idea.

Assuming that everything lives on a local filesystem is not a given.

[merwok]

Reopening. Eggs are still not officially blessed, but zipapps are.