Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mako should declare what extra dependencies are required for the custom gettext message extractors #304

Closed
sinoroc opened this issue Aug 21, 2019 · 11 comments

Comments

@sinoroc
Copy link
Contributor

commented Aug 21, 2019

Mako provides gettext message extractors for lingua and Babel. They are declared as setuptools entry points. Such entry points can and should specify a list of 'extras': third-party dependencies that should be installed in order to use these custom entry points.

See for example issue wichert/lingua#94:
lingua has the capability to use custom message extractors for both lingua and Babel. Now, if Mako and lingua are installed but Babel is not, then lingua blindly loads Mako's custom message extractors for Babel and fails while trying to use them since Babel is not installed.
To help lingua (and potentially other tools as well) to recognize that Babel should be installed in order to use the custom Mako-for-Babel message extractor, Mako should declare Babel as an extra dependency of the entry point for the message extractor.

See the setuptools documentation on "Dynamic Discovery of Services and Plugins" for details.

@sinoroc

This comment has been minimized.

Copy link
Contributor Author

commented Aug 21, 2019

I am working on fixes for both Mako and lingua.

sinoroc added a commit to sinoroc/mako that referenced this issue Aug 21, 2019
Specify extras for gettext message extractors
Mako provides 'gettext' message extractors for 'lingua' and 'Babel'.
They are declared as 'setuptools' entry points. Such entry points can
specify a list of 'extras'. Before loading the entry points
'pkg_resources' checks that these extra dependencies are available.
If they are missing pkg_resources raises the appropriate exception
and the caller can react accordingly.

GitHub: sqlalchemy#304

@zzzeek zzzeek added the installation label Aug 21, 2019

@zzzeek

This comment has been minimized.

Copy link
Member

commented Aug 21, 2019

OK silly question. Why isn't Babel a dependency of Lingua then?

@sinoroc

This comment has been minimized.

Copy link
Contributor Author

commented Aug 21, 2019

It's a bit confusing, but straightforward: lingua does not import any code from Babel, Mako does.

@zzzeek

This comment has been minimized.

Copy link
Member

commented Aug 21, 2019

OK that answer is not enough information - Mako only imports Babel in the Babel plugin which is not referenced anywhere else. in your stack trace at wichert/lingua#94, we see that lingua is not necessarily importing babel, but it is in fact calling upon it:

File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/lingua/extract.py", line 308, in main
    register_babel_plugins()

above is where lingua has a depedendency on babel. But it's not an "import" dependency, it's a set of dependencies at the entrypoint level, not something I've seen before. if lingua is hardcoded to install a babel entrypoint, I guess this could go either way but it still seems like lingua has babel as a pretty strong dependency.

@zzzeek

This comment has been minimized.

Copy link
Member

commented Aug 21, 2019

here's the better doc:

https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies

now I do have some "extras_require" in some other projects but I've not combined these with entrypoints. When exactly does "Babel" get installed?

@sinoroc

This comment has been minimized.

Copy link
Contributor Author

commented Aug 21, 2019

I have published a fix for lingua: wichert/lingua#95
Hopefully it clears up some points.

@sinoroc

This comment has been minimized.

Copy link
Contributor Author

commented Aug 21, 2019

My use case looks like this:

  • Pyramid application with Chameleon templates.
  • I use lingua to extract messages from my Chameleon templates and I don't want to use Babel at all.
  • For debug purposes I also have installed pyramid_debugtoolbar which has a non optional dependency on Mako. I assume that its Mako templates have already been localized, so I don't need to extract messages from these.

This causes the issue presented in wichert/lingua#94.

When exactly does "Babel" get installed?

When the user wants to use it to extract messages in their project. For this to work well whether or not Babel is installed, following changes are required:

  • Mako has to clearly publicize the fact that the entry point works only if Babel is installed.
  • lingua has to ignore the entry point if a dependency (Babel) is not installed.

This way lingua loads the Mako-for-Babel extractor only if Babel is explicitly installed, for example the user could add Mako[babel] to their project dependencies.

@zzzeek

This comment has been minimized.

Copy link
Member

commented Aug 23, 2019

OK by "when does babel get installed" I was more trying to determine the imperative effects of these directives being added. that is, without the directive, right now you're getting I assume the error, "can't find any module called Babel", or something like that (feel free to spell this out for me). and with the directive, what will happen in your case? the paths to this plugin will be silently ignored and your program just proceeds? (since this is all in a debug toolbar you aren't developing anyway)?

@sinoroc

This comment has been minimized.

Copy link
Contributor Author

commented Aug 23, 2019

Not entirely sure I understand the question, I will try to answer with this, I might still be off-topic though...

In short: with the extra directive lingua gains clearer knowledge of what is going wrong: a dependency is missing as well as its exact name ; and yes, it can then choose to ignore this plugin entirely.

Adding the extras to the entry points (when Babel is not installed), brings us from:

ModuleNotFoundError: No module named 'babel'

to:

pkg_resources.DistributionNotFound: The 'Babel' distribution was not found and is required by the application

(complete stack traces below ...)

The former is a relatively vague exception, the module name does not actually give viable information about which dependency is missing.

The latter shows clearly that a dependency is missing and what its name is.

In our case it makes sense to let lingua simply ignore the entry points that raise pkg_resources.DistributionNotFound, which is what has been done in wichert/lingua#95.

Alternatively lingua could show the user a warning listing the message extractors that have been ignored and which dependencies should be installed to activate them.


Complete stack traces

Without extras directive:

Traceback (most recent call last):
  File "/home/user/workspace/project/.tox/develop/bin/pot-create", line 11, in <module>
    sys.exit(main())
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/lingua/extract.py", line 308, in main
    register_babel_plugins()
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/lingua/extractors/babel.py", line 51, in register_babel_plugins
    extractor = entry_point.load(require=True)
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2324, in load
    return self.resolve()
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2330, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/mako/ext/babelplugin.py", line 8, in <module>
    from babel.messages.extract import extract_python
ModuleNotFoundError: No module named 'babel'

With extras directive:

Traceback (most recent call last):
  File "/home/user/workspace/project/.tox/develop/bin/pot-create", line 11, in <module>
    sys.exit(main())
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/lingua/extract.py", line 308, in main
    register_babel_plugins()
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/lingua/extractors/babel.py", line 51, in register_babel_plugins
    extractor = entry_point.load(require=True)
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2323, in load
    self.require(*args, **kwargs)
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2346, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/pkg_resources/__init__.py", line 778, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'Babel' distribution was not found and is required by the application
@zzzeek

This comment has been minimized.

Copy link
Member

commented Aug 23, 2019

OK so the change gives us a more informative error message, that's what I was looking for, thanks.

@sqla-tester

This comment has been minimized.

Copy link
Collaborator

commented Aug 26, 2019

sinoroc has proposed a fix for this issue in the master branch:

Specify extras for gettext message extractors https://gerrit.sqlalchemy.org/1436

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.