setup.py containing __requires__ fails when run without rwt

[anthrotype]

hi @jaraco,

Here's a setup.py which defines __requires__ for use with rwt tool. It's taken straight from your README.rst for the sake of example.

#!/usr/bin/env python

__requires__ = ['setuptools', 'setuptools_scm']

from setuptools import setup

setup(
    name='test_rwt',
    use_scm_version=True,
    setup_requires=__requires__,
)

Now, if I try to run this without also invoking python -m rwt -- setup.py ..., like the usual:

$ python3 setup.py --help

I get this error from pkg_resources:

Traceback (most recent call last):
  File "setup_rwt.py", line 5, in <module>
    from setuptools import setup
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/__init__.py", line 12, in <module>
    import setuptools.version
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/version.py", line 1, in <module>
    import pkg_resources
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3018, in <module>
    @_call_aside
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3002, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3031, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 654, in _build_master
    ws.require(__requires__)
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 962, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 848, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'setuptools_scm' distribution was not found and is required by the application

The error happens when a requirement listed in __requires__ (like setuptools_scm here) is not already installed in the current environment.

It seems like pkg_resources (imported by setuptools) also attempts to use the __requires__ variable, in WorkinSet._build_master method:

https://github.com/pypa/setuptools/blob/089cdeb489a0fa94d11b7307b54210ef9aa40511/pkg_resources/__init__.py#L640-L658

This behavior is documented in the pkg_resources docs:
http://setuptools.readthedocs.io/en/latest/pkg_resources.html#workingset-objects

All distributions available directly on sys.path will be activated automatically when pkg_resources is imported. This behaviour can cause version conflicts for applications which require non-default versions of those distributions. To handle this situation, pkg_resources checks for a requires attribute in the main module when initializing the default working set, and uses this to ensure a suitable version of each affected distribution is activated. For example:

__requires__ = ["CherryPy < 3"] # Must be set before pkg_resources import
import pkg_resources

In your REAME.rst, you say that this technique:

When invoked with rwt, the dependencies will be assured before the script is run, or if run with setuptools, the dependencies will be loaded using the older technique, so the script is backward compatible.

However, it appears to me that one cannot have such a backward-compatible setup.py which also takes advantage of rwt via the __requires__ variable, because once one uses that, it fails as one tries to run it in the old-fashioned way.

Am I missing something?
Thank you for your help.

In the meantime I think I'll just use python3 -m rwt -r requirements/setup.txt -- setup.py ... instead of __requires__.

[anthrotype]

forgot to mention, I'm using python3.6 and setuptools==34.0.0 released a couple of hours ago, but the error also occurs with previous versions of setuptools/pkg_resources as well as with python2.7.

[jaraco]

Hmm. I guess what I meant when I said the script was backward compatible is that if you were relying on __requires__ to declare your requirements before via setuptools/pkg_resources, that now they can be installed from your declaration by rwt, and that pkg_resources will continue to validate those declarations as it did before. In other words, if you had those dependencies installed using easy_install -m, pkg_resources would make those packages visible as a result of the __requires__. In that sense, the two syntaxes are compatible.

I'm not sure what you're expecting to happen beyond that. If you're using __requires__ in a script, you're declaring that the script can't run unless those distributions can be resolved, and rwt goes a step further to ensure they are.

I think you're right that if it's not your intention to require the script to be run by rwt or require the person invoking it to have installed those packages in advance, then you'll want to fall back to declaring the requirements elsewhere.

I had originally intended for rwt to be a solution to setup_requires, but it appears that rwt contradicts intentions established in PEP 518.

[jaraco]

Does that clarify?

[anthrotype]

Yes, thanks for the clarification.
It was a misunderstanding on my part, as I am not familiar with pkg_resources API.
I'm still not sure why would rwt be in contradiction with PEP518 though.
I'll re-read that thread more carefully.
Thanks again!

[jaraco]

I think the contradiction comes with the prescribed means of declaration. rwt uses __requires__ as a simple, embedded mechanism to solicit requirements, whereas PEP 518 declares a pyproject.toml file which requires external libraries just to parse but also requires an additional file, two uses that rwt is specifically seeking to avoid.

There's some overlap, though, and there's a good chance the core of rwt can be repurposed to implement PEP 518 after the requirements are parsed.