let me use setup_requires somehow so I can have abstractions in setup.py

[glyph]

The recommended mechanism for determining a project's version at setup time is completely ridiculous, and determining your own version is far from the most complicated thing that one might wish to do at setup / build time.

For years, I've wanted to write some abstractions for Twisted plugin developers to be able to properly install plugins without hand-crafting a solution in every single setup.py, and for years we've been stymied by the lack of any mechanism for setup.py itself to depend upon or import things.

The setup_requires field ostensibly solves this problem, except for the fact that it doesn't actually work. I can't figure out why it doesn't work, really, or how you're actually supposed to use it. It appears to invoke easy_install? It also appears to install the dependencies multiple times, but also not re-run setup.py to get correct information after they've been installed.

In the meanwhile, should I be looking to create a tool which, instead, generates a setup.py automatically?

(This is probably actually a distutils or setuptools bug, but I couldn't figure out which of the 9 bug trackers those projects have variously used in their forks and external maintenance and suchlike to use. But there's also some involvement from pip here, like, what the heck is going on with the easy_installed eggs of things I have perfectly good wheels of in my cache already…)

[qwcode]

yes, setup_requires processing invokes easy_install (at least for now), which has no knowledge of your wheel cache.

see here: https://pip.pypa.io/en/latest/reference/pip_install.html#controlling-setup-requires

which of the 9 bug trackers those projects have variously used in their forks

see the Packaging User Guide Projects page for current links: http://packaging.python.org/en/latest/projects.html

but also not re-run setup.py to get correct information after they've been installed.

I rarely use setup_requires myself, but are your imports after the setup() call. I think that's the key.

appears to install the dependencies multiple times

I'm not seeing that in a case I just worked up. can you paste your details?

[glyph]

With a fresh environment I can't reproduce. There are some other shenanigans going on in this setup.py, so one of them might be at issue 😫.

[dstufft]

So setup_requires is installed as the first step in setup(). However this means that it happens after the imports and such happen in the setup.py. There are ways around that... but they aren't very pretty. You might want to look at pbr which essentially does this by monkeypatching.

[dstufft]

but also not re-run setup.py to get correct information after they've been installed.

I rarely use setup_requires myself, but are your imports after the setup() call. I think that's the key.

I'm pretty sure this won't actually work.

[qwcode]

I rarely use setup_requires myself, but are your imports after the setup() call. I think that's the key.
I'm pretty sure this won't actually work.

btw, I only said that after I tried it. imports of setup_requires projects do work after the setup() call in setup.py. the reason I considered that at all was this line in the setuptools docs: "setuptools will attempt to obtain these (even going so far as to download them using EasyInstall) before processing the rest of the setup script"

[dstufft]

Right, the imports themselves will work, but you can't use those imports to pass anything into the setup() function. So it's not generally useful.

[qwcode]

yea, I guess there's nothing proper you'd ever want to be doing after the setup() call. I guess it's safe to say setup_requires only helps you to write setup.py command extensions.

[glyph]

Yeah my original understanding was that I was supposed to call setup() a second time. That … did not work properly :).

[dholth]

I've thought about defining a simple setup.cfg variable

install-requires = x y z

These would not necessarily be the full build requirements and would not require us to finish Metadata 2.0. Just what is required to execute setup.py itself.

I have a feeling that it could be messy to implement this in pip, but I haven't looked into it deeply.

[dstufft]

I think I am against adding an ad hoc setup.cfg variable like that.

[dstufft]

One thing we could do is change from a two pass system to a three pass system. The downside to that is that it would make the packages that rely on that pip specific which would do the wrong thing outside of pip. I don't think I actually like that very much to be honest though.

[dholth]

Whatever the hocness, there isn't another place to put it. You can't get setup.py's own requirements out of setup.py without running it and you can't run setup.py without installing its requirements.

It will still work about the same if you just run setup.py as long as you already have setup.py's requirements.

[dholth]

An alternative would be to allow setup_requires.txt in the root of the sdist. It is a pip requirements file. If pip finds it, then it will install it before running setup.py.

However we do it, static setup_requires would be a simple improvement that we can implement without having to invent anything else. I think people would love it and it would put us in a much better position to have alternative setup.py implementations.

We would not be breaking anything since the whole point of this is to enable cases where setup_requires doesn't work.

[dholth]

It looks like the thing to do is to write a new boilerplate setup.py that contains only a check to see whether the requirement were already installed, a subprocess call to "pip install the requirements in some directory" ; modify sys.path ; eval real-setup.py. It would read the setup_requires from a simple text file.

This would be sortof like what it does now with easy_install except it could possibly actually work, since the dependency-installing would happen before the dependency-importing does. No setup.py commands would need to be implemented in the new boilerplate file because the arguments are all just passed along to the implementation in the real-setup.py file.

[dholth]

Here's my first stab at the problem. Put a list of requirements, blank lines, and # comment lines in setup-requires.txt; rename setup.py to real-setup.py, and go. The script will subprocess.call() pip to install anything that's not installed into the setup-requires directory, and then calls the real setup.py in the same way pip does with exec(compile(...)).

https://gist.github.com/dholth/5c2296b5bfdf32deffb0

We should make sure it's polyglot, make it terser and see whether it works when a conflicting version of one of the setup-requirements is already installed.

[dholth]

Here's the full project. Copy setup.py into your own project to use. https://bitbucket.org/dholth/setup-requires

[glyph]

Interesting; thanks for that.

[piotr-dobrogost]

@dholth
Would it be possible to upgrade pip itself by placing it in setup-requires.txt? Upgrading pip itself by placing it in a requirement file was discussed in issue #1610

[dholth]

No, this isn't a way to upgrade pip. The linked setup-requires implementation installs things into a directory ./setup-requires/ and only adds them to PYTHONPATH for the duration of setup.py's execution. This is desirable for the use case.

[rbtcollins]

I think this is a dupe of #769

[rgommers]

Interesting, this is another piece of the puzzle that's needed for reliable xxx_requires. Adding ~30 LoC to an already long setup.py isn't too much of a problem I think.

I see that pkg_resources is imported in setup.py, so just to make sure: if setuptools is installed, is pkg_resources then always available? Or is it an extra dependency for a project using this approach?

[rbtcollins]

I've very skeptical about the viability of recursive call into pip - it caused massive issues for OpenStack when pbr did that, and when pip start locking environments to fix the known race conditions around distribution metadata changes, I expect it to break.

[dholth]

The intention is that pip should support setup-requires itself so that
my proof-of-concept / backwards compatibility shim would no longer have
to do the work. It takes years to add the feature to pip but it only
took a day to implement the shim.

On Mon, Oct 26, 2015, at 08:59 PM, rbtcollins wrote:

I've very skeptical about the viability of recursive call into pip -
it caused massive issues for OpenStack when pbr did that, and when pip
start locking environments to fix the known race conditions around
distribution metadata changes, I expect it to break.

— Reply to this email directly or view it on GitHub[1].

Links:

1. let me use setup_requires somehow so I can have abstractions in setup.py #1820 (comment)

[dstufft]

Closing this as a duplicate of #3691, which is the mechanism we will use to support this feature.