New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve index_url passing for setup_requires #238

Open
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
5 participants
@jonathanultis

jonathanultis commented Feb 12, 2015

4 tests fail in master before and after this patch.

This patch makes setuptools respect the index_url for setup_requires entries. They still won't respect the versions list. But this is a little better.

It fixes an issue with cliff and stevedore, which use the pbr library to share setup logic.

@gotcha

This comment has been minimized.

Show comment
Hide comment
@gotcha

gotcha Feb 16, 2015

Member

Can you add a test so that we understand what you are trying to achieve ?

Member

gotcha commented Feb 16, 2015

Can you add a test so that we understand what you are trying to achieve ?

@jonathanultis

This comment has been minimized.

Show comment
Hide comment
@jonathanultis

jonathanultis Feb 16, 2015

Not easily. Test setup requires a local pypiserver, and the ability to prevent access to pypi.python.org in some way. I accomplished that by setting a bogus entry for pypi.python.org in my /etc/hosts file for a while.

With that setup:

$ cat buildout.cfg
[buildout]
# point at the local pypiserver
index = localhost:8080
parts = scripts
develop = .

[scripts]
recipe = zc.recipe.egg:scripts
eggs =
    setuprequires
interpreter = python

$ cat setup.py
from setuptools import setup
setup(
    name = "setuprequires",
    version = "1.0",
    description = ("verify that setup_requires eggs use the proper index"),
    install_requires = ["setuptools", "cliff", "stevedore"]
)

First, bootstrap with access to pypi.python.org open. Then, remove access to pypi.python.org. Then buildout.

If pypi.python.org is inaccessible, the build will fail because of the setup_requires entry for "pbr" here

Unfortunately, buildout can't get the list of setup_requires eggs without triggering the download and install of those eggs. So, setup_requires dodge buildout's index url and version pinning, right now.

This patch improves the situation a little bit by asking the egg to use the correct index url for setup_requires eggs.

Internally, we use a bootstrap that pulls ez_setup.py from a bandersnatch mirror instead of bootstrap.pypi.io, and an ez_setup.py that pulls setuptools from the local bandersnatch mirror as well. Our goal is to keep our builds entirely isolated from pypi.python.org reliability issues.

jonathanultis commented Feb 16, 2015

Not easily. Test setup requires a local pypiserver, and the ability to prevent access to pypi.python.org in some way. I accomplished that by setting a bogus entry for pypi.python.org in my /etc/hosts file for a while.

With that setup:

$ cat buildout.cfg
[buildout]
# point at the local pypiserver
index = localhost:8080
parts = scripts
develop = .

[scripts]
recipe = zc.recipe.egg:scripts
eggs =
    setuprequires
interpreter = python

$ cat setup.py
from setuptools import setup
setup(
    name = "setuprequires",
    version = "1.0",
    description = ("verify that setup_requires eggs use the proper index"),
    install_requires = ["setuptools", "cliff", "stevedore"]
)

First, bootstrap with access to pypi.python.org open. Then, remove access to pypi.python.org. Then buildout.

If pypi.python.org is inaccessible, the build will fail because of the setup_requires entry for "pbr" here

Unfortunately, buildout can't get the list of setup_requires eggs without triggering the download and install of those eggs. So, setup_requires dodge buildout's index url and version pinning, right now.

This patch improves the situation a little bit by asking the egg to use the correct index url for setup_requires eggs.

Internally, we use a bootstrap that pulls ez_setup.py from a bandersnatch mirror instead of bootstrap.pypi.io, and an ez_setup.py that pulls setuptools from the local bandersnatch mirror as well. Our goal is to keep our builds entirely isolated from pypi.python.org reliability issues.

@reinout

This comment has been minimized.

Show comment
Hide comment
@reinout

reinout Jun 11, 2015

Contributor

The idea looks good in principle and I can see what you're trying to achieve.

But: why cannot buildout figure out the setup_requires? It can figure out the regular install_requires. Perhaps there's something we could fix to get that working? Or is there some internal setuptools reason why it is impossible for buildout?

If so, perhaps setuptools could be changed to allow it. That would get the version pinning working, too.

Contributor

reinout commented Jun 11, 2015

The idea looks good in principle and I can see what you're trying to achieve.

But: why cannot buildout figure out the setup_requires? It can figure out the regular install_requires. Perhaps there's something we could fix to get that working? Or is there some internal setuptools reason why it is impossible for buildout?

If so, perhaps setuptools could be changed to allow it. That would get the version pinning working, too.

@reinout

This comment has been minimized.

Show comment
Hide comment
@reinout

reinout Jun 11, 2015

Contributor

See also my comment #244 (comment)

Contributor

reinout commented Jun 11, 2015

See also my comment #244 (comment)

@idgserpro

This comment has been minimized.

Show comment
Hide comment
@idgserpro

idgserpro Aug 17, 2018

See also my comment #244 (comment)

@reinout as stated in the setuptools documentation, setup_requires isn't the dependency on the package that declares it but rather the dependency on the installation script.

So I don't think it necessary to buildout storage these eggs and nor to include in scritps that they are installing. In fact, this seems wrong. See the documentation that easy_install doesn't include these eggs in the python site-package.

But it would be a great idea the easy_install use the same buildout index. This avoids errors in an environment where the access of pypi is blocked and the installation of the packages is done through a local mirror.

@jonathanultis 's idea is great!

idgserpro commented Aug 17, 2018

See also my comment #244 (comment)

@reinout as stated in the setuptools documentation, setup_requires isn't the dependency on the package that declares it but rather the dependency on the installation script.

So I don't think it necessary to buildout storage these eggs and nor to include in scritps that they are installing. In fact, this seems wrong. See the documentation that easy_install doesn't include these eggs in the python site-package.

But it would be a great idea the easy_install use the same buildout index. This avoids errors in an environment where the access of pypi is blocked and the installation of the packages is done through a local mirror.

@jonathanultis 's idea is great!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment