New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop support for self upgrade/installation #581

Closed
jaraco opened this Issue May 10, 2016 · 29 comments

Comments

Projects
None yet
@jaraco
Member

jaraco commented May 10, 2016

I'd like to consider dropping support for setuptools installing or upgrading itself, including the removal of the bootstrap script, and instead, rely entirely on pip for installation of setuptools. Doing so would (a) allow for the simple declaration of dependencies over vendoring them, (b) eliminate the need for a bootstrap process besides simple 'pip install', (c) eliminate the special handling for self installation, and (d) limit setuptools installation to the single-version installs that pip allows.

I expect some environments like bootstrap or other systems might be affected, so I'm cautious about this possibility. This ticket serves to host the conversation and considerations.

@bhaugen

This comment has been minimized.

bhaugen commented Jul 31, 2016

I'd suggest you also put some official upgrade instructions in the readme and docs. I don't see anything there, and I see the usual conflicting advice on the internet. Same for pypi.

I tried this, which I found in some comments here:
sudo pip install setuptools –upgrade –no-use-wheel
which seemed to work fine.

@jaraco

This comment has been minimized.

Member

jaraco commented Aug 1, 2016

The no-use-wheel recommendation is concerning, because that option indicates to setuptools that it needs to build itself, which is contrary to the goal of this ticket. I think one assumption is that setuptools would always be installed from wheels, and the build step would only be invoked on systems that already had a setuptools installation.

But you're right, the upgrade story needs to be well understood and documented.

@bhaugen

This comment has been minimized.

bhaugen commented Aug 1, 2016

I hope it's clear that I was not recommending –no-use-wheel, just copying and pasting.

@jayvdb

This comment has been minimized.

Contributor

jayvdb commented Aug 1, 2016

I have recently needed to install setuptools manually because pip couldn't do it.

https://github.com/PyCQA/pyflakes/pull/76/files#diff-11c909939117928998b102a1fff7d363R33

@pombredanne

This comment has been minimized.

pombredanne commented Aug 26, 2016

I think it makes a lot of sense for a cleaner and consistent experience and clean separation of duties. It will make the overall package landscape more coherent: setuptools builds (and optionally installs from sources) eventually programmatically while pip bootstraps and installs.

The only area where I can fathom a possible issue for some rare tools is the lack of a stable function API in pip when using it as a library. But even then, spawning a pip subprocess is not a big issue (instead of calling its functions) from Python and this is something I routinely do.

@sigmavirus24

This comment has been minimized.

Member

sigmavirus24 commented Sep 7, 2016

@jayvdb let's be entirely forthright here, pyflakes needed to install setuptools manually on AppVeyor and pypy. That also should have been a separate bug report to fix that. It's related to this, yes, but not something that negates this goal.

@jaraco jaraco self-assigned this Jan 1, 2017

jaraco added a commit that referenced this issue Jan 1, 2017

Replace the bootstrap script with a simplified pip wrapper, maintaini…
…ng nominal API compatibility for use_setuptools(<version>) and disabling download_setuptools and support for command-line parameters to the script invocation itself. Ref #581.

jaraco added a commit that referenced this issue Jan 2, 2017

@jaraco

This comment has been minimized.

Member

jaraco commented Jan 2, 2017

In the feature/581-depend-not-bundle branch, I've drafted an implementation that passes tests and installs nicely as a wheel. If you would like to try it out, for a limited time, the wheel of g767dcea0 can be installed as so:

pip install https://dl.dropboxusercontent.com/s/nn2nh33fvqkz2u5/setuptools-33.0.0.post20170101-py2.py3-none-any.whl
@AraHaan

This comment has been minimized.

AraHaan commented Jan 2, 2017

hmm I think having it upgrade itself (as an optional configuration option) should be added and default to not set to do it. So that way those who prefers this self upgrades can do it but before it upgrades itself it checks for upgrades to the other dependencies instead of shipping them with setuptools. But wouldn't this break pip for a little while as some parts of pip use setuptools?

So basically (as an optional config) have to self updates go in this order:

  1. Any updated dependencies that setuptools requires? install them.
  2. Any updates for setuptools? if so install them.

This way for those who (like me) does not always remember to pip install modules to know when there is a new version it would be easier to handle. Besides I have been working on things like this for my code so I dont have to remember to update them constantly to robot pip.

@jaraco

This comment has been minimized.

Member

jaraco commented Jan 2, 2017

@AraHaan I don't think there's any way to achieve what you describe. Part of the problem is that easy_install of any package over a pip-installed version of that package is liable to break. easy_install doesn't know how to uninstall pip-installed packages, so I wouldn't even expect a self-upgrade to work.

Now if you're talking about as strictly easy_install-managed environment, perhaps your suggestion would be plausible, but still very tricky. The way easy_install works is it downloads the sdist of the target package (i.e. setuptools), expands it to a temporary folder, builds a bdist_egg of that package, installs that egg, then determines which unsatisfied dependencies exist and installs them. The problem lies in the 'build' step, which because a 'setuptools' directory is present in that build directory, means that it's already using the newer version of setuptools to build itself. One would have to specifically special-case setuptools to inspect that package metadata and determine its dependencies to install those in advance. While possible, this approach would have additional pitfalls. What if this new version of setuptools relies on a dependency that's not compatible with the dependency that the current version of setuptools requires? Then this pre-install of dependencies might succeed but the install of the new version of setuptools might not succeed, leaving the environment in a broken state. Additionally, the way that setuptools isolates itself from the installation of the target package is to use in-process tricks (sandbox module) to hide existing imports of setuptools - that approach would also need to be expanded to include hiding of existing imports of dependencies.

In other words, it may be possible, but it's an awful lot of work and investment in a tool chain that's long been superseded by pip. If others wish to invest this effort, I won't resist it, but personally, my focus will be on pushing to the future rather than dragging forward the past.

@jaraco

This comment has been minimized.

Member

jaraco commented Jan 16, 2017

I've addressed the concern above about pyflakes on appveyor by providing a technique for getting pip installed and using that to install setuptools on pypy.

I'm ready now to proceed with the release. I've been running the branch version of setuptools for some time now with success. I'd like to invite the community one last time to give this version a try and identify issues that may arise.

Here's a link to the wheel: https://dl.dropboxusercontent.com/s/4wwpuzpg1ortmgn/setuptools-33.1.1.post20170116-py2.py3-none-any.whl
Here's a link to the sdist (though you won't be able to pip install it without first satisfying dependencies): https://dl.dropboxusercontent.com/s/r1aae2zqik5px0h/setuptools-33.1.1.post20170116.zip

Please try it out and identify any issues in your environment before I cut a release.

@jaraco

This comment has been minimized.

Member

jaraco commented Jan 16, 2017

Oh, and if you're using the ez_setup.py bootstrap script, please try the version found in the bootstrap-581 branch.

@dstufft

This comment has been minimized.

Member

dstufft commented Jan 17, 2017

This may be a silly idea, but it seems like with this branch it'd make sense to just deprecate and remove ez_setup.py completely. It's current incarnation feels like it's likely to be fairly fragile since it depends on having pip installed already (and one of the cases where someone might use ez_setup.py is if they have an empty environment). It seems like directing people to use get-pip.py (which will currently install pip, setuptools, and wheel) and doesn't depends on anything but Python existing is a better solution, but I may be missing something.

@jaraco

This comment has been minimized.

Member

jaraco commented Jan 17, 2017

Not a silly idea at all. My goal was to be minimally invasive and provide a compatibility shim that would ease the transition and guide its users to the same conclusion (just use pip directly). I was particularly concerned about the bootstrap scenario, which invokes ez_setup.py through its API. They bundle ez_setup.py in their own bootstrap script. But now that you say it, it's probably the case that their bootstrap will fail at the implicit assumption that pip is installed, in which case it's no help.

It seems to me the options are to provide an ez_setup.py script that tries to do the right thing or to break it down and have it do nothing but raise an error message that it's no longer supported. Leaving it in place as-is will only lead users to an error during the install, which will appear like a bug:

$ curl https://bootstrap.pypa.io/ez_setup.py -O
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12764  100 12764    0     0  53073      0 --:--:-- --:--:-- --:--:-- 53183
$ python -m venv --without-pip ~/.envs/ez-setup     
$ ~/.envs/ez-setup/bin/python ez_setup.py --version=33.1.1.post20170116 --download-base=https://dl.dropboxusercontent.com/s/r1aae2zqik5px0h/
Downloading https://dl.dropboxusercontent.com/s/r1aae2zqik5px0h/setuptools-33.1.1.post20170116.zip
Extracting in /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmpltp77_zq
Now working in /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmpltp77_zq/setuptools-33.1.1.post20170116
Installing Setuptools
Traceback (most recent call last):
  File "setup.py", line 11, in <module>
    import setuptools
  File "/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmpltp77_zq/setuptools-33.1.1.post20170116/setuptools/__init__.py", line 10, in <module>
    from six.moves import filter, map
ModuleNotFoundError: No module named 'six'
Something went wrong during the installation.
See the error message above.

I'll go with the latter option and simply deprecate ez_setup.py.

jaraco added a commit that referenced this issue Jan 17, 2017

@dstufft

This comment has been minimized.

Member

dstufft commented Jan 17, 2017

Not so much a suggestion but really just a statement that this is going to currently make installing setuptools via sdist via pip more annoying since people will have to manually install setuptools' build dependencies (since pip has no concept of that yet, and setup_requires won't work in this case).

That being said, pypa/pip#4144 should hopefully land in the near future and allow setuptools to add a pyproject.toml file to itself that looks like:

[build-system]
requires = [
    "wheel",
    "packaging>=16.8",
    "six>=1.10.0",
    "appdirs>=1.4.0",
]

and pip will basically do what rwt setup.py would do, except with just those dependencies.

@jaraco

This comment has been minimized.

Member

jaraco commented Jan 17, 2017

Yes, thanks for that clarification and reference. The expectation is that users will typically install from wheels and anyone who wants to install from sdist will need to prep the environment in advance. Glad to hear that PEP 518 support is coming to pip.

@dstufft

This comment has been minimized.

Member

dstufft commented Jan 17, 2017

@jaraco I feel like this also paves the way for splitting pkg_resources out into its own package right? One that setuptools could depend on for both its own uses, and so everyne who currently used the setuptools == pkg_resources assumption in their dependency declaration is not broken, but start to allow people to depend on pkg_resources itself instead of all of setuptools?

jaraco added a commit that referenced this issue Jan 17, 2017

jaraco added a commit that referenced this issue Jan 23, 2017

Pin the bootstrap script to the latest version that supports self-ins…
…tall and signal that the bootstrap script is deprecated. Ref #581.
@jaraco

This comment has been minimized.

Member

jaraco commented Jan 23, 2017

As you can see ^ I've updated the bootstrap script to simply pin to the last version prior this new release, and that compatibility bootstrap script, while warning about the issue, should continue to support older environments.

mauritsvanrees added a commit to collective/i18ndude that referenced this issue Jun 19, 2017

Use virtualenv/pip on Travis.
Travis gives a conflict when trying it with bootstrap.

$ python bootstrap.py --buildout-version=2.9.3 --setuptools-version=33.1.1
ez_setup.py is deprecated and when using it setuptools will be pinned to 33.1.1 since it's the last version that supports setuptools self upgrade/installation, check pypa/setuptools#581 for more info; use pip to install setuptools
Downloading https://pypi.io/packages/source/s/setuptools/setuptools-33.1.1.zip
Extracting in /tmp/tmpeCj938
Now working in /tmp/tmpeCj938/setuptools-33.1.1
Building a Setuptools egg in /tmp/bootstrap-F88qGZ
warning: no files found matching '*' under directory 'setuptools/_vendor'
/tmp/bootstrap-F88qGZ/setuptools-33.1.1-py2.7.egg
no previously-included directories found matching 'doc'
no previously-included directories found matching 'old-tutorial'
Creating directory '/home/travis/build/collective/i18ndude/eggs'.
Creating directory '/home/travis/build/collective/i18ndude/bin'.
Creating directory '/home/travis/build/collective/i18ndude/parts'.
Creating directory '/home/travis/build/collective/i18ndude/develop-eggs'.
Generated script '/home/travis/build/collective/i18ndude/bin/buildout'.

$ bin/buildout
Getting distribution for 'mr.developer==1.25'.
Got mr.developer 1.25.
Getting distribution for 'setuptools==33.1.1'.
/opt/python/2.7.9/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)
warning: no files found matching '*' under directory 'setuptools/_vendor'
While:
  Installing.
  Loading extensions.
  Getting distribution for 'setuptools==33.1.1'.
An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/home/travis/build/collective/i18ndude/eggs/zc.buildout-2.9.3-py2.7.egg/zc/buildout/buildout.py", line 2123, in main
    getattr(buildout, command)(args)
  File "/home/travis/build/collective/i18ndude/eggs/zc.buildout-2.9.3-py2.7.egg/zc/buildout/buildout.py", line 637, in install
    self._load_extensions()
  File "/home/travis/build/collective/i18ndude/eggs/zc.buildout-2.9.3-py2.7.egg/zc/buildout/buildout.py", line 1163, in _load_extensions
    newest=self.newest, allow_hosts=self._allow_hosts)
  File "/home/travis/build/collective/i18ndude/eggs/zc.buildout-2.9.3-py2.7.egg/zc/buildout/easy_install.py", line 913, in install
    return installer.install(specs, working_set)
  File "/home/travis/build/collective/i18ndude/eggs/zc.buildout-2.9.3-py2.7.egg/zc/buildout/easy_install.py", line 714, in install
    for dist in self._get_dist(req, ws):
  File "/home/travis/build/collective/i18ndude/eggs/zc.buildout-2.9.3-py2.7.egg/zc/buildout/easy_install.py", line 570, in _get_dist
    dist = self._env.best_match(requirement, ws)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1040, in best_match
    dist = working_set.find(req)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pkg_resources/__init__.py", line 672, in find
    raise VersionConflict(dist, req)
VersionConflict: (setuptools 12.0.5 (/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages), Requirement.parse('setuptools==33.1.1'))

adityagilra added a commit to adityagilra/nengo that referenced this issue Jun 21, 2017

Remove ez_setup.py
This was previously the recommended way to install setuptools
when the user had a Python environment, but no setuptools. For some
time now, this has been a pretty rare case; further, setuptools
has deprecated this method
(pypa/setuptools#581)
and instead recommends the `get-pip.py` script, which will also
install pip.
@astrofrog

This comment has been minimized.

astrofrog commented Aug 28, 2017

Just to be clear, is the recommended route for packages previously requiring ez_setup.py to simply remove the file? Is the idea that if installing e.g from a cloned repo, one should then either do:

pip install .

or

pip install setuptools  # if not already installed
python setup.py install

?

johncheetham added a commit to johncheetham/gshogi that referenced this issue Sep 5, 2017

Use prebuilt binbook. Update setuptools version.
Using prebuilt binbook means you can install on the system without
having to run the build step first. create_book.py can be used to
manually create the book.

It would no longer install using setuptools 18.3.1 and python 3.5.3
on Debian 9. This has been fixed by using the latest ez_setup.py which
bootstraps setuptools 33.1.1. This is deprecated and 33.1.1 is the last
that will use the bootstrap method.
see pypa/setuptools#581

joshmoore added a commit to joshmoore/openmicroscopy that referenced this issue Oct 30, 2017

Update ez_setup.py files
Downloaded from https://bootstrap.pypa.io/ez_setup.py (which is now
deprecated) this corrects a failing build in travis. The overall
ez_setup/setuptools strategy will need to be reviewed, taking
pypa/setuptools#581 into account.

joshmoore added a commit to joshmoore/openmicroscopy that referenced this issue Nov 2, 2017

Update ez_setup.py files
Downloaded from https://bootstrap.pypa.io/ez_setup.py (which is now
deprecated) this corrects a failing build in travis. The overall
ez_setup/setuptools strategy will need to be reviewed, taking
pypa/setuptools#581 into account.

@joshmoore joshmoore referenced this issue Nov 2, 2017

Merged

Ez setup fix #5563

sjperkins added a commit to ska-sa/montblanc that referenced this issue Nov 13, 2017

sjperkins added a commit to ska-sa/montblanc that referenced this issue Nov 13, 2017

Fix various setup py issues (#231)
* Add RIME op tensorflow source to MANIFEST.in

* Deprecate ez_setup.py script
  See pypa/setuptools#581.

* Include missing setup.cfg

* Pin python-casacore==2.1.2 for now

lpsinger added a commit to lpsinger/gwcelery that referenced this issue Feb 21, 2018

sjperkins added a commit to ska-sa/montblanc that referenced this issue Feb 23, 2018

Fix various setup py issues (#231)
* Add RIME op tensorflow source to MANIFEST.in

* Deprecate ez_setup.py script
  See pypa/setuptools#581.

* Include missing setup.cfg

* Pin python-casacore==2.1.2 for now
@jaraco

This comment has been minimized.

Member

jaraco commented Mar 17, 2018

Yes, one should do pip install ..

@Mirona

This comment has been minimized.

Mirona commented May 11, 2018

FYI, this completely messed up this tutorial and I don't know how to proceed: http://newcoder.io/begin/setup-your-machine/#windows

https://github.com/econchick/new-coder

How should the tutorial here be updated to reflect the changes? @econchik

image

image

image

@The-Compiler

This comment has been minimized.

@Mirona

This comment has been minimized.

Mirona commented May 11, 2018

Thank you @The-Compiler, was just looking there...

On this step: https://packaging.python.org/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date

image

There is an error:
image

Could not fetch URL https://pypi.python.org/simple/pip/: There was a problem confirming the ssl certificate: <urlopen
error [Errno 1] _ssl.c:504: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version>
Will skip URL https://pypi.python.org/simple/pip/ when looking for download links for pip in c:\python27\lib\site-pac
kages\pip-1.4.1-py2.7.egg

andersk added a commit to mit-scripts/scripts-pony that referenced this issue May 27, 2018

Remove ez_setup (now deprecated)
pypa/setuptools#581

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment