Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop support for self upgrade/installation #581

Closed
jaraco opened this issue May 10, 2016 · 29 comments
Closed

Drop support for self upgrade/installation #581

jaraco opened this issue May 10, 2016 · 29 comments
Assignees

Comments

@jaraco
Copy link
Member

@jaraco jaraco commented May 10, 2016

I'd like to consider dropping support for setuptools installing or upgrading itself, including the removal of the bootstrap script, and instead, rely entirely on pip for installation of setuptools. Doing so would (a) allow for the simple declaration of dependencies over vendoring them, (b) eliminate the need for a bootstrap process besides simple 'pip install', (c) eliminate the special handling for self installation, and (d) limit setuptools installation to the single-version installs that pip allows.

I expect some environments like bootstrap or other systems might be affected, so I'm cautious about this possibility. This ticket serves to host the conversation and considerations.

@bhaugen
Copy link

@bhaugen bhaugen commented Jul 31, 2016

I'd suggest you also put some official upgrade instructions in the readme and docs. I don't see anything there, and I see the usual conflicting advice on the internet. Same for pypi.

I tried this, which I found in some comments here:
sudo pip install setuptools –upgrade –no-use-wheel
which seemed to work fine.

@jaraco
Copy link
Member Author

@jaraco jaraco commented Aug 1, 2016

The no-use-wheel recommendation is concerning, because that option indicates to setuptools that it needs to build itself, which is contrary to the goal of this ticket. I think one assumption is that setuptools would always be installed from wheels, and the build step would only be invoked on systems that already had a setuptools installation.

But you're right, the upgrade story needs to be well understood and documented.

@bhaugen
Copy link

@bhaugen bhaugen commented Aug 1, 2016

I hope it's clear that I was not recommending –no-use-wheel, just copying and pasting.

@jayvdb
Copy link
Contributor

@jayvdb jayvdb commented Aug 1, 2016

I have recently needed to install setuptools manually because pip couldn't do it.

https://github.com/PyCQA/pyflakes/pull/76/files#diff-11c909939117928998b102a1fff7d363R33

@pombredanne
Copy link

@pombredanne pombredanne commented Aug 26, 2016

I think it makes a lot of sense for a cleaner and consistent experience and clean separation of duties. It will make the overall package landscape more coherent: setuptools builds (and optionally installs from sources) eventually programmatically while pip bootstraps and installs.

The only area where I can fathom a possible issue for some rare tools is the lack of a stable function API in pip when using it as a library. But even then, spawning a pip subprocess is not a big issue (instead of calling its functions) from Python and this is something I routinely do.

@sigmavirus24
Copy link
Member

@sigmavirus24 sigmavirus24 commented Sep 7, 2016

@jayvdb let's be entirely forthright here, pyflakes needed to install setuptools manually on AppVeyor and pypy. That also should have been a separate bug report to fix that. It's related to this, yes, but not something that negates this goal.

@jaraco jaraco self-assigned this Jan 1, 2017
jaraco added a commit that referenced this issue Jan 1, 2017
…ng nominal API compatibility for use_setuptools(<version>) and disabling download_setuptools and support for command-line parameters to the script invocation itself. Ref #581.
jaraco added a commit that referenced this issue Jan 2, 2017
@jaraco
Copy link
Member Author

@jaraco jaraco commented Jan 2, 2017

In the feature/581-depend-not-bundle branch, I've drafted an implementation that passes tests and installs nicely as a wheel. If you would like to try it out, for a limited time, the wheel of g767dcea0 can be installed as so:

pip install https://dl.dropboxusercontent.com/s/nn2nh33fvqkz2u5/setuptools-33.0.0.post20170101-py2.py3-none-any.whl
@AraHaan
Copy link

@AraHaan AraHaan commented Jan 2, 2017

hmm I think having it upgrade itself (as an optional configuration option) should be added and default to not set to do it. So that way those who prefers this self upgrades can do it but before it upgrades itself it checks for upgrades to the other dependencies instead of shipping them with setuptools. But wouldn't this break pip for a little while as some parts of pip use setuptools?

So basically (as an optional config) have to self updates go in this order:

  1. Any updated dependencies that setuptools requires? install them.
  2. Any updates for setuptools? if so install them.

This way for those who (like me) does not always remember to pip install modules to know when there is a new version it would be easier to handle. Besides I have been working on things like this for my code so I dont have to remember to update them constantly to robot pip.

@jaraco
Copy link
Member Author

@jaraco jaraco commented Jan 2, 2017

@AraHaan I don't think there's any way to achieve what you describe. Part of the problem is that easy_install of any package over a pip-installed version of that package is liable to break. easy_install doesn't know how to uninstall pip-installed packages, so I wouldn't even expect a self-upgrade to work.

Now if you're talking about as strictly easy_install-managed environment, perhaps your suggestion would be plausible, but still very tricky. The way easy_install works is it downloads the sdist of the target package (i.e. setuptools), expands it to a temporary folder, builds a bdist_egg of that package, installs that egg, then determines which unsatisfied dependencies exist and installs them. The problem lies in the 'build' step, which because a 'setuptools' directory is present in that build directory, means that it's already using the newer version of setuptools to build itself. One would have to specifically special-case setuptools to inspect that package metadata and determine its dependencies to install those in advance. While possible, this approach would have additional pitfalls. What if this new version of setuptools relies on a dependency that's not compatible with the dependency that the current version of setuptools requires? Then this pre-install of dependencies might succeed but the install of the new version of setuptools might not succeed, leaving the environment in a broken state. Additionally, the way that setuptools isolates itself from the installation of the target package is to use in-process tricks (sandbox module) to hide existing imports of setuptools - that approach would also need to be expanded to include hiding of existing imports of dependencies.

In other words, it may be possible, but it's an awful lot of work and investment in a tool chain that's long been superseded by pip. If others wish to invest this effort, I won't resist it, but personally, my focus will be on pushing to the future rather than dragging forward the past.

@jaraco
Copy link
Member Author

@jaraco jaraco commented Jan 16, 2017

I've addressed the concern above about pyflakes on appveyor by providing a technique for getting pip installed and using that to install setuptools on pypy.

I'm ready now to proceed with the release. I've been running the branch version of setuptools for some time now with success. I'd like to invite the community one last time to give this version a try and identify issues that may arise.

Here's a link to the wheel: https://dl.dropboxusercontent.com/s/4wwpuzpg1ortmgn/setuptools-33.1.1.post20170116-py2.py3-none-any.whl
Here's a link to the sdist (though you won't be able to pip install it without first satisfying dependencies): https://dl.dropboxusercontent.com/s/r1aae2zqik5px0h/setuptools-33.1.1.post20170116.zip

Please try it out and identify any issues in your environment before I cut a release.

@jaraco
Copy link
Member Author

@jaraco jaraco commented Jan 16, 2017

Oh, and if you're using the ez_setup.py bootstrap script, please try the version found in the bootstrap-581 branch.

@dstufft
Copy link
Member

@dstufft dstufft commented Jan 17, 2017

This may be a silly idea, but it seems like with this branch it'd make sense to just deprecate and remove ez_setup.py completely. It's current incarnation feels like it's likely to be fairly fragile since it depends on having pip installed already (and one of the cases where someone might use ez_setup.py is if they have an empty environment). It seems like directing people to use get-pip.py (which will currently install pip, setuptools, and wheel) and doesn't depends on anything but Python existing is a better solution, but I may be missing something.

@jaraco
Copy link
Member Author

@jaraco jaraco commented Jan 17, 2017

Not a silly idea at all. My goal was to be minimally invasive and provide a compatibility shim that would ease the transition and guide its users to the same conclusion (just use pip directly). I was particularly concerned about the bootstrap scenario, which invokes ez_setup.py through its API. They bundle ez_setup.py in their own bootstrap script. But now that you say it, it's probably the case that their bootstrap will fail at the implicit assumption that pip is installed, in which case it's no help.

It seems to me the options are to provide an ez_setup.py script that tries to do the right thing or to break it down and have it do nothing but raise an error message that it's no longer supported. Leaving it in place as-is will only lead users to an error during the install, which will appear like a bug:

$ curl https://bootstrap.pypa.io/ez_setup.py -O
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12764  100 12764    0     0  53073      0 --:--:-- --:--:-- --:--:-- 53183
$ python -m venv --without-pip ~/.envs/ez-setup     
$ ~/.envs/ez-setup/bin/python ez_setup.py --version=33.1.1.post20170116 --download-base=https://dl.dropboxusercontent.com/s/r1aae2zqik5px0h/
Downloading https://dl.dropboxusercontent.com/s/r1aae2zqik5px0h/setuptools-33.1.1.post20170116.zip
Extracting in /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmpltp77_zq
Now working in /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmpltp77_zq/setuptools-33.1.1.post20170116
Installing Setuptools
Traceback (most recent call last):
  File "setup.py", line 11, in <module>
    import setuptools
  File "/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmpltp77_zq/setuptools-33.1.1.post20170116/setuptools/__init__.py", line 10, in <module>
    from six.moves import filter, map
ModuleNotFoundError: No module named 'six'
Something went wrong during the installation.
See the error message above.

I'll go with the latter option and simply deprecate ez_setup.py.

jaraco added a commit that referenced this issue Jan 17, 2017
@dstufft
Copy link
Member

@dstufft dstufft commented Jan 17, 2017

Not so much a suggestion but really just a statement that this is going to currently make installing setuptools via sdist via pip more annoying since people will have to manually install setuptools' build dependencies (since pip has no concept of that yet, and setup_requires won't work in this case).

That being said, pypa/pip#4144 should hopefully land in the near future and allow setuptools to add a pyproject.toml file to itself that looks like:

[build-system]
requires = [
    "wheel",
    "packaging>=16.8",
    "six>=1.10.0",
    "appdirs>=1.4.0",
]

and pip will basically do what rwt setup.py would do, except with just those dependencies.

@jaraco
Copy link
Member Author

@jaraco jaraco commented Jan 17, 2017

Yes, thanks for that clarification and reference. The expectation is that users will typically install from wheels and anyone who wants to install from sdist will need to prep the environment in advance. Glad to hear that PEP 518 support is coming to pip.

@dstufft
Copy link
Member

@dstufft dstufft commented Jan 17, 2017

@jaraco I feel like this also paves the way for splitting pkg_resources out into its own package right? One that setuptools could depend on for both its own uses, and so everyne who currently used the setuptools == pkg_resources assumption in their dependency declaration is not broken, but start to allow people to depend on pkg_resources itself instead of all of setuptools?

jaraco added a commit that referenced this issue Jan 17, 2017
jaraco added a commit that referenced this issue Jan 23, 2017
…tall and signal that the bootstrap script is deprecated. Ref #581.
@jaraco
Copy link
Member Author

@jaraco jaraco commented Jan 23, 2017

As you can see ^ I've updated the bootstrap script to simply pin to the last version prior this new release, and that compatibility bootstrap script, while warning about the issue, should continue to support older environments.

joshmoore added a commit to joshmoore/openmicroscopy that referenced this issue Oct 30, 2017
Downloaded from https://bootstrap.pypa.io/ez_setup.py (which is now
deprecated) this corrects a failing build in travis. The overall
ez_setup/setuptools strategy will need to be reviewed, taking
pypa/setuptools#581 into account.
joshmoore added a commit to joshmoore/openmicroscopy that referenced this issue Nov 2, 2017
Downloaded from https://bootstrap.pypa.io/ez_setup.py (which is now
deprecated) this corrects a failing build in travis. The overall
ez_setup/setuptools strategy will need to be reviewed, taking
pypa/setuptools#581 into account.
sjperkins added a commit to ska-sa/montblanc that referenced this issue Nov 13, 2017
sjperkins added a commit to ska-sa/montblanc that referenced this issue Nov 13, 2017
* Add RIME op tensorflow source to MANIFEST.in

* Deprecate ez_setup.py script
  See pypa/setuptools#581.

* Include missing setup.cfg

* Pin python-casacore==2.1.2 for now
lpsinger added a commit to lpsinger/gwcelery that referenced this issue Feb 21, 2018
sjperkins added a commit to ska-sa/montblanc that referenced this issue Feb 23, 2018
* Add RIME op tensorflow source to MANIFEST.in

* Deprecate ez_setup.py script
  See pypa/setuptools#581.

* Include missing setup.cfg

* Pin python-casacore==2.1.2 for now
@jaraco
Copy link
Member Author

@jaraco jaraco commented Mar 17, 2018

Yes, one should do pip install ..

@Mirona
Copy link

@Mirona Mirona commented May 11, 2018

FYI, this completely messed up this tutorial and I don't know how to proceed: http://newcoder.io/begin/setup-your-machine/#windows

https://github.com/econchick/new-coder

How should the tutorial here be updated to reflect the changes? @econchik

image

image

image

@Mirona
Copy link

@Mirona Mirona commented May 11, 2018

Thank you @The-Compiler, was just looking there...

On this step: https://packaging.python.org/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date

image

There is an error:
image

Could not fetch URL https://pypi.python.org/simple/pip/: There was a problem confirming the ssl certificate: <urlopen
error [Errno 1] _ssl.c:504: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version>
Will skip URL https://pypi.python.org/simple/pip/ when looking for download links for pip in c:\python27\lib\site-pac
kages\pip-1.4.1-py2.7.egg

andersk added a commit to mit-scripts/scripts-pony that referenced this issue May 27, 2018
pypa/setuptools#581

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
joshmoore added a commit to ome/omero-dropbox that referenced this issue Aug 8, 2019
Downloaded from https://bootstrap.pypa.io/ez_setup.py (which is now
deprecated) this corrects a failing build in travis. The overall
ez_setup/setuptools strategy will need to be reviewed, taking
pypa/setuptools#581 into account.
guyer added a commit to guyer/fipy that referenced this issue Feb 7, 2020
The version here [generates deprecations](https://app.circleci.com/jobs/github/usnistgov/fipy/2031)
and [ez_setup.py itself is deprecated](pypa/setuptools#581)
guyer added a commit to guyer/fipy that referenced this issue Feb 10, 2020
The version here [generates deprecations](https://app.circleci.com/jobs/github/usnistgov/fipy/2031)
and [ez_setup.py itself is deprecated](pypa/setuptools#581)
guyer added a commit to usnistgov/fipy that referenced this issue Feb 14, 2020
* Raise errors for DeprecationWarnings during CI tests

* Remove ez_setup.py

The version here [generates deprecations](https://app.circleci.com/jobs/github/usnistgov/fipy/2031)
and [ez_setup.py itself is deprecated](pypa/setuptools#581)

* Change "FORTRAN" array ordering to 'F'

Missed in #707

* Update from deprecated Pysparse API

This stuff was deprecated [ages ago](https://sourceforge.net/p/pysparse/git/ci/241a05d7a6e5f0f8ae28c53ac79a2111fe2d7cc0/)

* Switch from itsolvers classes to krylov functions

The itsolvers classes aren't written right (e.g., `solve()` doesn't
return anything) and don't encapsulate anything we care about.
The `krylov` calls are set up the way we expect.

* Raise errors for DeprecationWarnings only in test suite

Raising errors for all of `python setup.py test` causes
errors for deprecations in things we don't control, like versioneer.

* Use legacy absolute tolerance for scipy Krylov solvers

Avoid DeprecationWarning

* Introduce specialized TestProgram to throw Deprecation errors

In Py3k, TestProgram makes it very hard to change DeprecationWarnings
into errors. [TestProgram gives all warnings an action of `default`](https://github.com/python/cpython/blob/master/Lib/unittest/main.py#L84)
which causes [TestRunner to clobber our DeprecationWarning filter](https://github.com/python/cpython/blob/master/Lib/unittest/runner.py#L167)
(passing any other action to TestProgram just causes that action to clobber instead)

* Force Py27 builds on Visual Studio 2015

scikit-fmm (and maybe others) needs VC++ 9.0
https://help.appveyor.com/discussions/problems/26278-organization-build-fails-because-vc-90-is-not-present

* Choose appropriate build environment

[Exclude unwanted build configurations](https://www.appveyor.com/docs/build-configuration/#exclude-configuration-from-the-matrix)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet