update to bootstrap.py

[papachoco]

setuptools 34.0.0 just introduced 3 new deps and deprecated the ez_setup.py (pinned to 33.1.1). I am not sure if bootstrap.py can be updated to reflect the setuptools upgrade.

Carlos

[anton-tagunov]

+1
setuptools brake buildout every months :-(

[idgserpro]

We had to pin it to 33.1.1 in bootstrap-buildout.py (thank god we have --setuptools-version argument) to make buildout work in some situations in the Plone world after this new setuptools release. We still don't have time to fully debug it. :(

[mauritsvanrees]

The PyPA people did warn us about it and asked our opinion in buildout issue #306.

Meanwhile https://bootstrap.pypa.io/ez_setup.py (which is used by our bootstrap-buildout.py) has been updated to use setuptools 33.1.1 by default. So a buildout bootstrap still works. And it will continue working unless they remove ez_setup.py. That might happen at some point.

The deprecation warning that ez_setup.py gives is: "ez_setup.py is deprecated; use pip to install setuptools". I think that is one more reason to give up on our bootstrap.py (issue #289) and recommend to use a virtualenv and pip install zc.buildout, probably with a requirements.txt.

For the record, pip freeze currently says this with the latest versions:

$ pip freeze
appdirs==1.4.0
packaging==16.8
pyparsing==2.1.10
six==1.10.0
zc.buildout==2.5.3

[idgserpro]

So a buildout bootstrap still works

We had an installation that still needed to use the --setuptools-version parameter to make it work. Maybe it was caching when downloading ez_setup.py, who knows. Didn't have the time look for it though, so thanks for this clarification.

I wasn't a big fan of giving up on bootstrap.py but now I see we don't have a choice.

[Cykooz]

use a virtualenv and pip install zc.buildout, probably with a requirements.txt

Oh no. It is the last thing I want to do. Why then need a buildout if in most cases enough only "venv + pip + requirements.txt"?

I like a buildout because it requires only two command to prepare an environment for a project:

python bootstrap.py
./bin/buildout

[leorochael]

We could always write a new bootstrap that uses get-pip.py to install zc.buildout directly instead of relying on ez_setup.

All that is necessary is to do the Python equivalent of this shell snippet:

python <( curl https://bootstrap.pypa.io/get-pip.py ) -t lib zc.builout
PYTHONPATH=lib python -c "import site; site.addsitedir('lib'); from zc.buildout.buildout import main; main()" bootstrap

of course, the buildout bootstrap above relies on the lib directory being on sys.path and also being on site.addsitedir(...) so we would have to fix the buildout bootstrap command to make truly "stand-alone" buildouts (i.e. with all recursive dependencies properly installed in the eggs directory).

Another tweak to the above sequence is to create a local virtualenv and use that to install buildout. Again, imagine the python equivalent of:

python <( curl https://bootstrap.pypa.io/get-pip.py ) -t tmp virtualenv
PYTHONPATH=tmp python -m virtualenv .
bin/pip install zc.buildout
rm -rf tmp

A new bootstrap script that did just the above would work right now, with no changes to buildout.

The possibilities are endless. There's no need to give up on buildout bootstrapping.

[mauritsvanrees]

Quoting @Cykooz:

Oh no. It is the last thing I want to do. Why then need a buildout if in most cases enough only "venv + pip + requirements.txt"?

If you only need to install some packages, then virtualenv, pip, and a requirements file are the simplest way, and probably faster. In such a case I would only use buildout because I am used to it.

For me the extra value that buildout gives, is in what it allows the recipes to do: running commands, creating custom scripts, creating config files from templates, etcetera.

[mauritsvanrees]

Quoting @leorochael:

We could always write a new bootstrap that uses get-pip.py to install zc.buildout directly instead of relying on ez_setup.

Such a bootstrap should use python -m ensurepip if available.

The possibilities are endless. There's no need to give up on buildout bootstrapping.

Sure, there are possibilities. And let me not create stop energy for anyone wanting to improve the bootstrapping. bootstrap.py has been very helpful.
But virtualenv and pip are well known for Python folks and may offer new buildout users less surprises.

[idgserpro]

If you only need to install some packages, then virtualenv, pip, and a requirements file are the simplest way, and probably faster. In such a case I would only use buildout because I am used to it.

Indeed. Recent experimental tools like pipenv are trying to make this whole workflow more intuitive and simple (it's debatable since it's another layer, but...).

[idgserpro]

FYI @hvelarde

Plone installations that have somehow setuptools = in it's buildout to force the last release of setuptools will have problems.

[hvelarde]

@idgserpro thanks! I already stumbled upon this.

[jimfulton]

It looks like this was fixed by a newer setuptools version.

Does anyone disagree that this can be closed?

[Cykooz]

It looks like this was fixed by a newer setuptools version.

Does anyone disagree that this can be closed?

bootstarp.py is not a main problem. The main problem is that buildout not work with setuptools 34+. Because buildout suggests that setuptools does not require other packages. For example the code form zc/buildout/easy_install.py:

runsetup_template = """
import sys
sys.path.insert(0, %(setupdir)r)
sys.path.insert(0, %(setuptools)r)

import os, setuptools

__file__ = %(__file__)r

os.chdir(%(setupdir)r)
sys.argv[0] = %(setup)r

with open(%(setup)r, 'U') as f:
    exec(compile(f.read(), %(setup)r, 'exec'))
"""

This code adds into sys.path only path for setuptools but not for dependencies of setuptools.

[jimfulton]

I see. Thanks.

So there seem to be some straightforward fixes:

a) Require setuptools < 34. This is probably a quick fix in the short term.

This should be quick. I'll try to whip this up shortly.

b) Change buildout to take into account that setuptools has dependencies.

[jimfulton]

Well, a) won't work. There's something stupid going on. When an old buildout tries to upgrade, it gets the new buildout, but fails to take its requirement into account when it gets setuptools. :(
This looks like a separate bug that needs to be investigated.

I'll explore plan b tomorrow.

[leorochael]

b) Change buildout to take into account that setuptools has dependencies.

If you look into that, do you mind also looking at #312? Should be in the same place in the code.

[jimfulton]

Yes, I mind. This fix is urgent.

[jimfulton]

#323

[jimfulton]

Is anyone going to review #323?

[leorochael]

Yes, I mind. This fix is urgent.

Well, sure, this symptom is a lot more urgent, but this issue and #312 are just symptoms of the same underlying bug: the buildout upgrade process and the buildout bootstrap process both make wrong assumptions about the (recursive) dependencies of buildout itself and where they are located in the file system.

Fixing them both at the same time should take the same amount of time and energy than fixing just one of them and from a quick look at #323 it might have fixed #312 accidentally. It'd be a shame if it didn't...

Is anyone going to review #323?

I'll try to take a look at it tomorrow

[jimfulton]

This fix has nothing to do with the bootstrapping process, which wasn't broken (for now).

I'm not sure I'll wait till tomorrow.

BTW, I'm working on a fix (tests) for Python 3.6.

[jimfulton]

Released zc.buildout-2.6.0

[papachoco]

Thanks a lot Jim... a minot issue zc.buildout 2.6.0 is writing itself (as well as setuptools ) to the version.cfg file after bootstraping even though they are already in the file.

Carlos

[jimfulton]

Yup: #326

[jimfulton]

Fixed released in 2.7.1

[papachoco]

Thanks a lot Jim

Carlos

[idgserpro]

The PyPA people did warn us about it and asked our opinion in buildout issue #306.

@mauritsvanrees we still think there's room for improvement at least for more documentation for people that don't follow setuptools/buildout development and just download bootstrap.py and expect it to work.

Usually when someone is using bootstrap.py and the buildout breaks (specially after a new setuptools release), one of the normal actions is to download latest bootstrap from https://bootstrap.pypa.io/bootstrap-buildout.py and run it again. Can we have a message like this:

Warning: 33.1.1 release that is being downloaded is the last setuptools version that gives support to ez_setup.py, which buildout uses for bootstraping. You may have to pin setuptools to 33.1.1 in your buildout.cfg or equivalent to avoid erros when running bin/buildout. Check pypa/setuptools#581 for more info.

In bootstrap.py? We would do a PR but we still don't have a signed contributor's agreement yet, so that's why we're asking for help.

And don't forget to change

buildout/bootstrap/bootstrap.py

Line 28 in 9ba1e31

__version__ = '2015-07-01'

as well. :)

[jimfulton]

This doesn't make sense to me.

This case wasn't a bootstrapping failure.

Any existing buildout that upgraded was also broken.

Perhaps we need an option to disable automatic upgrades. And/or perhaps it would be interesting to provide a means to downgrade to the state before an upgrade.

[idgserpro]

This case wasn't a bootstrapping failure.

Indeed. We've addressed the message in ez_setup.py itself.

And/or perhaps it would be interesting to provide a means to downgrade to the state before an upgrade.

Is it feasible to create a zip/tar.gz of bin and other files created by earlier calls of bootstrap and bin/buildout with a timestamp everytime you run python bootstrap.py, in a pre-defined directory (like buildout_baks)? Or should this be handled by the user himself/herself before running buildout?

[jimfulton]

That sounds pretty heavy handed.

I was simply pondering remembering the previous versions of buildout and setuptools and having an option to downgrade to the previous versions. If this was invoked, it would likely imply somehow recording the versions in the versions section.

It might be simpler to just disable automatic upgrading, which hasn't been the most popular feature. :) Not automatically upgrading when buildout and setuptools aren't pinned violates repeatability, although hopefully buildout and setuptools are stable enough at this point that that doesn't matter so much.