Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PLIP: get rid of bootstrap.py #2001

Closed
gforcada opened this issue Apr 10, 2017 · 21 comments

Comments

@gforcada
Copy link
Contributor

commented Apr 10, 2017

PLIP (Plone Improvement Proposal)

Responsible Persons

Proposer: Gil Forcada (@gforcada)

Seconder: Jens Klein (@jensens)

Abstract

bootstrap.py is a python module coming from zc.buildout that tries to create an isolated environment where zc.buildout can then install wherever you tell to build without affecting the global python environment.

This is nowadays better solved with virtualenv, if zc.buildout did not use that from the very beginning is mostly because it predates it 😄

Motivation

It is 2017 and python, virtualenv and zc.buildout have improved a lot on the meantime, so it's time to say farewell to bootstrap.py and warmly welcome and embrace virtualenv everywhere.

The problems with bootstrap.py are:

  • you either have to keep a local version in every project (i.e. check-in in your git repository) or that you have to manually download it from Internet (not reliable at all, no offline bootstrapping...) a version that you do not have control over
  • virtualenv is far better at providing a working isolated system where to run zc.buildout on
  • there is no big community around it, whereas there is a huge one with virtualenv, in fact, it is so cornerstone to Python that since Python 3.3 (released on September 29, 2012, +4 years ago) it has been an official module: venv

Boostraping a project with bootstrap.py usually means:

git clone XXX
cd XXX
python bootstrap.py
./bin/buildout

With virtualenv that would be:

git clone XXX
cd XXX
virtualenv . # on python 3 that would be: python3 -m venv .
bin/pip install zc.buildout setuptools
./bin/buildout

Assumptions

There is an agreement to remove bootstrap.py and encourage use of virtualenv everywhere.

This discussion has been brought quite a few times, but given that Plone is going through quite a lot of major changes already (newer Zope, Python 3, all the JS/CSS major changes on Plone 5.x) it seems to be a good time to introduce yet another change, that while tiny (one does not bootstrap a project every day) it can be very frustrating if it does not work.

Proposal & Implementation

  • update all references to bootstrap.py to point to virtualenv in our documentation, that, by far would be the biggest part of the implementation
  • remove a few bootstrap.py, update .travis.yml and update .gitignore in our git repositories (there are way too many to change them in one swell foop) and write a porting guide
  • update our Jenkins CI infrastructure to use virtualenvs as well

And extended proposal could be to create a small script that goes through all repositories in plone and collective github organizations and creates an issue if a bootstrap.py script is found, pointing to the porting guide mentioned above.

Deliverables

Pull requests to documentation repository with:

  • the bootstrap to virtualenv update
  • the porting guide
  • a basic introduction to virtualenv

Pull requests to a few repositories to test against the porting guide (candidate repositories welcome)

The script to go through all repositories could most probably be suited to mr.roboto where it could even be made a bit more clever and only create issues on repositories that do have activity (there should no need to ping repositories long dead on collective org).

Risks

The main risk is to keep using bootstrap.py 😄

The only risk would be the introduction of a new way to do things, though, virtualenv has been around since forever and every developer that knows bootstrap.py most probably already knows and has used virtualenv.

virtualenv is being used already by a lot of Plone contributors on their own workflows, so no major/minor problems would be expected due to the switch.

The only main concern would be to have a time window where to test the new Jenkins setup before going live with it, although personally (@gforcada) I already have a Jenkins setup where virtualenv is used.

Participants

@hvelarde

This comment has been minimized.

Copy link
Member

commented Apr 10, 2017

this has to be closed first: buildout/buildout#289

@jensens

This comment has been minimized.

Copy link
Member

commented Apr 10, 2017

I second this PLIP.

@gforcada

This comment has been minimized.

Copy link
Contributor Author

commented Apr 11, 2017

@hvelarde why that has to be closed? Sure it would be nice, but it is working already as it is now...

And on top of that nobody was against the idea on that issue either.

@hvelarde

This comment has been minimized.

Copy link
Member

commented Apr 11, 2017

because that should be a good indicator that everybody agrees on this and on how to do it.

@mauritsvanrees

This comment has been minimized.

Copy link
Member

commented Apr 11, 2017

I agree to remove bootstrap.py.

But then it would be good to have a requirements.txt instead, with pins for zc.buildout and setuptools. And with setuptools 34 or higher this needs more pins. Coredev 5.1 currently has this:

appdirs==1.4.2
packaging==16.8
pyparsing==2.1.10
setuptools==34.3.0
six==1.10.0
zc.buildout==2.8.0

It would be good to publish that file in dist.plone.org/release/xyz/.

(And maybe a constraints.txt with all Plone/Zope/ztk version pins, which can be used with pip install -c constraints.txt, but that is outside of the scope of this PLIP.)

@gforcada

This comment has been minimized.

Copy link
Contributor Author

commented Apr 11, 2017

@mauritsvanrees that sounds fine, but how are we supposed to distribute these two files? Should we mention them on the regular versions.cfg and also on the documentation?

About the files themselves: requirements.txt could easily be a copy of buildout.coredev's one and constraints.txt be generated out of versions.cfg more or less.

Then the bootstraping should be(?):

git clone XXX
cd XXX
wget dist.plone.org/release/5.1/requirements.txt
wget dist.plone.org/release/5.1/constraints.txt
virtualenv .
bin/pip install -r requirements.txt -c constraints.txt
bin/buildout
@mauritsvanrees

This comment has been minimized.

Copy link
Member

commented Apr 11, 2017

A requirements.txt would mostly be mentioned in documentation, although a link in a comment in versions.cfg could be useful too.

For bootstrapping a buildout, you would only need the requirements file:

wget dist.plone.org/release/5.1/requirements.txt
bin/pip install -r requirements.txt

The constraints file would just be there to make it easier for people to for example try installing Plone with pip:

bin/pip install -c constraints.txt Plone

If the constraints file contains all our pins, then the buildout bootstrapping line could actually just be this:

bin/pip install -c constraints.txt zc.buildout

I just checked: this also updates the installed setuptools version if that is in the constraints.

Ideally, buildout would be able to read a pip constraints or requirements file, so there is no duplication. Or it could be nice if it could output such a file.

If one or both files are on dist.plone.org, add-ons could download and use them in .travis.yml files to ease bootstrapping on various Plone versions.

@gforcada

This comment has been minimized.

Copy link
Contributor Author

commented Apr 11, 2017

@mauritsvanrees install plone with pip is still not supported though, right? Then, if I'm not mistaken, rather than a requirements.txt and constraints.txt, we only need constraints.txt, as one could only need to do:

git clone XXX
cd XXX
virtualenv .
wget dist.plone.org/xyz/constraints.txt
bin/pip install -c constraints.txt zc.buildout setuptools
bin/buildout
@gforcada

This comment has been minimized.

Copy link
Contributor Author

commented Apr 11, 2017

There was a discussion on making buildout read requirements.txt as well: buildout/buildout#287

@tomgross

This comment has been minimized.

Copy link
Member

commented Apr 18, 2017

I guess the .gitignore of all the relevant packages should be updated to exclude the virtualenv files and directories.

/include
/lib*
pip-selfcheck.json
/bin/activate*
/bin/python*
/bin/easy_install*
/bin/pip*
/bin/wheel
@tomgross

This comment has been minimized.

Copy link
Member

commented Apr 18, 2017

Addionatly the venv directories need to be excluded from coverage checks.

@jensens jensens changed the title PLIP: get rid of boostrap.py PLIP: get rid of bootstrap.py Apr 18, 2017

@idgserpro

This comment has been minimized.

Copy link
Member

commented Apr 24, 2017

Sorry if we're the only ones "complaining" about this PLIP, but we actually prefer buildout/buildout#289 and buildout/buildout#287 to settle down before doing this to all Plone packages, specially buildout/buildout#287 (comment), so we could try to make buildout/buildout#289 (comment) possible. Plone being able to be installed by pip would be the icying on the cake. @mauritsvanrees, did you really installed Plone using pip?

We agree on @hvelarde on this one #2001 (comment): we don't want to have

  • A buildout way;
  • A python traditional way;
  • A Plone way that's a mix of both.

See how it all started: just "add a requirements.txt with setuptools and zc.buildout", and now we need appdirs, packaging, pyparsing and six as well and who knows what dependency in the future will needed to be added by setuptools. So if you have an "old" requirements.txt in your repository, today, it will possibly break in the future anyway as it would an old bootstrap.py. We're even talking about a constraints.txt...

IMHO, this is not simplifying bootstraping, it's adding another layer of complexity mixing up buildout and pip infrastructure. The same experienced user that can handle all the virtualenv insanity can get away with a problematic bootstrap.py as well.

virtualenv is being used already by a lot of Plone contributors on their own workflows, so no major/minor problems would be expected due to the switch.

For experienced Plone developers, yes; For new users (we're getting some of them getting in touch in community.plone.org in the last months), maybe not.

Please, don't get us wrong on this message, and sorry if it sounds "rude" (we don't mean it): we just thought that a contrary opinion would be useful, at least for historical record (or we can even change our minds).

@hvelarde

This comment has been minimized.

Copy link
Member

commented Apr 24, 2017

exactly, we're going to get rid of one file (bootstrap.py) just to add 2 new files to every package (requirements.txt and constraints.txt) and this is going to be worse next time we find an issue.

besides that, what problem are we solving? all the isolation code was removed from Buildout 2 anyway.

it's too early: we have to wait for consensus on the Buildout world before moving on any direction.

@jensens

This comment has been minimized.

Copy link
Member

commented Apr 25, 2017

I know at least, that @do3cc has one 4.3 installed woth pip inside a docker container, which afaik was a PITA because the infrastructure is not there.

Don't get me wrong, this PLIP does not aim to be the full story but the beginning of one.

I know you folks would love always to get the full feature at once. It's usually not possible in a community driven world, except if members make it happen! In this case I prefer to have small steps working instead a vision nobody is able to invest time in. We can do this right now and then improve upon it.

@do3cc

This comment has been minimized.

Copy link
Member

commented Apr 25, 2017

It is true, I have a Plone 4.3 site that I install with pip only. It is not a pita for me, I am really happy with it.

But this proposal is only about replacing bootstrap.py and the docs with using pip with requirements.txt

There is only one reason that buildout/buildout#289 isn't closed. Changing the tests after the doc changes is a lot of work. I gave up after a few hours.

I saw two complaints here:

  1. you replace bootstrap.py with requirements.txt and it has the same problems of possibly being outdated. It is a bit easier to compare different requirement.txt files than different bootstrap.py files.
  2. We add two files, requirements.txt and constraints.txt and remove only one. These are just future ideas.

For me it looks like the only reason we are still having bootstrap.py is that nobody implemented buildout/buildout#289 for a year. Maybe add buildout/buildout#289 to the deliverables?

@hvelarde

This comment has been minimized.

Copy link
Member

commented Apr 25, 2017

you guys know more about this issue than me; if you say this is a step in the right direction I have to follow you even as is that not clear to me.

@idgserpro

This comment has been minimized.

Copy link
Member

commented Apr 25, 2017

exactly, we're going to get rid of one file (bootstrap.py) just to add 2 new files to every package (requirements.txt and constraints.txt) and this is going to be worse next time we find an issue.

you replace bootstrap.py with requirements.txt and it has the same problems of possibly being outdated. It is a bit easier to compare different requirement.txt files than different bootstrap.py files.

Using constraints.txt, if I want to debug something and play with dependencies and it's versions, I'll have to think: is this package installed with pip or buildout? Remove it from eggs or run a pip uninstall? Should I pin it in [versions]? BTW, a more important question we should be doing is, how do I know when I should be using requirements.txt or a buildout.cfg for adding my dependencies and pinnings?

The problem here is not just "they both become outdated", you have now two ways of doing the same thing. "Hey, don't forget to use virtualenv, so you can have an isolated buildout instance with your own polluted site-packages that you'll have to figure it out by yourself when you shoot yourself in the foot". In the beginning, we only had zc.buildout and setuptools, it seemed simple and requirements.txt was seemed as unharmful, but now that I see the path you're following it scares me a little bit.

Like @hvelarde said, we'll follow you, it doesn't mean we agree with that. 😆

@mauritsvanrees

This comment has been minimized.

Copy link
Member

commented Apr 27, 2017

@idgserpro wrote:

Plone being able to be installed by pip would be the icying on the cake. @mauritsvanrees, did you really installed Plone using pip?

I tried something with an experimental recipe, but ran into problems with z3c.autoinclude.

I would be in favour of using a requirements.txt instead of bootstrap.py. I agree that it is not nice that you need to keep requirements.txt and versions.cfg in sync. But with bootstrap.py you usually have the same problem: python bootstrap.py will install the latest zc.buildout and setuptools 33.1.1 (using the already deprecated https://bootstrap.pypa.io/ez_setup.py) instead of what you have in versions.cfg, so you need to do python bootstrap.py --buildout-version=BUILDOUT_VERSION --setuptools-version=SETUPTOOLS_VERSION.

I can image a buildout extension that reads pinned versions from requirements.txt and adds them to the already pinned versions:

[buildout]
extensions = buildout.extension.requirements
# default settings:
requirements-files =
    requirements.txt
    constraints.txt
requirements-override = false

By default this would not override versions from the versions section, but only add versions for packages that are not pinned yet. Changing that could be an option.

@gforcada

This comment has been minimized.

Copy link
Contributor Author

commented Apr 29, 2017

after spending the whole morning trying to set up a new worker in jenkins

sorry but bootstrap.py is totally broken with current buildout.coredev 5.1: bootstrap.py and latest setuptools and latest zc.buildout don't play together at all.

The same bootstraping problem I still haven't solved was just a 5 commands with a virtualenv and pip installing zc.buildout and setuptools.

Up until recently, as @mauritsvanrees mentions with the newer dependencies of setuptools, it was mostly possible to keep using bootstrap.py, but if you have a completely empty environment... good luck.

So, given that only @hvelarde and @idgserpro had some concerns but anyway said that they would follow what we suggest, I will try to put this PLIP on next @plone/framework-team meeting agenda.

gforcada added a commit to plone/buildout.coredev that referenced this issue Apr 29, 2017

Revert setuptools to 26.1.1
Anything prior to setuptools 33.1.1 should be fine.

Plone uses basic features of setuptools and as our Jenkins CI environment
is already pining 26.1.1 is easier to stick to this version.

As soon as PLIP 2001 plone/Products.CMFPlone#2001 is approved and merged,
we can happily move to virtualenv which solves the problem completely.

jensens added a commit to plone/buildout.coredev that referenced this issue May 2, 2017

Revert setuptools to 26.1.1
Anything prior to setuptools 33.1.1 should be fine.

Plone uses basic features of setuptools and as our Jenkins CI environment
is already pining 26.1.1 is easier to stick to this version.

As soon as PLIP 2001 plone/Products.CMFPlone#2001 is approved and merged,
we can happily move to virtualenv which solves the problem completely.

jensens added a commit to plone/buildout.coredev that referenced this issue May 2, 2017

Revert setuptools to 33.1.1
Anything prior to setuptools 33.1.1 should be fine.

Plone uses basic features of setuptools and as our Jenkins CI environment
is already pining 26.1.1 is easier to stick to this version.

As soon as PLIP 2001 plone/Products.CMFPlone#2001 is approved and merged,
we can happily move to virtualenv which solves the problem completely.
@hvelarde

This comment has been minimized.

Copy link
Member

commented Aug 28, 2017

@idgserpro idgserpro referenced this issue Sep 28, 2017

Merged

3.0 #217

@idgserpro idgserpro referenced this issue Mar 13, 2018

Open

WIP: remove bootstrap #895

0 of 7 tasks complete
@tisto

This comment has been minimized.

Copy link
Member

commented Feb 27, 2019

@gforcada @jensens can't we close that PLIP? I mean we do not use bootstrap.py anywhere, correct? Did anybody go through the docs to check that is all gone?

@jensens jensens closed this May 21, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.