Don't install packages that could mess packaging up #397

kennethreitz · 2017-05-30T16:12:41Z

No description provided.

cclauss · 2017-05-30T16:35:42Z

vendor/pip-pop/pip-clean

+def main():
+    args = docopt(__doc__, version='pip-grep')
+    req_file = args['<req-file>']
+


Could this be streamlined?

def good_package(line): package_name = line.split('=')[0].split('<')[0].split('>')[0] return package_name not in BAD_PACKAGES lines = [line for line in f if good_package(line)]

I want to do as little parsing as possible, for fear of breaking anything with crazy requirements files.

Understood... see other package names that begin with but do not end with six for instance https://pypi.python.org/pypi?%3Aaction=search&term=Six&submit=search

I was planning to cross that bridge when I came to it, which I honestly doubt would happen any time soon. I'll see what I can do though.

@cclauss i updated the code; integrated what you suggested and made a few improvements

hone · 2017-05-30T17:07:41Z

Should we have a message that tells people we're replacing stuff in their requirements.txt with things in the Python buildpack or overwriting their requirements.txt file at all.

Also, how does this affect pipenv?

kennethreitz · 2017-05-30T17:08:15Z

Pipenv has this functionality built-in.

I don't think a message is needed, but I could add one saying "Removing packaging libraries from requirements.txt".

kennethreitz · 2017-05-30T17:08:44Z

In my mind, if someone has any of these listed in their requirements.txt, it is accidental, not intentional.

kennethreitz · 2017-05-30T17:09:01Z

e.g. they don't understand what's going on. so, i don't want to overload them with information.

hone · 2017-05-30T17:09:36Z

@kennethreitz is there a place you could link them to that would help?

kennethreitz · 2017-05-30T17:17:41Z

No.

kennethreitz · 2017-05-30T17:18:33Z

This is a new problem, and one that is temporary and will go away soon (but we need to have this code for people using this version range of setuptools locally for the future).

It's not well-understood or documented anywhere that I know of.

kennethreitz · 2017-05-30T17:30:38Z

Waiting, waiting, waiting... for CI to pass.

jayvdb · 2017-05-30T17:39:49Z

bin/steps/setuptools

+# Syntax sugar.
+source $BIN_DIR/utils
+
+pip-clean requirements.txt


missing EOL

jayvdb · 2017-05-30T17:44:16Z

vendor/pip-pop/pip-clean

+    return package_name not in BAD_PACKAGES
+
+def main():
+    args = docopt(__doc__, version='pip-grep')


version='pip-grep'

jayvdb · 2017-05-30T17:55:06Z

In my mind, if someone has any of these listed in their requirements.txt, it is accidental, not intentional.
e.g. they don't understand what's going on. so, i don't want to overload them with information.

The more usual explanation is they have put them in because they have minimum and maximum requirements.
setuptools installs pkg_resources which is the main mechanism people use for fetching plugins (implicit namespaces is the newer alternative).
And more importantly, because of the fact that pip doesnt have a proper resolver, sometimes it is necessary to mention one of these packages in requirements.txt so that pip follows a particular path in its resolution strategy. You remove one of them of requirements.txt, and pip behaves differently when resolving indirect dependencies many levels deep.
At the very least, there should be a note to say why the deploy is using different versions from the versions they see when they deploy the app somewhere else.

kennethreitz · 2017-05-30T19:20:18Z

Related: #393

cclauss · 2017-05-30T18:23:30Z

vendor/pip-pop/pip-clean

+
+
+def good_package(line):
+    package_name = line.split('=')[0].split('<')[0].split('>')[0].split(' ')[0].split('#')[0].split('\n')[0]


You can just do .split() with no parameters instead of .split(' ').split('\n') because it deals with all whitespaces. Given the for line in f, \n is not really an issue here.

\n is included in each line

Buggy. c.f. #400

edmorley · 2017-06-01T17:34:18Z

This is a new problem, and one that is temporary and will go away soon

Was this to work around setuptools unvendoring their dependencies in v34.0.0 (which they've then reverted in v36.0.0) and relying on install_requires instead? (Which presumably could cause issues if the versions they required conflicted with those in requirements.txt?)

kennethreitz · 2017-06-01T20:25:37Z

@edmorley yes

This reverts commit 5496c02.

* Revert "Fix pyyaml (#402)" This reverts commit ff94908. * Revert "Don't install packages that could mess packaging up (#397)" This reverts commit 5496c02.

The pip-uninstall step stopped working when it was moved to after the pip-install step in f27a84e. This regression was temporarily fixed by part of #397, however that PR was reverted in #404. Adds a test to hopefully catch any future regressions :-) Fixes #393.

kennethreitz added 6 commits May 30, 2017 11:12

updated changelog

52758c3

remove setuptools sanity check

3544a4c

update changelog

3a8e27c

Merge branch 'master' into setuptools

9e026ba

pip-clean

4ce42fe

changelog note

2cc1d0d

kennethreitz mentioned this pull request May 30, 2017

How to bootstrap setuptools from source pypa/setuptools#980

Closed

cclauss reviewed May 30, 2017

View reviewed changes

kennethreitz requested review from jmorrell, hone and dzuelke May 30, 2017 16:56

hone approved these changes May 30, 2017

View reviewed changes

refactor codebase to improve package name detection

5ea0382

jayvdb reviewed May 30, 2017

View reviewed changes

kennethreitz added 2 commits May 30, 2017 13:45

fix version string

24f670e

cleanup

2a5c6f5

kennethreitz added 2 commits May 30, 2017 14:04

add messaging, improve execution

b8f8f5b

do uninstall first

1ede61e

cclauss reviewed May 30, 2017

View reviewed changes

Merge branch 'master' into setuptools

dafc912

kennethreitz merged commit 5496c02 into master May 31, 2017

kennethreitz deleted the setuptools branch May 31, 2017 00:04

kennethreitz added a commit that referenced this pull request Jun 3, 2017

Revert "Don't install packages that could mess packaging up (#397)"

017e0a7

This reverts commit 5496c02.

kennethreitz mentioned this pull request Jun 3, 2017

Revert "Don't install packages that could mess packaging up" #404

Merged

This was referenced Jun 13, 2017

Heroku-16 Python runtime archives contain additional packages (boto/bob/...) #379

Closed

Fix pip uninstall by moving it prior to pip install #413

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't install packages that could mess packaging up #397

Don't install packages that could mess packaging up #397

kennethreitz commented May 30, 2017

cclauss May 30, 2017

kennethreitz May 30, 2017 •

edited

cclauss May 30, 2017

kennethreitz May 30, 2017

kennethreitz May 30, 2017

hone commented May 30, 2017

kennethreitz commented May 30, 2017

kennethreitz commented May 30, 2017

kennethreitz commented May 30, 2017

hone commented May 30, 2017

kennethreitz commented May 30, 2017

kennethreitz commented May 30, 2017 •

edited

kennethreitz commented May 30, 2017

jayvdb May 30, 2017

kennethreitz May 30, 2017

jayvdb May 30, 2017

kennethreitz May 30, 2017

jayvdb commented May 30, 2017

kennethreitz commented May 30, 2017

cclauss May 30, 2017

kennethreitz May 30, 2017 •

edited

jayvdb May 31, 2017

edmorley commented Jun 1, 2017

kennethreitz commented Jun 1, 2017



		def good_package(line):
		package_name = line.split('=')[0].split('<')[0].split('>')[0].split(' ')[0].split('#')[0].split('\n')[0]

Don't install packages that could mess packaging up #397

Don't install packages that could mess packaging up #397

Conversation

kennethreitz commented May 30, 2017

Choose a reason for hiding this comment

kennethreitz May 30, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hone commented May 30, 2017

kennethreitz commented May 30, 2017

kennethreitz commented May 30, 2017

kennethreitz commented May 30, 2017

hone commented May 30, 2017

kennethreitz commented May 30, 2017

kennethreitz commented May 30, 2017 • edited

kennethreitz commented May 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jayvdb commented May 30, 2017

kennethreitz commented May 30, 2017

Choose a reason for hiding this comment

kennethreitz May 30, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edmorley commented Jun 1, 2017

kennethreitz commented Jun 1, 2017

kennethreitz May 30, 2017 •

edited

kennethreitz commented May 30, 2017 •

edited

kennethreitz May 30, 2017 •

edited