Upgrade pip to 20.3.3 and virtualenv to 20.4.0 #98

amanda11 · 2021-01-25T23:35:59Z

Using pip 20.3.3 and legacy resolver

This reverts commit a9825a5.

arm4b · 2021-02-03T19:36:47Z

You can check the https://hub.docker.com/repository/registry-1.docker.io/stackstorm/buildpack/builds/f12894d9-f84b-4842-a2c5-7fea9e8abd7e build at Docker Hub.

Copy-paste error log:

Step 4/5 : RUN yum -y install python-devel && wget -qO - https://bootstrap.pypa.io/get-pip.py | python && pip install --upgrade "pip==20.3.3" && pip install setuptools virtualenv && rm -rf /root/.cache
---> Running in 55c806a1803a
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
* base: mirror.centos.org
* extras: mirror.centos.org
* updates: mirror.centos.org
Resolving Dependencies
--> Running transaction check
---> Package python-devel.x86_64 0:2.7.5-90.el7 will be installed
--> Processing Dependency: python(x86-64) = 2.7.5-90.el7 for package: python-devel-2.7.5-90.el7.x86_64
--> Processing Dependency: python2-rpm-macros > 3-30 for package: python-devel-2.7.5-90.el7.x86_64
--> Processing Dependency: python-rpm-macros > 3-30 for package: python-devel-2.7.5-90.el7.x86_64
--> Running transaction check
---> Package python.x86_64 0:2.7.5-89.el7 will be updated
---> Package python.x86_64 0:2.7.5-90.el7 will be an update
--> Processing Dependency: python-libs(x86-64) = 2.7.5-90.el7 for package: python-2.7.5-90.el7.x86_64
---> Package python-rpm-macros.noarch 0:3-34.el7 will be installed
---> Package python2-rpm-macros.noarch 0:3-34.el7 will be installed
--> Running transaction check
---> Package python-libs.x86_64 0:2.7.5-89.el7 will be updated
---> Package python-libs.x86_64 0:2.7.5-90.el7 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
python-devel x86_64 2.7.5-90.el7 updates 399 k
Installing for dependencies:
python-rpm-macros noarch 3-34.el7 base 9.1 k
python2-rpm-macros noarch 3-34.el7 base 8.1 k
Updating for dependencies:
python x86_64 2.7.5-90.el7 updates 96 k
python-libs x86_64 2.7.5-90.el7 updates 5.6 M
Transaction Summary
================================================================================
Install 1 Package (+2 Dependent packages)
Upgrade ( 2 Dependent packages)
Total download size: 6.1 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
--------------------------------------------------------------------------------
Total 13 MB/s | 6.1 MB 00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : python-libs-2.7.5-90.el7.x86_64 1/7
Updating : python-2.7.5-90.el7.x86_64 2/7
Installing : python-rpm-macros-3-34.el7.noarch 3/7
Installing : python2-rpm-macros-3-34.el7.noarch 4/7
Installing : python-devel-2.7.5-90.el7.x86_64 5/7
Cleanup : python-2.7.5-89.el7.x86_64 6/7
Cleanup : python-libs-2.7.5-89.el7.x86_64 7/7
Verifying : python2-rpm-macros-3-34.el7.noarch 1/7
Verifying : python-libs-2.7.5-90.el7.x86_64 2/7
Verifying : python-devel-2.7.5-90.el7.x86_64 3/7
Verifying : python-2.7.5-90.el7.x86_64 4/7
Verifying : python-rpm-macros-3-34.el7.noarch 5/7
Verifying : python-libs-2.7.5-89.el7.x86_64 6/7
Verifying : python-2.7.5-89.el7.x86_64 7/7
Installed:
python-devel.x86_64 0:2.7.5-90.el7
Dependency Installed:
python-rpm-macros.noarch 0:3-34.el7 python2-rpm-macros.noarch 0:3-34.el7
Dependency Updated:
python.x86_64 0:2.7.5-90.el7 python-libs.x86_64 0:2.7.5-90.el7
Complete!
�[91mTraceback (most recent call last):
�[0m
�[91m File "<stdin>", line 24244, in <module>
�[0m
�[91m File "<stdin>", line 199, in main
�[0m
�[91m File "<stdin>", line 82, in bootstrap
�[0m
�[91m File "�[0m
�[91m/tmp/tmpbsXUAQ/pip.zip/pip/_internal/cli/main.py�[0m
�[91m", line �[0m
�[91m60�[0m
�[91m
�[0m
�[91m �[0m
�[91msys.stderr.write(f"ERROR: {exc}")
�[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m �[0m
�[91m ^
�[0m
�[91mSyntaxError: invalid syntax
�[0m
Removing intermediate container 55c806a1803a
The command '/bin/sh -c yum -y install python-devel && wget -qO - https://bootstrap.pypa.io/get-pip.py | python && pip install --upgrade "pip==20.3.3" && pip install setuptools virtualenv && rm -rf /root/.cache' returned a non-zero code: 1

…lds)

amanda11 · 2021-02-04T10:03:16Z

Putting back to draft whilst we fix why still getting complaints about no-site-packages. Also may want to use pip 20.3.3 as it matches the version used by virtualenv.

buildpack/centos7/Dockerfile

cognifloyd

The buildpack images were removed in #101 (614b461 and c345407) but they were reintroduced as empty files in this merge commit: 973b39a So, please delete:

buildpack/*/Dockerfile
packagingbuild/Dockerfile.debian
packagingbuild/Dockerfile.template

amanda11 · 2021-03-11T21:40:49Z

The buildpack images were removed in #101 (614b461 and c345407) but they were reintroduced as empty files in this merge commit: 973b39a So, please delete:

buildpack/*/Dockerfile

packagingbuild/Dockerfile.debian

packagingbuild/Dockerfile.template

Thanks for picking that up - will fix

Kami · 2021-03-12T13:05:21Z

packagingbuild/bionic/Dockerfile

 RUN apt-get -y install \
        python-virtualenv python-setuptools python-mock && \
        apt-get clean && \
-        git clone --branch stackstorm_patched_use_custom_pip https://github.com/StackStorm/dh-virtualenv.git /tmp/dh-virtualenv && \
+        git clone --branch rm_site_packages_upgrade_pip https://github.com/StackStorm/dh-virtualenv.git /tmp/dh-virtualenv && \


Per my comment on Slack - please open PR with your changes against stackstorm_patched branch in dh-virtualenv repo and let's just use stackstorm_patched branch.

I assume you will update this line here when the other PR is merged, right?

@Kami Yes - updated now - to use stackstorm_patched.

Kami · 2021-03-12T13:06:01Z

packagingbuild/xenial/Dockerfile

+# This line just busts Docker's cache so it re-runs the next line
+# The GitHub API will return different results when the branch HEAD changes
+# See https://stackoverflow.com/a/39278224
+# ADD https://api.github.com/repos/StackStorm/dh-virtualenv/git/refs/heads/rm_site_packages_upgrade_pip version.json


One option would also to be pin to check out a specific commit revision in the git clone / checkout line

Kami

I approved this PR, but let's please also make the dh-virtualenv change mentioned in the comment.

cognifloyd · 2021-03-12T16:51:45Z

The buildpack images were removed in #101 (614b461 and c345407) but they were reintroduced as empty files in this merge commit: 973b39a So, please delete:

buildpack/*/Dockerfile

packagingbuild/Dockerfile.debian

packagingbuild/Dockerfile.template

Thanks for picking that up - will fix

You got rid of the buildpack empty files. 👍
Could you also remove the empty packagingbuild files?

amanda11 · 2021-03-12T16:57:16Z

The buildpack images were removed in #101 (614b461 and c345407) but they were reintroduced as empty files in this merge commit: 973b39a So, please delete:

buildpack/*/Dockerfile

packagingbuild/Dockerfile.debian

packagingbuild/Dockerfile.template

Thanks for picking that up - will fix

You got rid of the buildpack empty files.
Could you also remove the empty packagingbuild files?

Oops - missed them. Thanks - hopefully got them all now!

amanda11 · 2021-03-12T17:40:13Z

TODO: Update the PR to use stackstorm-patched branch after the dh-virtualenv PR has merged changes into stackstorm-patched branch. Will push up that change after other PR merged....

amanda11 · 2021-03-16T15:02:37Z

@armab Would you be able to re-review, as I believe your review comments have now been addressed. Thanks.

Kami

LGTM 👍

arm4b · 2021-03-16T19:19:23Z

packagingbuild/bionic/Dockerfile

+#
+# We use our dh-virtualenv version, since it fixes shebangd lines rewrites.
+# The shebangd rewrites are in master branch, but that requires dependency
+# on debhelper 12.


I think in the future it's OK to switch to debhelper 12.

From what I see we're on 10 right now: https://github.com/StackStorm/st2-packages/blob/master/packages/st2/debian/compat

When I tried on bionic in our images, the latest debhelper it could download was 11. I don't think 12 is in the bionic repos.

Makes sense, good to know!
So probably 10 for Ubuntu Xenial then.

arm4b

Looks good 👍

Thanks, everyone!

5060 upgrade virtualenv

1fd6017

amanda11 marked this pull request as draft January 25, 2021 23:36

Use pip 20.3.4

4ac9df2

blag mentioned this pull request Jan 28, 2021

Template re-sync and various other features and fixes #99

Closed

amanda11 mentioned this pull request Jan 28, 2021

Update fixate-requirements.py to cope with PIP 20.3.3 StackStorm/st2#5123

Merged

amanda11 added this to the 3.4.0 milestone Jan 28, 2021

Downgrade to pip 20.3.3 now using legacy resolver to match virtualenv

8b7da65

arm4b mentioned this pull request Jan 30, 2021

5060 upgrade virtualenv #97

Closed

amanda11 changed the title ~~[WIP] Upgrade pip to 20.3.4~~ [WIP] Upgrade pip to 20.3.3 and virtualenv to 20.4.0 Feb 1, 2021

amanda11 and others added 3 commits February 1, 2021 14:59

Move up to virtualenv 20.4.0

ba33dc9

Use upstream (Spotify) dh-virtualenv

a9825a5

Revert "Use upstream (Spotify) dh-virtualenv"

5b0f25d

This reverts commit a9825a5.

blag added 5 commits February 3, 2021 14:20

Use pip 20.3.4

2cf3b37

Use Python 3 to install pip on centos7

5daf01a

Use Docker's RUN directives as intended (this greatly speeds up rebui…

205f505

…lds)

Remove unnecessary whitespace

b2c0a46

Use Docker's RUN directives more like Docker intended

f2bd6b1

blag marked this pull request as ready for review February 4, 2021 00:29

blag requested a review from arm4b February 4, 2021 01:05

amanda11 marked this pull request as draft February 4, 2021 10:02

arm4b suggested changes Feb 4, 2021

View reviewed changes

buildpack/centos7/Dockerfile Outdated Show resolved Hide resolved

blag added 6 commits February 4, 2021 15:53

Back out some of the changes to the centos7 buildpack Dockerfile

aafe8aa

Improve Dockerfile formatting

61d1805

Use our updated fork of dh-virtualenv

91fd291

Merge branch 'master' into upgrade_pip

973b39a

Cleanup more whitespace

f0ca260

Merge branch 'master' into upgrade_pip

1938a47

Merge branch 'master' into upgrade_pip

85f0774

amanda11 marked this pull request as draft March 11, 2021 14:56

amanda11 marked this pull request as ready for review March 11, 2021 16:23

cognifloyd requested changes Mar 11, 2021

View reviewed changes

amanda11 added 2 commits March 12, 2021 09:01

Remove buildpack Dockerfiles

bec5709

Use branch with upgrade-pip-to backport

c676306

amanda11 mentioned this pull request Mar 12, 2021

Upgrade venv and pip StackStorm/st2-packages#684

Merged

Kami reviewed Mar 12, 2021

View reviewed changes

Kami approved these changes Mar 12, 2021

View reviewed changes

Missed empty files needed to remove

5b5d0d0

cognifloyd approved these changes Mar 12, 2021

View reviewed changes

amanda11 marked this pull request as draft March 12, 2021 17:40

amanda11 marked this pull request as ready for review March 16, 2021 09:27

amanda11 marked this pull request as draft March 16, 2021 09:28

Use stackstorm-patched branch

3b21c8f

amanda11 marked this pull request as ready for review March 16, 2021 15:01

Kami approved these changes Mar 16, 2021

View reviewed changes

arm4b reviewed Mar 16, 2021

View reviewed changes

arm4b approved these changes Mar 16, 2021

View reviewed changes

amanda11 merged commit baf9e8f into master Mar 16, 2021

amanda11 deleted the upgrade_pip branch March 16, 2021 20:06

amanda11 added the feature label Mar 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade pip to 20.3.3 and virtualenv to 20.4.0 #98

Upgrade pip to 20.3.3 and virtualenv to 20.4.0 #98

amanda11 commented Jan 25, 2021 •

edited

Loading

arm4b commented Feb 3, 2021 •

edited

Loading

amanda11 commented Feb 4, 2021

cognifloyd left a comment

amanda11 commented Mar 11, 2021

Kami Mar 12, 2021 •

edited

Loading

Kami Mar 16, 2021

amanda11 Mar 16, 2021

Kami Mar 12, 2021

Kami left a comment

cognifloyd commented Mar 12, 2021

amanda11 commented Mar 12, 2021

amanda11 commented Mar 12, 2021

amanda11 commented Mar 16, 2021

Kami left a comment

arm4b Mar 16, 2021

amanda11 Mar 16, 2021

arm4b Mar 16, 2021

arm4b left a comment

Upgrade pip to 20.3.3 and virtualenv to 20.4.0 #98

Upgrade pip to 20.3.3 and virtualenv to 20.4.0 #98

Conversation

amanda11 commented Jan 25, 2021 • edited Loading

arm4b commented Feb 3, 2021 • edited Loading

amanda11 commented Feb 4, 2021

cognifloyd left a comment

Choose a reason for hiding this comment

amanda11 commented Mar 11, 2021

Kami Mar 12, 2021 • edited Loading

Choose a reason for hiding this comment

Kami Mar 16, 2021

Choose a reason for hiding this comment

amanda11 Mar 16, 2021

Choose a reason for hiding this comment

Kami Mar 12, 2021

Choose a reason for hiding this comment

Kami left a comment

Choose a reason for hiding this comment

cognifloyd commented Mar 12, 2021

amanda11 commented Mar 12, 2021

amanda11 commented Mar 12, 2021

amanda11 commented Mar 16, 2021

Kami left a comment

Choose a reason for hiding this comment

arm4b Mar 16, 2021

Choose a reason for hiding this comment

amanda11 Mar 16, 2021

Choose a reason for hiding this comment

arm4b Mar 16, 2021

Choose a reason for hiding this comment

arm4b left a comment

Choose a reason for hiding this comment

amanda11 commented Jan 25, 2021 •

edited

Loading

arm4b commented Feb 3, 2021 •

edited

Loading

Kami Mar 12, 2021 •

edited

Loading