Update pip to 9.0.1 #23615

koffie · 2017-08-13T17:14:21Z

At #20913 pip was patched so that it works without ssl support, and this patch was also submitted upstream. Upstream has since merged the patch at pypa/pip#1165 into 9.0.1. So it makes sense to upgrade pip to this new version in order to have an unpatched pip in sage.

The pip tarbal can be found at https://pypi.python.org/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz#md5=35f01da33009719497f01a4ba69d63c9

Component: packages: standard

Author: Maarten Derickx

Branch: b6b9954

Reviewer: Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/23615

The text was updated successfully, but these errors were encountered:

koffie · 2017-08-13T19:06:15Z

Commit: 84ecb82

koffie · 2017-08-13T19:06:15Z

Branch: u/mderickx/23615

koffie · 2017-08-13T19:06:15Z

New commits:

`84ecb82`	`Upgrade pip to 9.0.1 trac #23615`

sagetrac-git · 2017-08-14T04:15:55Z

Changed commit from 84ecb82 to a8d1697

sagetrac-git · 2017-08-14T04:15:55Z

Branch pushed to git repo; I updated commit sha1. New commits:

`a8d1697`	`Removed use of decrepated pip list format: trac 23615`

jdemeyer · 2017-08-14T12:31:41Z

comment:5

This looks very scary: eval(stdout). If you want to parse JSON, I would recommend the json module.

sagetrac-git · 2017-08-14T16:26:41Z

Branch pushed to git repo; I updated commit sha1. New commits:

`b6b9954`	`use json for parsing json instead of eval: trac 23615`

sagetrac-git · 2017-08-14T16:26:41Z

Changed commit from a8d1697 to b6b9954

koffie · 2017-08-14T16:28:25Z

comment:7

Yeah, I shouldn't have written that. Changed it into json.loads(stdout).

jhpalmieri · 2017-08-14T18:23:39Z

comment:8

Builds fine and passes tests for me on OS X. Note that on this machine, the python2 log says that ssl is not built:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _ssl               dl              
gdbm               imageop            linuxaudiodev   
ossaudiodev        spwd               sunaudiodev

and import ssl raises an ImportError. So it's good that this new pip works.

jdemeyer · 2017-08-15T13:07:46Z

comment:9

Replying to @jhpalmieri:

Builds fine and passes tests for me on OS X.

Build from scratch or just a regular upgrade? I'm going to test a build from scratch on Linux.

jhpalmieri · 2017-08-15T14:36:21Z

comment:10

Replying to @jdemeyer:

Replying to @jhpalmieri:

Builds fine and passes tests for me on OS X.

Build from scratch or just a regular upgrade? I'm going to test a build from scratch on Linux.

Build from scratch.

jdemeyer · 2017-08-23T08:18:01Z

Reviewer: Jeroen Demeyer

vbraun · 2017-09-02T09:33:59Z

Changed branch from u/mderickx/23615 to b6b9954

fchapoton · 2017-09-10T07:08:34Z

comment:13

It seems that this breaks something with python3... :(

chapoton@icj-laptop:~/sage3$ ./sage -br
cd . && export                                    \
    SAGE_ROOT=/doesnotexist                               \
    SAGE_SRC=/doesnotexist                                \
    SAGE_SRC_ROOT=/doesnotexist                           \
    SAGE_DOC_SRC=/doesnotexist                            \
    SAGE_BUILD_DIR=/doesnotexist                          \
    SAGE_PKGS=/home/chapoton/sage3/build/pkgs                \
    SAGE_CYTHONIZED=/home/chapoton/sage3/src/build/cythonized      \
&& sage-python23 -u setup.py --no-user-cfg build install

Usage:   
  pip list [options]

no such option: --format
************************************************************************
Traceback (most recent call last):
  File "setup.py", line 69, in <module>
    from module_list import ext_modules, library_order, aliases
  File "/home/chapoton/sage3/src/module_list.py", line 166, in <module>
    from sage_setup.optional_extension import OptionalExtension
  File "/home/chapoton/sage3/src/sage_setup/optional_extension.py", line 24, in <module>
    all_packages = list_packages(local=True)
  File "/home/chapoton/sage3/src/sage/misc/package.py", line 226, in list_packages
    installed = installed_packages(exclude_pip)
  File "/home/chapoton/sage3/src/sage/misc/package.py", line 286, in installed_packages
    installed.update(pip_installed_packages())
  File "/home/chapoton/sage3/src/sage/misc/package.py", line 148, in pip_installed_packages
    return {package['name'].lower():package['version'] for package in json.loads(stdout)}
  File "/home/chapoton/sage3/local/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/home/chapoton/sage3/local/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/chapoton/sage3/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
************************************************************************
Error building the Sage library
************************************************************************
Makefile:34 : la recette pour la cible « sage » a échouée
make: *** [sage] Erreur 1

fchapoton · 2017-09-10T07:08:34Z

Changed commit from b6b9954 to none

fchapoton · 2017-09-10T07:09:21Z

comment:14

oh, wait a moment. I did not rebuild..

fchapoton · 2017-09-10T09:55:55Z

comment:15

Apparently, this really did break the python3 build. I am not amused.

koffie · 2017-09-10T10:57:11Z

comment:16

Hi Chapoton,

I would find it weird if this really did break the python3 build. The error message you posted is one that one would get if the installed pip is not actually 9.0.1:

For example on my machine with sage before this was merged

Maartens-MacBook-Pro:sagedev mderickx$ sage -pip -V
pip 8.1.2 from /Applications/sage/local/lib/python2.7/site-packages (python 2.7)
Maartens-MacBook-Pro:sagedev mderickx$ sage -pip list --format json

Usage:   
  pip list [options]

no such option: --format

and after it is merged

Maartens-MacBook-Pro:sagedev mderickx$ ./sage -pip -V
pip 9.0.1 from /Applications/sagedev/local/lib/python2.7/site-packages (python 2.7)
Maartens-MacBook-Pro:sagedev mderickx$ ./sage -pip list --format json
[{"version": "0.7.8", "name": "alabaster"}, ..., {"version": "4.2.0", "name": "zope.interface"}]

Could you give me the output of sage -pip -V and ls $SAGE_ROOT/upstream | grep pip?

koffie · 2017-09-10T11:02:18Z

comment:17

A quick solution would be:

sage -pip uninstall pip
sage -i pip

This avoids doing:

make distclean
make install

fchapoton · 2017-09-10T11:07:35Z

comment:18

chapoton@icj-laptop:~/sage3$ ./sage -pip -V
pip 9.0.1 from /home/chapoton/sage3/local/lib/python2.7/site-packages (python 2.7)
chapoton@icj-laptop:~/sage3$ ls upstream/pip*
upstream/pip-8.1.2.tar.gz  upstream/pip-9.0.1.tar.gz
chapoton@icj-laptop:~/sage3$ ./sage -pip list --format json
[{"version": "1.0.0", "name": "cypari2"}, {"version": "1.6.5", "name": "cysignals"}, {"version": "0.26", "name": "Cython"}, {"version": "9.0.1", "name": "pip"}, {"version": "33.1.1", "name": "setuptools"}]

so maybe this is a matter of pip versus pip3 ?

EDIT: in a sage shell, pip3 says version 9.0.1 too.

koffie · 2017-09-10T11:28:09Z

comment:19

Weird, then the only possibility is that sage is picking up some pip from outside its installation.

Could you add the lines

proc = subprocess.Popen(["pip", "-V"], stdout=subprocess.PIPE)
stdout = str(proc.communicate()[0])
print(stdout)

before the lines:

proc = subprocess.Popen(["pip", "list", "--no-index", "--format", "json"], stdout=subprocess.PIPE)
     stdout = str(proc.communicate()[0])

in src/sage/misc/package.py in order to see which pip is being picked up at the point where the build fails?

Alternatively could you describe how to reproduce this error so I can look at it myself?

fchapoton · 2017-09-10T11:31:48Z

comment:20

The result of the added print is

b'pip 9.0.1 from /home/chapoton/sage3/local/lib/python2.7/site-packages (python 2.7)\n'

To reproduce, I think this is enough:

- git clone a new sage
- export SAGE_PYTHON3=yes
- make build

koffie · 2017-09-10T12:31:46Z

comment:21

Ah I see the problem it is a unicode thing. If you do:

stdout = proc.communicate()[0].decode()

then everything should work. I will create a new ticket with a fix.

koffie · 2017-09-10T13:07:55Z

comment:22

See #23822

fchapoton · 2018-08-16T10:14:57Z

comment:23

I am hurt by all this again when going from sage 8.4.beta0 to 8.4.beta1.

jdemeyer · 2018-08-16T10:18:04Z

comment:24

Hurt by what again?

fchapoton · 2018-08-16T10:45:52Z

comment:25

I got build failures for sagelib on the very same line

return {package['name'].lower():package['version'] for package in json.loads(stdout)}

but I have now realized that this is probably due to the existence of a personal "pip.conf".

koffie added this to the sage-8.1 milestone Aug 13, 2017

koffie added c: packages: standard labels Aug 13, 2017

This comment has been minimized.

Sign in to view

koffie added the s: needs review label Aug 14, 2017

jdemeyer added s: needs work and removed s: needs review labels Aug 14, 2017

koffie added s: needs review and removed s: needs work labels Aug 14, 2017

jdemeyer added s: positive review and removed s: needs review labels Aug 23, 2017

vbraun removed the s: positive review label Sep 2, 2017

vbraun closed this as completed in 6ecf3e3 Sep 2, 2017

koffie mentioned this issue Sep 20, 2017

Python3 build error: unicode problem in pip_installed_packages() #23822

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update pip to 9.0.1 #23615

Update pip to 9.0.1 #23615

koffie commented Aug 13, 2017

This comment has been minimized.

koffie commented Aug 13, 2017

koffie commented Aug 13, 2017

koffie commented Aug 13, 2017

sagetrac-git mannequin commented Aug 14, 2017

sagetrac-git mannequin commented Aug 14, 2017

jdemeyer commented Aug 14, 2017

sagetrac-git mannequin commented Aug 14, 2017

sagetrac-git mannequin commented Aug 14, 2017

koffie commented Aug 14, 2017

jhpalmieri commented Aug 14, 2017

jdemeyer commented Aug 15, 2017

jhpalmieri commented Aug 15, 2017

jdemeyer commented Aug 23, 2017

vbraun commented Sep 2, 2017

fchapoton commented Sep 10, 2017

fchapoton commented Sep 10, 2017

fchapoton commented Sep 10, 2017

fchapoton commented Sep 10, 2017

koffie commented Sep 10, 2017

koffie commented Sep 10, 2017

fchapoton commented Sep 10, 2017

koffie commented Sep 10, 2017

fchapoton commented Sep 10, 2017

koffie commented Sep 10, 2017

koffie commented Sep 10, 2017

fchapoton commented Aug 16, 2018

jdemeyer commented Aug 16, 2018

fchapoton commented Aug 16, 2018

Update pip to 9.0.1 #23615

Update pip to 9.0.1 #23615

Comments

koffie commented Aug 13, 2017

This comment has been minimized.

koffie commented Aug 13, 2017

koffie commented Aug 13, 2017

koffie commented Aug 13, 2017

sagetrac-git mannequin commented Aug 14, 2017

sagetrac-git mannequin commented Aug 14, 2017

jdemeyer commented Aug 14, 2017

sagetrac-git mannequin commented Aug 14, 2017

sagetrac-git mannequin commented Aug 14, 2017

koffie commented Aug 14, 2017

jhpalmieri commented Aug 14, 2017

jdemeyer commented Aug 15, 2017

jhpalmieri commented Aug 15, 2017

jdemeyer commented Aug 23, 2017

vbraun commented Sep 2, 2017

fchapoton commented Sep 10, 2017

fchapoton commented Sep 10, 2017

fchapoton commented Sep 10, 2017

fchapoton commented Sep 10, 2017

koffie commented Sep 10, 2017

koffie commented Sep 10, 2017

fchapoton commented Sep 10, 2017

koffie commented Sep 10, 2017

fchapoton commented Sep 10, 2017

koffie commented Sep 10, 2017

koffie commented Sep 10, 2017

fchapoton commented Aug 16, 2018

jdemeyer commented Aug 16, 2018

fchapoton commented Aug 16, 2018