Cleaning sage-package-list #19213

jhpalmieri · 2015-09-14T21:26:34Z

This ticket enhances the module sage.misc.package and the script sage-list-packages:

add independent functions in sage.misc.package to list the (new style) packages
add pip functionalities to sage.misc.package
do not list anymore old style packages (see also get rid of remote old style packages #21132)
relies on sage.misc.packages in sage-list-packages

pip type packages were created in #19187. This ticket is such that these packages will be listed with sage -optional. Sadly, it will still not be possible for these packages to be used in optional doctests. The reason for that is that we do not have proper version control for them.

CC: @videlec @jdemeyer @vbraun @embray

Component: scripts

Author: Vincent Delecroix

Branch: f054998

Reviewer: Matthias Koeppe, Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/19213

The text was updated successfully, but these errors were encountered:

jhpalmieri · 2015-09-14T21:27:32Z

Branch: u/jhpalmieri/pip_packages

videlec · 2015-09-15T01:15:49Z

Commit: 00224aa

videlec · 2015-09-15T01:15:49Z

comment:3

To my mind, it would nice to also list non installed pip packages as well as the last version available. The pip version can be obtained from Python with the json API

import urllib2
import json

def pip_version(package_name):
    url = "https://pypi.python.org/pypi/{}/json".format(package_name)

    try:
        data = json.load(urllib2.urlopen(urllib2.Request(url)))
    except urllib2.HTTPError:
        return None

    try:
        return data["info"]["version"]
    except KeyError:
        return None

The last version available might differ from what is installed. And it is worth mentioning that an update is available.

New commits:

`93ec09b`	`Change sage -p to always install a package`
`c6ddd0e`	`Better help`
`f3d7ef3`	`Remove -f option to sage-spkg when running sage -i`
`752e269`	`Use PKG-clean target to implement sage -f PKG`
`1812e4f`	`Merge tag '6.9.beta6' into t/19119/ticket/19119`
`adef308`	`Add rules for installing packages with pip`
`fdba3ee`	`Add mercurial and sqlalchemy (ex-standard packages)`
`00224aa`	`trac 19213: include packages from build/pkgs/piprules in 'sage --optional'`

jhpalmieri · 2015-09-15T01:38:23Z

comment:4

Replying to @videlec:

To my mind, it would nice to also list non installed pip packages as well as the last version available. The pip version can be obtained from Python with the json API

One disadvantage to this is that it requires internet access, which sage --optional --local does not do. But this would be a good addition if --local were not given. If we do list the non-installed packages, then it's going to get unwieldy if there are hundreds of entries in build/pkgs/piprules. What if that file gets big?

Do we want to list these as optional packages or in another category? sage --pip_packages?

jdemeyer · 2015-09-15T07:25:58Z

comment:5

I do not want these pip packages to be treated like optional packages. If anything, they are closer to "experimental" than to "optional". We don't have control over them, so they are more likely to break when the upstream package gets upgraded.

jhpalmieri · 2015-09-15T15:26:57Z

comment:6

Replying to @jdemeyer:

I do not want these pip packages to be treated like optional packages. If anything, they are closer to "experimental" than to "optional". We don't have control over them, so they are more likely to break when the upstream package gets upgraded.

Do you mean

you want them listed in sage --experimental, not sage --optional, or
you don't want them detected for the purpose of doctests,

or both?

jdemeyer · 2015-09-15T16:54:56Z

comment:7

Replying to @jhpalmieri:

Do you mean

you want them listed in sage --experimental, not sage --optional, or

you don't want them detected for the purpose of doctests,

or both?

The default for --optional flags in doctests really should be optional packages only, so I don't want pip packages to be tested by doctests by default. Therefore, I also don't want those packages to appear in sage --optional. So I really prefer to see them as a new category: not standard, not optional, not experimental.

jhpalmieri · 2015-09-15T18:29:06Z

comment:8

Wait, if an (old-style) optional package is listed in piprules, you still don't want it listed in sage --optional? Should it be removed altogether from the listing, or listed as not_installed? If it's old-style, by the way, having it listed in sage --optional is somewhat independent of having it used in doctests, since the doctest framework only looks at local packages. So in my current branch, if we delete the changes to doctest/control.py, then these pip packages won't be used automatically in doctests.

videlec · 2015-09-16T14:49:27Z

comment:9

Replying to @jdemeyer:

Replying to @jhpalmieri:

Do you mean

you want them listed in sage --experimental, not sage --optional, or

you don't want them detected for the purpose of doctests,

or both?

The default for --optional flags in doctests really should be optional packages only, so I don't want pip packages to be tested by doctests by default. Therefore, I also don't want those packages to appear in sage --optional. So I really prefer to see them as a new category: not standard, not optional, not experimental.

I see several solutions to deal with "pip packages":

having a new category of packages (let say python-optional)
be stricter about versioning, i.e. in piprules specify the version of the packages we support

brian=1.3.1
trac=2.3

Very simple to maintain!

I agree that it would be hard to prevent a pip package to just break because versions are not backward compatible. So solution 2 looks appropriate to me. Otherwise, we can have a more precise tagging system for python-optional packages

sage: my_test()  # python-optional -- 3.5 <= beautifulsoup <= 4.1.2

It is more flexible than the piprules strict versioning solution but the cost of maintenance is much higher.

Vincent

videlec · 2015-09-16T15:32:24Z

comment:10

I have a working version of sage-list-packages doing

$ ./sage-list-packages python
[package] .............................. [latest version] ([installed version])

beautifulsoup........................... 3.2.1 (3.2.1)
biopython............................... 1.65 (1.65)
brian................................... 1.4.1 (1.4.1)
guppy................................... 0.1.10 (0.1.10)
mercurial............................... 3.5.1 (3.5.1)
mpi4py.................................. 1.3.1 (1.3.1)
nibabel................................. 2.0.1 (2.0.1)
pybtex.................................. 0.18 (0.18)
pyflakes................................ 0.9.2 (0.9.2)
pyopenssl............................... 0.15.1 (0.15.1)
sqlalchemy.............................. 1.0.8 (1.0.8)
trac.................................... 1.0.9 (1.0.9)

The only problem is that it is relatively long to get the information from the pip json API for all the packages. Perhaps, one can be smarter with urllib2.

John, am I free to change the branch in order to make my proposition publicly available?

Vincent

videlec · 2015-09-16T15:44:47Z

comment:11

Furthermore, could we separate the issues of:

listing python packages
including optional doctests for them

Vincent

jhpalmieri · 2015-09-16T16:02:43Z

comment:12

Replying to @videlec:

John, am I free to change the branch in order to make my proposition publicly available?

Please do so.

jhpalmieri · 2015-09-16T16:56:43Z

comment:13

I have a few questions and comments:

if Archive superseded/broken old-style packages #19220 is merged (archiving superseded packages, including the piprules packages), how are users supposed to find out about biopython, brian, etc.? We need to include an option to sage which will call sage-list-packages python. Maybe we should also have an option which lists all packages: first standard, then optional, then experimental, then python/pip. Or maybe group all of the non-standard packages together, sorted by name. In any case, we shouldn't expect users to even be aware that there are several different types of non-standard packages; they should instead just be able to get one list to see if some particular package is supported.

Maybe a list like

[package (type)] .......... [latest version] ([installed version])

...
biopython (pip) ........... latest version (installed version)
...
chomp (experimental) ...... latest version (installed version)
...
ore_algebra (optional) .... latest version (installed version)
...

I'm not really sure, though.

Regarding the versions, I am not convinced that we can accurately specify which version of each package we support. Who is going to maintain this? Unless there is some automatic way of determining the versions supported by Sage, the versions will quickly get out of date. If brian gets updated to 1.4.2 but we list brian=1.4.1, what happens if a user does sage -i brian? What happens if a user does sage --pip install brian?

videlec · 2015-09-16T21:53:41Z

Changed commit from 00224aa to none

videlec · 2015-09-16T21:53:41Z

Changed branch from u/jhpalmieri/pip_packages to public/pip_packages

videlec · 2015-09-16T21:53:57Z

Commit: 19e3b86

videlec · 2015-09-16T21:53:57Z

New commits:

`93ec09b`	`Change sage -p to always install a package`
`c6ddd0e`	`Better help`
`f3d7ef3`	`Remove -f option to sage-spkg when running sage -i`
`752e269`	`Use PKG-clean target to implement sage -f PKG`
`1812e4f`	`Merge tag '6.9.beta6' into t/19119/ticket/19119`
`adef308`	`Add rules for installing packages with pip`
`fdba3ee`	`Add mercurial and sqlalchemy (ex-standard packages)`
`00224aa`	`trac 19213: include packages from build/pkgs/piprules in 'sage --optional'`
`19e3b86`	`trac 19213: separation of python packages`

videlec · 2015-09-16T21:53:57Z

Changed branch from public/pip_packages to public/19213

mkoeppe · 2016-07-16T17:17:25Z

comment:16

What is the status on this?

videlec · 2016-08-10T21:23:10Z

comment:49

Replying to @jdemeyer:

I would like to see what the patchbot thinks of the startup time (however, it seems down now). If this ticket does not give a statistically significant increase to the startup time, it is ok for me. See [comment:43]

Since the lists are built with local=True this should even be faster than before. The old way to build these lists was via the script sage-package-list...

jdemeyer · 2016-08-10T21:26:57Z

comment:50

Replying to @videlec:

The old way to build these lists was via the script sage-package-list...

Unless I'm missing something, in the old version, the list of packages was not computed every time that Sage starts.

videlec · 2016-08-10T21:29:07Z

comment:51

Replying to @jdemeyer:

Replying to @videlec:

The old way to build these lists was via the script sage-package-list...

Unless I'm missing something, in the old version, the list of packages was not computed every time that Sage starts.

It was the very same. All of _STANDARD_PACKAGES, _OPTIONAL_PACKAGES, _EXPERIMENTAL_PACKAGES where computed at module loading time (since they are used in the doc of the module). And the module is loaded on startup.

jhpalmieri · 2016-08-11T17:23:51Z

comment:52

I see a significant difference in startup time: on one OS X machine it takes about 800ms longer on average with this branch. (I ran sage --startuptime 10 times with and without the branch, discarding the first run in each case. Average without: 1760ms. With: 2560ms.) If I run sage --startuptime sage.misc.package, without the branch it takes about 0.8ms to import that module, while with the branch: 800ms.

sagetrac-git · 2016-08-11T18:45:58Z

Changed commit from de597b2 to c6578f4

sagetrac-git · 2016-08-11T18:45:58Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`ad92490`	`19213: clean sage.misc.package`
`8840aa3`	`19213: clean sage-list-packages script`
`383171a`	`19213: use list_packages in sage.doctest.control`
`3b97166`	`19213: work around MacOSX (absent) TLS support`
`c91197c`	`19213: a warning with tests`
`c6578f4`	`Trac 19213: remove X_PACKAGES from the doc + deprecations`

videlec · 2016-08-11T18:48:04Z

comment:54

Indeed... I was wrong the variables X_PACKAGES where hardcoded (and unmaintained).

I rebased and added some deprecations to all of standard_packages, optional_packages, experimental_packages and package_versions (since all the information is now available from list_packages).

jdemeyer · 2016-08-12T08:30:54Z

comment:56

Replying to @videlec:

Indeed... I was wrong the variables X_PACKAGES where hardcoded

Yes, that's what I meant.

jdemeyer · 2016-08-12T08:32:31Z

comment:57

I don't see why standard_packages() and friends should be deprecated. They offer a user-friendly interface for list_packages().

jdemeyer · 2016-08-12T08:38:06Z

comment:58

Anyway, I don't really care much about this.

sagetrac-git · 2016-08-12T14:10:49Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f054998`	`Trac 19213: remove deprecations`

sagetrac-git · 2016-08-12T14:10:49Z

Changed commit from c6578f4 to f054998

videlec · 2016-08-12T14:11:14Z

comment:60

I don't care either. But a pair of lists is not a lot more friendly than a dictionary.

vbraun · 2016-08-17T06:40:33Z

Changed branch from u/vdelecroix/19213 to f054998

koffie · 2017-09-13T17:42:01Z

comment:64

It seems that the deprecation warnings were removed in the latest commit f054998 , but the documentation still says they are deprecated. I modified the documentation accordingly in #23849 .

koffie · 2017-09-13T17:42:01Z

Changed commit from f054998 to none

jhpalmieri added this to the sage-6.9 milestone Sep 14, 2015

jhpalmieri added c: packages: optional labels Sep 14, 2015

jdemeyer added s: needs work and removed s: needs review labels Sep 15, 2015

mkoeppe modified the milestones: sage-6.9, sage-7.3 Jul 16, 2016

jhpalmieri added s: needs work and removed s: needs review labels Aug 11, 2016

videlec added s: needs review and removed s: needs work labels Aug 11, 2016

This comment has been minimized.

Sign in to view

jdemeyer added s: positive review and removed s: needs review labels Aug 15, 2016

vbraun removed the s: positive review label Aug 17, 2016

vbraun closed this as completed in 46c1a6f Aug 17, 2016

jdemeyer mentioned this issue Sep 20, 2015

Add rules for installing packages with pip #19187

Closed

mkoeppe mentioned this issue May 17, 2020

package type = pip needs documentation #21033

Closed

videlec mentioned this issue Aug 17, 2020

get rid of remote old style packages #21132

Closed

koffie mentioned this issue Sep 24, 2017

Remove deprecation from optional_packages documentation and friends #23849

Closed

Cleaning sage-package-list #19213

Cleaning sage-package-list #19213

Comments

jhpalmieri commented Sep 14, 2015

jhpalmieri commented Sep 14, 2015

videlec commented Sep 15, 2015

videlec commented Sep 15, 2015

jhpalmieri commented Sep 15, 2015

jdemeyer commented Sep 15, 2015

jhpalmieri commented Sep 15, 2015

jdemeyer commented Sep 15, 2015

jhpalmieri commented Sep 15, 2015

videlec commented Sep 16, 2015

videlec commented Sep 16, 2015

videlec commented Sep 16, 2015

jhpalmieri commented Sep 16, 2015

jhpalmieri commented Sep 16, 2015

videlec commented Sep 16, 2015

videlec commented Sep 16, 2015

videlec commented Sep 16, 2015

videlec commented Sep 16, 2015

videlec commented Sep 16, 2015

mkoeppe commented Jul 16, 2016

videlec commented Aug 10, 2016

jdemeyer commented Aug 10, 2016

videlec commented Aug 10, 2016

jhpalmieri commented Aug 11, 2016

sagetrac-git mannequin commented Aug 11, 2016

sagetrac-git mannequin commented Aug 11, 2016

videlec commented Aug 11, 2016

This comment has been minimized.

jdemeyer commented Aug 12, 2016

jdemeyer commented Aug 12, 2016

jdemeyer commented Aug 12, 2016

sagetrac-git mannequin commented Aug 12, 2016

sagetrac-git mannequin commented Aug 12, 2016

videlec commented Aug 12, 2016

This comment has been minimized.

vbraun commented Aug 17, 2016

koffie commented Sep 13, 2017

koffie commented Sep 13, 2017