Test all installed optional packages by default

[nathanncohen]

With this branch, running "sage -t" will automatically involve testing all installed up-to-date new-style optional Sage packages.

Depends on #18456
Depends on #18124
Depends on #18559
Depends on #18563
Depends on #18579
Depends on #18581

CC: @vbraun @jdemeyer @sagetrac-tmonteil @seblabbe

Component: doctest framework

Author: Nathann Cohen, Jeroen Demeyer

Branch/Commit: 82f3eec

Reviewer: Jeroen Demeyer, Karl-Dieter Crisman, John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/18558

[nathanncohen]

Branch: u/ncohen/18558

[nathanncohen]

Commit: dd586b9

[nathanncohen]

New commits:

a3402c9	trac #18456: Re-Fix standard_packages(), optional_packages(), and experimental_packages()
dd586b9	trac #18558: Test all (installed) optional packages by default

[jdemeyer]

comment:2

See also #13540.

[nathanncohen]

comment:3

See also #13540.

Cool, a 3 years old ticket without a branch. I will drop them a line.

Nathann

[jdemeyer]

comment:4

Replying to @nathanncohen:

See also #13540.

Cool, a 3 years old ticket without a branch.

Well, there is a patch.

[jdemeyer]

comment:5

Note that tests for the cbc package are marked # optional - Coin so I guess this needs to be fixed.

[nathanncohen]

comment:6

Well, there is a patch.

It is two years old only. Which do you prefer to review, theirs or mine? :-P

Note that tests for the cbc package are marked # optional - Coin so I guess this needs to be fixed.

True, I will create another ticket in a second (done: #18559)

Nathann

[jdemeyer]

comment:7

Instead of patching sage-runtests, patch the doctest framework, and add some doctests.

[jdemeyer]

comment:8

And instead of naming the option sage_and_installed_packages, I would prefer to have an option like alloptional (all installed optional packages), so one can do --optional=sage,alloptional,internet (which is not possible with your patch).

[nathanncohen]

comment:9

I did not patch the doctest framework because the default value of 'optional' is systematically overwritten by sage-runtests. As for 'add some doctest', I do not understand what you mean so be more specific.

[jdemeyer]

comment:10

Replying to @nathanncohen:

I did not patch the doctest framework because the default value of 'optional' is systematically overwritten by sage-runtests.

What do you mean with this? The sage-runtests script is really a very thin front-end for the doctest framework in the Sage library, it doesn't parse the value of the --optional option.

[jdemeyer]

comment:11

Replying to @nathanncohen:

As for 'add some doctest', I do not understand what you mean so be more specific.

I meant "add a doctest in the doctest framework", but I admit I haven't thought much about what this should be.

[jdemeyer]

comment:12

On the topic of the option name: make it just optional instead of alloptional. So the default value should be --optional=sage,optional.

[nathanncohen]

comment:13

What do you mean with this? The sage-runtests script is really a very thin front-end for the doctest framework in the Sage library, it doesn't parse the value of the --optional option.

I mean that I was surprised to find that the DocTestDefaults class is not even instanciated when you run doctests.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

e45591a

trac #18558: now with an 'installed_optional' keyword

[sagetrac-git]

Changed commit from dd586b9 to e45591a

[nathanncohen]

comment:15

There is now an installed_optional tag which gets replaced by the list of all (installed) optional packages. The replacement is done in doctest.py.

Nathann

[jdemeyer]

Changed dependencies from #18456 to #18456, #18124

[vbraun]

comment:17

Is there going to be a separate installed_experimental keyword? If not then it would be better to shorten it; Just --optional=optional (somewhat confusing) or --optional=installed. Imho the latter is best since its obvious enough that it refers to packages installed by Sage. It would also fit nicely with auto-detection on #13540, which would then be --optional=auto.

[jdemeyer]

comment:127

Replying to @jdemeyer:

For information, this is a list of all optional tags with count (assuming #18124 and #18559):

    302 
     59 4ti2
    622 arb
    114 axiom
      7 benzene
     31 bliss
     10 buckygen
    209 cbc
     25 chevie
     33 CHomP
    383 coxeter3
    185 CPLEX
    185 cryptominisat
      1 CryptoMiniSat
      4 cunningham
     11 CVXOPT
     24 database_cremona_ellcurve
    112 database_gap
     14 database_jones_numfield
      2 database_kohel
      3 database_odlyzko_zeta
      8 database_pari
     28 database_stein_watkins
     13 database_symbolic_data
      1 debug
      3 dot2tex
     31 dot2tex graphviz
     43 FES
      8 ffmpeg
     65 fricas
     51 frobby
    115 gambit
     93 gap3
      2 gap3chevie
     67 gap_packages
      5 gdb
    142 giac
      2 ginv
      1 gnuplot
     19 gperftools
    201 Gurobi
      6 GUROBI
      1 Hmisc R package
     72 ImageMagick
    272 internet
      2 java
      2 java internet
    109 kash
      8 latex
     29 latte_int
      1 libjpeg
     96 lie
      9 long time
     30 lrs
      2 M2
     15 macaulay2
    660 magma
    151 maple
      2 Maple
    119 mathematica
     59 matlab
      4 mcqd
     10 modular_decomposition
     79 mupad
      2 MuPAD
     20 nauty
    157 Nonexistent_LP_solver
     61 octave
     69 phc
     35 plantri
     15 polymake
      5 polytopes_db_4d
    180 qepcad
      1 random
      2 requires maple
     11 rgraphics
      2 runsnake
     98 scilab
      5 sloane_database
     16 souvigner
      3 surf
      4 tides
      8 time
     40 TOPCOM
      3 webbrowser

I am fixing most mistakes in the above list in #18621 and #18637.

[kcrisman]

comment:128

An interesting thing that comes up with #18637's fixes is that I don't think the internet optional doctests were tested in my run. It that something that your changes (or something) could also automatically assess? Those tests constantly are breaking, and it would be wonderful if that could be done here.

However, I recognize that may be orthogonal enough (or difficult enough) to not bother with here, and instead better to deal with elsewhere.

[jdemeyer]

comment:129

Replying to @kcrisman:

An interesting thing that comes up with #18637's fixes is that I don't think the internet optional doctests were tested in my run.

They are not run automatically run. This ticket is only about optional Sage packages.

However, I recognize that may be orthogonal enough (or difficult enough) to not bother with here, and instead better to deal with elsewhere.

Yes, see #13540

[jhpalmieri]

comment:130

I think it's easy to automatically run the internet tests. I don't actually know if that's a good idea: how often will someone start doctesting with internet access (at which point the tag will get added to the list) and then lose it during doctesting (they lose their internet connection, they take their laptop outside, etc.)?

Adapted from #13540:

diff --git a/src/sage/doctest/control.py b/src/sage/doctest/control.py
index ce846b2..7b07f2d 100644
--- a/src/sage/doctest/control.py
+++ b/src/sage/doctest/control.py
@@ -256,6 +256,14 @@ class DocTestController(SageObject):
                     for pkg, versions in optional_pkgs.items():
                         if versions[0] == versions[1]:
                             options.optional.add(pkg)
+                    # If there is an internet connection, add
+                    # 'internet' to the list of optional tags.
+                    try:
+                        import urllib2
+                        urllib2.urlopen('http://sagemath.org/')
+                        options.optional.add('internet')
+                    except urllib2.URLError:
+                        pass
 
                 # Check that all tags are valid
                 for o in options.optional:

I also expect lots of failing internet doctests, and I don't know how to fix all of them. For example, I don't understand the doctesting structure for the files in sage/dev in the first place.

Of course this all belongs on another ticket, either #13540 or a piece of it.

[jhpalmieri]

comment:131

Actually, for reasons I really don't understand, my previous code with urllib2 gave me a ton of segfaults when doctesting Sage (?!). In fact I was getting segfaults just from inserting the lines

import urllib2
urllib2.urlopen('http://sagemath.org/')

without the line options.optional.add('internet'). I must be doing something wrong, but I don't know what.

Anyway, the following doesn't produce segfaults (taken from stackoverflow):

diff --git a/src/sage/doctest/control.py b/src/sage/doctest/control.py
index ce846b2..ff365dd 100644
--- a/src/sage/doctest/control.py
+++ b/src/sage/doctest/control.py
@@ -256,6 +256,15 @@ class DocTestController(SageObject):
                     for pkg, versions in optional_pkgs.items():
                         if versions[0] == versions[1]:
                             options.optional.add(pkg)
+                    # If there is an internet connection, add
+                    # 'internet' to the list of optional tags.
+                    try:
+                        import socket
+                        host = socket.gethostbyname('sagemath.org')
+                        socket.create_connection((host, 80), 2)
+                        options.optional.add('internet')
+                    except:
+                        pass
 
                 # Check that all tags are valid
                 for o in options.optional:

[jhpalmieri]

comment:132

Finally, for the actual ticket under discussion, I am happy with it. Anyone have any remaining objections?

[jhpalmieri]

comment:133

My brain is not quite working today. I am almost happy with the current ticket. I am not sure about the message Optional tags: gcc,mpir,python2,sage,scons: I think this is open to misinterpretation. Maybe it could say instead something like Doctesting using '--optional=gcc,mpir,python2,sage,scons'?

[kcrisman]

Changed reviewer from Jeroen Demeyer, Karl-Dieter Crisman to Jeroen Demeyer, Karl-Dieter Crisman, John Palmieri

[kcrisman]

comment:134

That would suffice to address my very similar concern in comment:117 as well.

[jdemeyer]

comment:135

How about

Using --optional=gcc,mpir,python2,sage,scons

[seblabbe]

comment:136

Optional packages that are outside of sage (like graphviz and mathematica) are currently not detected as tags. With mathematica 10.0 and sage 6.8.beta3 and graphviz and this ticket I get:

$ sage -t src/sage/interfaces/mathematica.py 
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------

but

sage -t --optional=mathematica src/sage/interfaces/mathematica.py 
----------------------------------------------------------------------
sage -t src/sage/interfaces/mathematica.py  # 50 doctests failed
----------------------------------------------------------------------

Similarly, less tests are done because the tag graphviz is not detected:

$ sage -t src/sage/graphs/graph_latex.py
Git branch: t/18558
Optional tags: dot2tex,gcc,mpir,python2,sage,scons
Doctesting 1 file.
sage -t src/sage/graphs/graph_latex.py
    [186 tests, 0.17 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------

compared to

$ sage -t --optional=graphviz,sage,dot2tex src/sage/graphs/graph_latex.py
Git branch: t/18558
Optional tags: dot2tex,graphviz,sage
Doctesting 1 file.
sage -t src/sage/graphs/graph_latex.py
    [190 tests, 2.48 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------

[jdemeyer]

comment:137

Replying to @seblabbe:

Optional packages that are outside of sage (like graphviz and mathematica) are currently not detected as tags.

I know, but that's outside the scope of this ticket.

[seblabbe]

comment:138

Okay. I just installed as many optional packages that I could. Below is the list of my optional tags:

Optional tags: 
arb,benzene,bliss,buckygen,cbc,cryptominisat,
database_cremona_ellcurve,database_gap,database_odlyzko_zeta,database_pari,
database_stein_watkins,database_symbolic_data,dot2tex,gambit,gcc,gdb,mcqd,
modular_decomposition,mpir,nauty,plantri,python2,sage,scons,tides

I am starting a make ptestlong now and I will tell you tomorrow the result.

[jhpalmieri]

comment:139

Replying to @jdemeyer:

How about

Using --optional=gcc,mpir,python2,sage,scons

Sounds okay to me.

[sagetrac-git]

Changed commit from e3f463d to 82f3eec

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

82f3eec

Change "Using --optional" message

[seblabbe]

comment:142

Replying to @seblabbe:

I am starting a make ptestlong now and I will tell you tomorrow the result.

Running doctests with ID 2015-06-09-16-25-45-44c48cbc.
Git branch: t/18558
Optional tags: arb,benzene,bliss,buckygen,cbc,cryptominisat,
database_cremona_ellcurve,database_gap,database_odlyzko_zeta,
database_pari,database_stein_watkins,database_symbolic_data,
dot2tex,gambit,gcc,gdb,mcqd,modular_decomposition,
mpir,nauty,plantri,python2,sage,scons,tides
Doctesting entire Sage library.
Sorting sources by runtime so that slower doctests are run first....
Doctesting 3135 files using 4 threads.
[...]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------

[vbraun]

Changed branch from u/jdemeyer/18558 to 82f3eec