Upgrade numpy to 1.5.0 and scipy to 0.8

[sagetrac-maldun]

This ticket updates two packages, which must be updated together,

http://sage.math.washington.edu/home/kcrisman/numpy-1.5.0.spkg

http://sage.math.washington.edu/home/palmieri/SPKG/scipy-0.8.spkg

If you are applying these to any version before Sage 4.6.alpha3, then you also have to update the scipy_sandbox package too (#10092). This has been merged in 4.6.alpha3, though.

http://boxen.math.washington.edu/home/kirkby/patches/scipy_sandbox-20071020.p7.spkg

After installing Numpy, one needs to execute sage -ba, or do

touch $SAGE_LOCAL/lib/python/site-packages/Cython/Includes/numpy.pxd

or else one will get runtime warnings. (Or if someone wants less hassle, they can patch sage-4.6.alpha3.spkg before building Sage).

trac_9808_changed_doctests.patch in the attachment has to be applied, in order to get all doctests running because some of the output has changed.

For reviewers:
changes.txt holds a summary of all changes with reference to the diffs, and links to other tickets

Upstream: Fixed upstream, but not in a stable release.

Component: packages: standard

Keywords: numpy, scipy

Author: Stefan Reiterer, François Bissey, John Palmieri, David Kirkby, Karl-Dieter Crisman

Reviewer: Karl-Dieter Crisman, David Kirkby, Leif Leonhardy, François Bissey

Merged: sage-4.6.1.alpha0

Issue created by migration from https://trac.sagemath.org/ticket/9808

[sagetrac-maldun]

comment:1

I have now workable spkg files for the both packages.
Scipy worked well.
Numpy need some work, but it's functioning.

I host the files at megaupload, at:
Numpy: http://www.megaupload.com/?d=6GCFZD65
Scipy: http://www.megaupload.com/?d=JORT40DK

I think the problem is the same as described in Trac # 7791 (#7791 )

I get the following errors:

/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/plot/all.py:22: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from complex_plot import complex_plot
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/plot/all.py:22: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from complex_plot import complex_plot
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/plot/plot3d/implicit_plot3d.py:5: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from implicit_surface import ImplicitSurface
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/plot/plot3d/implicit_plot3d.py:5: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from implicit_surface import ImplicitSurface
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/calculus/all.py:17: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from riemann import Riemann_Map
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/calculus/all.py:17: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from riemann import Riemann_Map
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/calculus/all.py:19: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from interpolators import polygon_spline, complex_cubic_spline
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/calculus/all.py:19: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from interpolators import polygon_spline, complex_cubic_spline
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/stats/hmm/all.py:8: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from hmm  import DiscreteHiddenMarkovModel
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/stats/hmm/all.py:8: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from hmm  import DiscreteHiddenMarkovModel

What's this?
Has someone hardcoded the sizes of the routines?

[sagetrac-maldun]

comment:2

numpy 1.5.0rc1: http://www.megaupload.com/?d=KK31RSZV

[mwhansen]

comment:3

I don't think that we should upgrade to 1.5.0rc1 -- we should do 1.4.1 for now and wait until 1.5 is released.

[kiwifb]

comment:4

Replying to @mwhansen:

I don't think that we should upgrade to 1.5.0rc1 -- we should do 1.4.1 for now and wait until 1.5 is released.

That has to be double checked but maldun says he needs features in 1.5.
It may be that the features he wants are 1.4.x and he doesn't know.
Do we know roughly when 1.5 will be released? They are at 1.5rc1 9 days after beta2
so it could very well be upon us in a very short time.

[sagetrac-maldun]

comment:5

The warnings go away after rebuilding the source.
And the doctests only throw warnings but there don't seem to be errors, and the output is
correct

@mwhansen, fbissey
I think numpy has the same issues since version 1.4.
and I'm quite sure that resolving it in 1.5.0rc1 would, solve the problem
with 1.4.1. too. (and perhaps also with 1.5. later)
So I suggest the following procedure:
I solve the issues with the latest. Dobule check if 1.4.1 also works, and we decide then
which of the versions we keep, or for example keep 1.4.1 in standard and 1.5.0rc1 as experimental

[sagetrac-drkirkby]

comment:6

If 1.5.0rc1 is merged, #7166 can be closed. I don't know about 1.4.1, but in any case that is not a critical bug.

Some time back I made a Solaris-specific change to Numpy, as I wanted to get it reviewed with the least hassle - that generally means making it Solaris specific, as reviewers are easier to please if it only effects a rarer platform.

But I think the change should be implemented on OS X too. Currently there's a really nasty hack on OS X to build Numpy, that involves a script called fake_gcc, copying that to $SAGE/LOCAL/bin/gcc, then using the fake gcc to build Numpy. Finally this fake gcc file gets deleted.

The far neater option is the way I did it on Solaris. I suggest the changes I made for Solaris are implemented whenever SAGE64 is set to "yes", irrespective of whatever platform it may be. Then all this fake gcc rubbish can be removed.

If you want, I can create a patch.

[sagetrac-maldun]

comment:7

Replying to @sagetrac-drkirkby:

If you want, I can create a patch.

would be nice! But first I have to sort some things out, I hope it will get ready soon...

[kiwifb]

comment:8

Looking at all that stuff in the spkg and comparing to Gentoo. Not very pretty.
Do we really still need to use sage_fortran? On most platform we now use a native
fortran compiler rather than a sage shipped one - I don't know if we still do that
for OS X.
In the patch to gnu.py there is:

+        # Note that sse flags and so on lead to gfortran code that segfaults, so disable arch flags
+        return opt
+

An extract of the Gentoo set up:

	export NUMPY_FCONFIG="config_fc --noopt --noarch"

Actually here is the full set up that you might find interesting:

	# See progress in http://projects.scipy.org/scipy/numpy/ticket/573
	# with the subtle difference that we don't want to break Darwin where
	# -shared is not a valid linker argument
	if [[ ${CHOST} != *-darwin* ]] ; then
		append-ldflags -shared
	fi

	# only one fortran to link with:
	# linking with cblas and lapack library will force
	# autodetecting and linking to all available fortran compilers
	use lapack || return
	[[ -z ${FC} ]] && FC=$(tc-getFC)
	# when fortran flags are set, pic is removed.
	FFLAGS="${FFLAGS} -fPIC"
	export NUMPY_FCONFIG="config_fc --noopt --noarch"

The other patches we have are relatively minor, a fix to the f2py man page,
a patch for freebsd - that may be usefull, a patch for interix(!).
I cannot comment on the cygwin patches but they look very small.
We are are a bit more anal with site.cfg, I don't think it is useful in the
context of sage - we just have more complex possible combinations of blas/lapack.

The NUMPY_FCONFIG is passed to distutils as an argument of

python setup.py install

i.e. in the end what we do boils down to "python setup.py install ${NUMPY_FCONFIG}".

Can you point me to your solaris fix Dave?
I'd like to see if I can tidy all that up in something that requires less patching
and is more based on FLAGS settings.

[sagetrac-drkirkby]

comment:9

Replying to @kiwifb:

Looking at all that stuff in the spkg and comparing to Gentoo. Not very pretty.

What a surprise.

Do we really still need to use sage_fortran? On most platform we now use a native
fortran compiler rather than a sage shipped one - I don't know if we still do that
for OS X.

I've never understood the need for this SAGE_FORTRAN. I've tried arguing for it to be removed and use FC instead, but I had no luck.

A Fortran compiler is still shipped on OS X, but I don't see why the variable FC can't be made to point to that, rather than have the variable SAGE_FORTRAN.

Can you point me to your solaris fix Dave?
I'd like to see if I can tidy all that up in something that requires less patching
and is more based on FLAGS settings.

On Solaris, and some OS X versions, if you want a 64-bit build, you must add the compiler flag -m64 with gcc. Usually, putting that in CFLAGS is enough. However, for Numpy that does not work.

Someone came up with a fix for this which was implemented only on OS X, that involved creating a wrapper script called gcc_fake which basically calls gcc with the -m64 option. You can see the script yourself in the top level directory.

Since they had done this only on OS X, it did not work on Solaris. So I came up with a solution for Solaris, but I avoided the wrapper script. Instead I set the variable to CC=gcc -m64

I'm attaching a patch, which basically uses the Solaris on any system, including OS X. I think this is the sensible way to do it, not have a wrapper script.

I've not tested the attached patch - not even on Solaris!! But I think you can see what I am trying to do. I was going to try to explain it in words, but a bit of code, even if untested, should be more sensible.

[sagetrac-drkirkby]

Attachment: 9808-remove-gcc_fake.patch.gz

An untested patch, which makes Numpy build the same was on OS X as it does on Solaris or other platforms where SAGE64=yes. It removes the stupid wrapper script.

[kiwifb]

comment:10

Ok - so we still use g95 on some targets. So we need to keep some patches
just for these - bother.

[sagetrac-drkirkby]

comment:11

Replying to @kiwifb:

Ok - so we still use g95 on some targets. So we need to keep some patches
just for these - bother.

I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but SAGE_FORTRAN can't be removed.

Dave

[kiwifb]

comment:12

Replying to @sagetrac-drkirkby:

Replying to @kiwifb:

Ok - so we still use g95 on some targets. So we need to keep some patches
just for these - bother.

I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but SAGE_FORTRAN can't be removed.

That's good! That means we probably can give the shove to the gnu.py and init.py
patches. I wouldn't be sorry to see the back of these.

There is a comment in SPKG.txt:

Special Update/Build Instructions:
  * The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this file and must be updated if/when 
    the file src/numpy/doc/cython/numpy.pxi is updated.

I cannot find the file in question in that location. There is however a numpy.pxi under src/numpy/random/mtrand but I am not sure that's the file in question.
Furthermore I don't appear to have a numpy.pxi in $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi .
Does anyone know if these instructions are obsolete?

[jasongrout]

comment:13

FYI: Numpy 1.5 has been officially released now.

[jasongrout]

comment:14

Replying to @kiwifb:

Replying to @sagetrac-drkirkby:

Replying to @kiwifb:

Ok - so we still use g95 on some targets. So we need to keep some patches
just for these - bother.

I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but SAGE_FORTRAN can't be removed.

That's good! That means we probably can give the shove to the gnu.py and init.py
patches. I wouldn't be sorry to see the back of these.

There is a comment in SPKG.txt:

Special Update/Build Instructions:
  * The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this file and must be updated if/when 
    the file src/numpy/doc/cython/numpy.pxi is updated.

I cannot find the file in question in that location. There is however a numpy.pxi under src/numpy/random/mtrand but I am not sure that's the file in question.
Furthermore I don't appear to have a numpy.pxi in $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi .
Does anyone know if these instructions are obsolete?

I think I was the one that added those instructions, and I'm pretty sure they're obsolete instructions now. I believe we took care of merging the differences between the two numpy.pxi/pxd files a while ago.

[kiwifb]

comment:15

Replying to @jasongrout:

I think I was the one that added those instructions, and I'm pretty sure they're obsolete instructions now. I believe we took care of merging the differences between the two numpy.pxi/pxd files a while ago.

Thank you for the confirmation. I have done some cleaning to numpy's spkg-install and it seems to work as intended on my machine. I guess we'll update to 1.5 and give it a spin.

[sagetrac-drkirkby]

comment:16

Replying to @kiwifb:

Replying to @mwhansen:

I don't think that we should upgrade to 1.5.0rc1 -- we should do 1.4.1 for now and wait until 1.5 is released.

That has to be double checked but maldun says he needs features in 1.5.
It may be that the features he wants are 1.4.x and he doesn't know.
Do we know roughly when 1.5 will be released? They are at 1.5rc1 9 days after beta2
so it could very well be upon us in a very short time.

In general though, we should not be upgrading to pre-release versions just because one person needs a feature that's only available in a pre-release. Everyone should not have to suffer the extra risks a pre-release gives unless there are compelling arguments for it.

I realise in this case 1.5 has since been released, so it's immaterial now.

Dave

[jasongrout]

comment:17

Replying to @sagetrac-drkirkby:

In general though, we should not be upgrading to pre-release versions just because one person needs a feature that's only available in a pre-release. Everyone should not have to suffer the extra risks a pre-release gives unless there are compelling arguments for it.

+1

[kiwifb]

comment:18

Replying to @sagetrac-drkirkby:

In general though, we should not be upgrading to pre-release versions just because one person needs a feature that's only available in a pre-release. Everyone should not have to suffer the extra risks a pre-release gives unless there are compelling arguments for it.

I realise in this case 1.5 has since been released, so it's immaterial now.

Yes, I didn't think it would be worth working on that particular version if it wasn't
for the real proximity of the release (curses pari svn upgrade). I would probably
have put pressure for 1.4.1 otherwise (very keen to see numpy upgraded).

[sagetrac-maldun]

comment:19

Replying to @jasongrout:

FYI: Numpy 1.5 has been officially released now.

So I was right =) Til I'm done with 1.5.0rc1 1.5 is out...

So changed so far:

The following doctest had to be rewritten:

File "/home/maldun/sage/sage-4.5.2/devel/sage/doc/en/faq/faq-usage.rst", line 341:
    sage: stats.ttest_ind(list([1,2,3,4,5]),list([2,3,4,5,.6]))
Expected:
    doctest...DeprecationWarning...
    (0.076752955645333687, 0.94070490247380478)
Got:
    (0.076752955645333687, 0.94070490247380478)

That was easy =)

The next prob was:

File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/graphs/graph.py", line 615:
    sage: Graph(A)
Exception raised:
    Traceback (most recent call last):
      File "/home/maldun/sage/sage-4.5.2/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/home/maldun/sage/sage-4.5.2/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/home/maldun/sage/sage-4.5.2/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_1[22]>", line 1, in <module>
        Graph(A)###line 615:
    sage: Graph(A)
      File "/home/maldun/sage/sage-4.5.2/local/lib/python/site-packages/sage/graphs/graph.py", line 846, in __init__
        data = networkx.MultiGraph(data)
      File "/home/maldun/sage/sage-4.5.2/local/lib/python2.6/networkx/classes/graph.py", line 220, in __init__
        convert.from_whatever(data,create_using=self)
      File "/home/maldun/sage/sage-4.5.2/local/lib/python2.6/networkx/convert.py", line 157, in from_whatever
        if isinstance(thing,numpy.core.defmatrix.matrix) or \
    AttributeError: 'module' object has no attribute 'defmatrix'

that was also easy. defmatrix just changed from numpy.core to numpy.matrix to numpy.matrixlib
Only changed that line in /local/lib/python2.6/networkx/classes/graph.py

Here comes now a trickier one:

       sage -t -valgrind "devel/sage/sage/rings/polynomial/polynomial_element.pyx"
Total time for all tests: 716.4 seconds
maldun@zauberbuch:~/sage/sage-4.5.2$ sage -t  -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx
> "
ERROR: File ./devel/sage/sage/rings/polynomial/real_roots.pyx
 is missing
 
----------------------------------------------------------------------
The following tests failed:


        ./devel/sage/sage/rings/polynomial/real_roots.pyx
 # File not found
Total time for all tests: 0.0 seconds
maldun@zauberbuch:~/sage/sage-4.5.2$ sage -t  -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx"
sage -t -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx"
**********************************************************************
File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1819, in __main__.example_76
Failed example:
    oc.find_roots()###line 3064:_sage_    >>> oc.find_roots()
Expected nothing
Got:
    doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
**********************************************************************
File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1840, in __main__.example_77
Failed example:
    oc.find_roots()###line 3085:_sage_    >>> oc.find_roots()
Expected nothing
Got:
    doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
**********************************************************************
File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1934, in __main__.example_80
Failed example:
    oc.find_roots()###line 3157:_sage_    >>> oc.find_roots()
Expected nothing
Got:
    doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
**********************************************************************
File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 2320, in __main__.example_98
Failed example:
    real_roots(x**Integer(5) * (x**Integer(2) - Integer(9999))**Integer(2) - Integer(1))###line 3870:_sage_    >>> real_roots(x^5 * (x^2 - 9999)^2 - 1)
Expected:
    [((-29274496381311/9007199254740992, 419601125186091/2251799813685248), 1), ((2126658450145849453951061654415153249597/21267647932558653966460912964485513216, 4253316902721330018853696359533061621799/42535295865117307932921825928971026432), 1), ((1063329226287740282451317352558954186101/10633823966279326983230456482242756608, 531664614358685696701445201630854654353/5316911983139663491615228241121378304), 1)]
Got:
    doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
    [((-29274496381311/9007199254740992, 419601125186091/2251799813685248), 1), ((2126658450145849453951061654415153249597/21267647932558653966460912964485513216, 4253316902721330018853696359533061621799/42535295865117307932921825928971026432), 1), ((1063329226287740282451317352558954186101/10633823966279326983230456482242756608, 531664614358685696701445201630854654353/5316911983139663491615228241121378304), 1)]
**********************************************************************
4 items had failures:
   1 of  10 in __main__.example_76
   1 of   9 in __main__.example_77
   1 of  10 in __main__.example_80
   1 of  44 in __main__.example_98
***Test Failed*** 4 failures.
         [227.6 s]
 
----------------------------------------------------------------------
The following tests failed:


        sage -t -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx"
Total time for all tests: 227.6 seconds

This is not the only one, but the root of evil seems to be in real_roots (how ironic...)
more precisly: I tracked it down, with help of valgrind, to the class ocean. Has anyone an Idea??

[sagetrac-maldun]

comment:21

Update: Found the problem. see: http://groups.google.com/group/cython-users/browse_thread/thread/624c696293b7fe44

It seems that all versions 1.5.x hold this bug....
Tried downgrading to 1.4.1. and all corrections I have done so far worked.

I will now apply the patch from drkirkby, pack it again, test it overnight and hopefully we are done with 1.4.1, and hopefully they get it right in the next time. perhaps I should send the numpy/scipy guys the doctests I've done so far

scipy 0.8. don't seem to make any problems so far

[sagetrac-maldun]

comment:23

an doctest I oversaw:

Type ``numpy.typecodes`` for a list of the possible
        typecodes::

            sage: import numpy
            sage: sorted(numpy.typecodes.items())
            [('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]

The output is now:

[('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]

But since it's an extension, and no downgrade it is also no prob

[jhpalmieri]

comment:287

Updates: my doctest failure in citation.pyx is a "red herring". In particular, it's not because of the patches here; it's because of a flaw in how the function get_systems works. Often when I've been testing the packages and patches on this ticket, I've created a new version of sage in a directory called "numpy", and get_systems sees that string in the path name and adds "numpy" to the systems used by the particular command. If I rename the directory to something else, then the doctest passes. So let's not worry about this; at some point, someone can fix get_systems so it ignores the initial chunk of the path name (ignore the parent of SAGE_ROOT, for example).

My doctest failure in matrix1.pyx is still there. For some reason, the "Datetime" type codes are not there on some systems. Any ideas why? I glanced at the install log for numpy, but I didn't see anything suspicious, not that I really know what to look for. Here's the log from my OS X 10.6 machine, which is one place I see this problem: http://sage.math.washington.edu/home/palmieri/misc/numpy-1.5.0.log.

Finally, I have a new version of the scipy spkg, which works for me on all of the machines I've tested on: linux, OpenSolaris, Solaris on x86, Mac OS X 10.6. I'm putting the link in the ticket description, and I'm posting the Mercurial patch -- it's very small.

I'm leaving this as "needs work" because of the matrix1.pyx issue, but I think that's the only remaining problem. Please test the new scipy spkg on other systems to make sure, though.

[jhpalmieri]

for reference only: diff between old scipy 0.8 spkg and new one

[jhpalmieri]

comment:288

Attachment: trac_9808-scipy.patch.gz

See #10129 for the citation.pyx issue.

[sagetrac-maldun]

comment:289

@jhpalmieri I looked again on your matrix1.pyx problem, and this really interesting if you look on comment #23 ( #9808 comment:23 ) we already had this change in linux also, but in numpy-1.4.1 and it vanished for numpy-1.5.0 again.

So whatever happen in Mac OS X it appears like an old numpy output comes up again, and if you look at it carefully it actually isn't really an error, more like that more keywords are available.

So perhaps it has something to do with the numpy installation of numpy itself on OS X. (perhaps another bug of numpy)

[jhpalmieri]

comment:290

Replying to @sagetrac-maldun:

So whatever happen in Mac OS X it appears like an old numpy output comes up again, and if you look at it carefully it actually isn't really an error, more like that more keywords are available.

It looked to me like the "Datetime" keywords were missing, rather than more being available, but I might be wrong.

So perhaps it has something to do with the numpy installation of numpy itself on OS X. (perhaps another bug of numpy)

I also see this on OpenSolaris, for what that's worth.

In any case, it's not a big deal, is it? Can we just change the doctest somehow?

[sagetrac-maldun]

comment:291

lol I have the picture now: it seemed you installed my old patch which inherits the datetime change which was left from numpy-1.4.1

If you try the new patch trac_9808_changed_doctests.patch in the attachement instead of
trac_9808_changed_doctests.patch it will work. (look in the patch)

The one which is at fault here is the wiki: I overrided the file, but the wiki let you download the old patch.
Try it out: Download both patches, and look at them. The old one has the false filesize and inherits this change what had to be applied to numpy 1.4.1, in the new one this is left out

[jhpalmieri]

comment:293

Okay, I understand. I don't know if it's the wiki's fault or mine: I may have just been using an old copy of the patch file which I had on my computer. Anyway, I'm trying again now. I've marked this "needs review", but things are looking pretty good now.

[jhpalmieri]

comment:294

With the new spkgs and the new patch, all tests pass for me (except for known unrelated failures) on skynet machines eno (linux), lena (linux), and fulvia (Solaris on x86). The build on mark (Solaris on sparc) is ongoing; it's a very slow machine. All tests also pass on my OS X 10.6 box and on Dave Kirkby's OpenSolaris machine hawk, as well as on sage.math.

Is this enough for a positive review? I guess not quite: since I prepared the most recent scipy spkg, someone else should definitely review the changes (given in attachment: trac_9808-scipy.patch) and test it out.

[kcrisman]

comment:295

I was able to successfully install both packages on my slowest PPC OS X system; doing ./sage -b after touching the include will take a while, and I would have to wait until Monday to do all tests. But I will at least test devel/sage/sage/matrix/, devel/sage/sage/functions/, devel/sage/sage/finance/, and devel/sage/sage/numerical, because those use Scipy and Numpy a lot and are NOT eons to test.

Certainly the patch makes sense. I can't check now whether the spkg was correctly created and whether it has any changes not checked in, but assuming all is well, this should be considered a successful, if long, process. And I hope 1.5.1 won't come out the day after - though it looks like the Enthought folks are already looking to 2.0 as well.

[kcrisman]

comment:296

Okay, these tests (which are most of the numpy/scipy ones in Sage, though not all) are passing fine, with the expected errors (I didn't apply the patch, but am seeing precisely what is in the patch). Great!

I also checked this now

Creating Sage package scipy-0.8/

Created package scipy-0.8.spkg.

    NAME: scipy
 VERSION: 0.8
    SIZE: 4.6M
 HG REPO: Good
SPKG.txt: Good

from the latest package John posted. Positive review!

[sagetrac-maldun]

comment:297

Replying to @kcrisman:

from the latest package John posted. Positive review!

I've never thought I will read those words... I think I need a beer now 8)

The patch also works on Sage-4.6.alpha3 on Ubuntu. (Which was expected, but it is better to check too often then one time too less)

Thanks to everyone who worked on this ticket! I'm quite the beginner, but I think I learned a lot by helping upgrading numpy and can finally going back working on #9706 which should be a lot easier, but I needed the update to complete this task, because the new scipy holds a lot new features for orthogonal polynomials.
Hope I didn't cause too much grey hairs!

Kind regards,
Stefan aka maldun

[jasongrout]

comment:298

Kudos to you guys!

(and 3 more comments to reach 300!)

[sagetrac-mhampton]

comment:300

Congratulations, I got exhausted just trying to follow this!

[jdemeyer]

Changed reviewer from Karl-Dieter Crisman, David Kirkby, Leif Leonhardy, Francois Bissey to Karl-Dieter Crisman, David Kirkby, Leif Leonhardy, François Bissey

[jdemeyer]

Changed author from Stefan Reiterer, Francois Bissey, John Palmieri, David Kirkby, Karl-Dieter Crisman to Stefan Reiterer, François Bissey, John Palmieri, David Kirkby, Karl-Dieter Crisman

[jdemeyer]

Merged: sage-4.6.1.alpha0

[nexttime]

comment:303

Replying to @jhpalmieri:

Replying to @kiwifb:

Better use plain 'uname' as it should return Darwin for you if I am not mistaken.

No, the point is you want to test not only for Darwin, but for Darwin version 10.6, as opposed to 10.5 or 10.4. uname -r should return strings like 10.4.0, 9.3.1, and 8.8.0, respectively, for these (I think). (The last parts of the string, like 4.0 or 3.1 or 8.0, are the minor version numbers, which I don't think we care about.)

This seems to work for me, but I'm not a sed expert:

VER=`uname -r | sed 's/\([0-9]*\)\..*/\1/'`

(This takes the output from uname -r, sends it to sed, which does a regular expression match to return the digits found before the first period.) Then you do something like

if [ $VER -ge 10 ]; then
   ...
fi

(Might as well test whether VER is at least 10, rather than equal to 10 on the nose.)

Just for the record:

The easiest (and by the way more efficient and less error-prone) way to test for e.g. Darwin 8 / MacOS X 10.4 / Tiger is to use the (Bourne) shell's built-in pattern matching:

case "$UNAME" in # set in sage-env
    Darwin)
        case "`uname -r`" in # quotes not mandatory
            8*) # Tiger / 10.4
                ...
                ;;
            9*) # Leopard / 10.5
                ...
                ;;
            10*) # Snow Leopard / 10.6
                ...
                ;;
            *)  # other, "default"
                ...
        esac
    # add other OSs like Linux here if appropriate
esac

Or, if you want to use sed or tr, something like:

os_with_ver=`uname -sr | sed -e 's/ /-/g'` # order of options to 'uname' doesn't matter

os_with_ver_and_arch=`uname -srm | tr ' ' '-'` # using the simpler 'tr'(anslate) command

case $os_with_ver in
    Darwin-8*) # Tiger / Darwin 8 / MacOS X 10.4
        ...
        ;;
    ...
esac

# More specific:
case $os_with_ver_and_arch in
    Darwin-8*-ppc) # Tiger / Darwin 8 / MacOS X 10.4 on PPC (32-bit)
        ...
        ;;
    Darwin-9*-ppc64) # Leopard / Darwin 9 / MacOS X 10.5 on PPC (64-bit)
        ...
        ;;
    Linux-*-x86_64|Linux-*-ia64) # Any 64-bit Linux version on Intel
        ...
        ;;
    Linux-*-i[3456]86) # Any 32-bit Linux version on Intel
        ...
        ;;
    ...
esac