os x 10.6 port -- numerous mysterious errors caused by weird "abort trap" issue

[williamstein]

After testing sage-4.1.2.rc0 (and applying a tiny numerical noise fix), we have 12 files that all fail doctests with "myserious error":

wstein@bsd:~/build/sage-4.1.2.rc0$ grep mysterious testlong.log |wc -l
      12

The files with failures are:

        sage -t -long "devel/sage/sage/calculus/calculus.py"
        sage -t -long "devel/sage/sage/calculus/tests.py"
        sage -t -long "devel/sage/sage/calculus/wester.py"
        sage -t -long "devel/sage/sage/functions/hyperbolic.py"
        sage -t -long "devel/sage/sage/functions/other.py"
        sage -t -long "devel/sage/sage/functions/trig.py"
        sage -t -long "devel/sage/sage/gsl/interpolation.pyx"
        sage -t -long "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
        sage -t -long "devel/sage/sage/rings/polynomial/pbori.pyx"
        sage -t -long "devel/sage/sage/symbolic/constants.py"
        sage -t -long "devel/sage/sage/symbolic/expression.pyx"
        sage -t -long "devel/sage/sage/symbolic/function.pyx"

To fix, use the following new spkgs, plus the two patches to the Sage library (trac-7095.patch and trac_7095-ref.patch), plus the patch to sage_scripts, plus (finally) add the new sage_env also to SAGE_ROOT/spkg/base (available only in a Sage source distribution), mind to chmod the right flags for this sage-env:

• http://sage.math.washington.edu/home/craigcitro/sage/SPKGs/python-2.6.4.p4.spkg

• http://sage.math.washington.edu/home/craigcitro/sage/SPKGs/matplotlib-0.99.1.p4.spkg

Component: porting

Author: Craig Citro, John Palmieri, Francis Clarke, William Stein

Reviewer: David Kirkby, Georg S. Weber

Merged: sage-4.3.1.rc1

Issue created by migration from https://trac.sagemath.org/ticket/7095

[williamstein]

comment:2

A good example:

sage: factorial(factorial(x))

Program received signal SIGABRT, Aborted.
0x00007fff82adf096 in __kill ()
(gdb) bt
#0  0x00007fff82adf096 in __kill ()
#1  0x00007fff82b800c2 in abort ()
#2  0x0000000100204d51 in uw_init_context_1 ()
#3  0x00000001002054cc in _Unwind_Resume ()
#4  0x000000010742c4cb in __gnu_cxx::new_allocator<GiNaC::ex>::deallocate () at new_allocator.h:97
#5  0x000000010742c4cb in std::_Vector_base<GiNaC::ex, std::allocator<GiNaC::ex> >::_M_deallocate () at new_allocator.h:137
#6  0x000000010742c4cb in ~auto_ptr [inlined] () at new_allocator.h:123
#7  0x000000010742c4cb in ~vector [inlined] () at new_allocator.h:593
#8  0x000000010742c4cb in ~_Vector_base [inlined] () at /usr/include/c++/4.2.1/memory:259
#9  0x000000010742c4cb in GiNaC::container<std::vector>::subs (this=0x10b010e70, m=@0x7fff5fbfbdb0, options=0) at new_allocator.h:271
#10 0x00000001077b5923 in GiNaC::ex::subs () at /Users/was/build/sage-4.1.2.rc0/local/include/pynac/ex.h:827
#11 0x00000001077b5923 in __pyx_pf_4sage_8symbolic_10expression_10Expression_substitute (__pyx_v_self=0x10b1a9638, __pyx_args=<value temporarily unavailable, due to optimizations>, __pyx_kwds=0x10b059480) at ex.h:13728
#12 0x0000000100006fd2 in PyObject_Call (func=0x10b18ca28, arg=0x10b1b0c50, kw=0x0) at Objects/abstract.c:2492
#13 0x0000000107799f2d in __pyx_pf_4sage_8symbolic_10expression_10Expression_test_relation (__pyx_v_self=0x109bea7a0, __pyx_args=<value temporarily unavailable, due to optimizations>, __pyx_kwds=<value temporarily unavailable, due to optimizations>) at sage/symbolic/expression.cpp:9041
#14 0x0000000100006fd2 in PyObject_Call (func=0x1006c7200, arg=0x100230050, kw=0x0) at Objects/abstract.c:2492
#15 0x00000001077a6d84 in __pyx_pf_4sage_8symbolic_10expression_10Expression___nonzero__ (__pyx_v_self=0x109bea7a0) at sage/symbolic/expression.cpp:7677
#16 0x000000010004d336 in PyObject_IsTrue (v=<value temporarily unavailable, due to optimizations>) at Objects/object.c:1547
#17 0x00000001000b1ec3 in PyEval_EvalFrameEx (f=0x10b066620, throwflag=<value temporarily unavailable, due to optimizations>) at Python/ceval.c:2185
#18 0x00000001000b7650 in PyEval_EvalCodeEx (co=0x103c33300, globals=<value temporarily unavailable, due to optimizations>, locals=<value temporarily unavailable, due to optimizations>, args=0x10b1b0c28, argcount=1, kws=0x10b00b4f0, kwcount=0, defs=0x103c46d68, defcount=1, closure=0x0) at Python/ceval.c:2968
#19 0x0000000100035f0d in function_call (func=0x103c47848, arg=0x10b1b0c10, kw=0x10b062680) at Objects/funcobject.c:524
#20 0x0000000100006fd2 in PyObject_Call (func=0x103c47848, arg=0x10b1b0c10, kw=0x10b062680) at Objects/abstract.c:2492
#21 0x00000001000b1a92 in PyEval_EvalFrameEx (f=0x10b066410, throwflag=<value temporarily unavailable, due to optimizations>) at Python/ceval.c:4019

[sagetrac-fwclarke]

comment:3

Some of these errors seems to arise from test_relation in sage/symbolic/expression.pyx. Checking out

bool(diff(csch(x), x) == diff(Integer(1)/sinh(x), x))

which gives rise to the "mysterious error" in sage/functions/hyperbolic.py, I find the following phenomena (with Mac OS X 10.61, 2 GHz Intel Core 2 Duo, 2 GB):

----------------------------------------------------------------------
| Sage Version 4.1.2.rc0, Release Date: 2009-09-30                   |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
**********************************************************************
*                                                                    *
* Warning: this is a prerelease version, and it may be unstable.     *
*                                                                    *
**********************************************************************
sage: sinh(x).subs({x: 2})
sinh(2)
sage: sinh(x).subs({x: 2.2})
4.45710517053589
sage: sinh(x).subs({x: RIF(2.2)})
4.457105170535895?
sage: sinh(x).subs({x: CIF(2.2)})
/Users/mafwc/sage-4.1.2.rc0/local/bin/sage-sage: line 202:  2260 Abort trap              sage-ipython "$@" -i

and similarly for cosh, tanh and several other hyperbolic functions. But

----------------------------------------------------------------------
| Sage Version 4.1.2.rc0, Release Date: 2009-09-30                   |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
**********************************************************************
*                                                                    *
* Warning: this is a prerelease version, and it may be unstable.     *
*                                                                    *
**********************************************************************
sage: coth(x).subs({x: 2})
coth(2)
sage: coth(x).subs({x: 2.2})
coth(2.20000000000000)
sage: coth(x).subs({x: RIF(2.2)})
coth(2.2000000000000002?)
sage: coth(x).subs({x: CIF(2.2)})
coth(2.2000000000000002?)
sage: 

There's a serious inconsistency here, and clearly a problems with CIF.

[williamstein]

comment:4

I was reading this blog post and it mentions that Apple made some massive changes to GCC-4.2.x in XCode:

http://dobbscodetalk.com/index.php?option=com_myblog&show=Snow-Leopard-Changes-to-C.html&Itemid=29

In particular, they added closures to C!

This page talks about weird issues with the new Xcode and GCC-4.2.x on OS X and Python: http://bugs.python.org/issue6957
But it also points out that XCode for 10.6 comes with GCC-4.0.1. The default is 32-bit for GCC-4.0.1, unlike GCC-4.2.x, which on OS X defaults to 643-bit. So... I'm now trying fresh 32 and 64-bit builds of Sage using GCC-4.0.1. I'm doing this by changing my path to point to versions of gcc and g++ that are the 4.0.1 versions, instead of trusting environment variables, which are bound to fail. I'll report more here later.

Anyway, if this works, we'll be in good shape -- we could just revisit 4.2.x support later as components of Sage stabilize and people iron out the bugs in the huge changes Apple may have made to C.

[williamstein]

comment:5

Major progress -- a lot of problems appear to just "go away" when using GCC-4.0.x! In particular, the above issue with "Abort Trap" goes away. The main problem is that some of the fixes I put in for OS X 10.6 with GCC-4.2.x actually break the 4.0.x build.... anyway, I'm very optimistic for a complete resolution to the problem of building Sage on OS X 10.6 using GCC 4.0.x.

[sagetrac-drkirkby]

comment:6

Two things

• sage -t "devel/sage/sage/gsl/interpolation.pyx fails on Solaris too.
• If gcc 4.2.x is an issue on OS X 10.6, I could create a prereq-0.5 which gives a warning with this combination of operating system version and compiler, so people don't get caught out. Just let me know
  • The exact OS version
  • The exact compiler version
  • The exact wording you want.

I was intending updating prereq anyway, after getting some feedback. If nothing else, I can fix #7156 at some point, but that is not an important fix. But this seems a bit more important.

Dave

[williamstein]

comment:7

Major progress -- a lot of problems appear to just "go away" when
using GCC-4.0.x! In particular, the above issue with "Abort Trap" goes away.

NOTE: Actually, it is only in 32-bit mode that this Abort Trap issue goes away. In 64-bit mode with gcc-4.0.x, the problem remains. Unfortunately a full build in 32-bit mode has 97 files that have failing doctests. In 64-bit mode the one remaining problem above doesn't go away, and we still have tons of failures. So using GCC-4.0.x is no solution at all, unfortunately... not even close.

[williamstein]

comment:8

The factorial(factorial(x)) error above is really from factorial(CIF(1)) which leads to the same error. Combining this with fwclarke's remark above, suggests complex interval fields are somehow the one place where things are broken somehow.

[williamstein]

comment:9

No, the problem isn't CIF, since:

sage: factorial(CC(1.0))
^P/Users/was/build/sage-4.1.2.rc1.alpha3/local/bin/sage-sage: line 202: 67490 Abort trap              sage-ipython "$@" -i
sage: factorial(CDF(1.0))
/Users/was/build/sage-4.1.2.rc1.alpha3/local/bin/sage-sage: line 202: 67569 Abort trap              sage-ipython "$@" -i

[sagetrac-fwclarke]

comment:10

I wonder about lines 123--124 of local/lib/python/config/Makefile (which have been there since at least 3.1.4):

MACOSX_DEPLOYMENT_TARGET=10.3
export MACOSX_DEPLOYMENT_TARGET

It seems to be these lines that are giving rise (in 10.6 but not in 10.5) to many warnings of the form

#warning Building for Intel with Mac OS X Deployment Target < 10.4 is invalid.

Whether this is causing the problems, I don't know, but it does need to be changed (presumably to 10.4).

[williamstein]

comment:11

Since we still have no clue, I'm targeting this for 4.3.

[sagetrac-drkirkby]

comment:12

Given the comment I put on sage-devel about there two quite respectable parties believing gcc 4.3.3 is the most stable gcc, it might be worth trying a build with gcc 4.3.3.

Although it will lengthen the build process a bit, I would be tempted to enable all the mpfr tests by default. They have a habbit of finding compiler bugs. Having known of several mpfr test failures, my experience has always been that the test is ok, and mpfr has discovered a bug in other code. In one case it was the Solaris memset() call, but on an several cases it is gcc bugs.

Dave

[williamstein]

comment:13

Given the comment I put on sage-devel about there two quite respectable parties
believing gcc 4.3.3 is the most stable gcc, it might be worth trying a build
with gcc 4.3.3.

My understanding is that there is no GCC 4.3.3 for OS X. There's only 4.2.x and 4.0.x, as provided by Apple. I don't think Apple has ported GCC > 4.2 to OS X yet.

Although it will lengthen the build process a bit, I would be tempted to enable all > the mpfr tests by default. They have a habbit of finding compiler bugs. Having known > of several mpfr test failures, my experience has always been that the test is ok, .
and mpfr has discovered a bug in other code. In one case it was the Solaris memset() > call, but on an several cases it is gcc bugs.

I think effort would be better spent making export SPKG_CHECK="yes" fully work; then we would always do test builds of Sage with that flag set, and hence run all tests.

[sagetrac-drkirkby]

comment:14

The fact Apple does not ship gcc 4.3.3 should not mean it can not be built.

I'm not sure what SPKG_CHECK is supposed to do, but if its to run testsuites on all packages, that would undoubtedly take a very long time. I think there needs to be a way of doing it on specific packages. Perhaps RUN_MPFR_TESTSUITE could be a variable, but personally I think for the time it takes, it's one worth always running.

PS, I never get any emails when trac items I comment on (even if I start them) are updated. So unless I constantly keep checking trac items. I was not aware you had commented on this. Can you see if there is something wrong with how my account is set up, as I thought that was supposed to happen. Even if I 'cc' drkirkby, that does not work either.

Dave

[jhpalmieri]

comment:16

Replying to @sagetrac-fwclarke:

I wonder about lines 123--124 of local/lib/python/config/Makefile (which have been there since at least 3.1.4):

MACOSX_DEPLOYMENT_TARGET=10.3
export MACOSX_DEPLOYMENT_TARGET

It seems to be these lines that are giving rise (in 10.6 but not in 10.5) to many warnings of the form

#warning Building for Intel with Mac OS X Deployment Target < 10.4 is invalid.

Whether this is causing the problems, I don't know, but it does need to be changed (presumably to 10.4).

This message is fixed (it seems) in Python 2.6.4 -- we use 2.6.2 right now. Updating to 2.6.4 doesn't help with the main problem here, though.

Here are a few random comments which don't tell me much, but maybe they'll help someone else. The failure in arith.py is a false positive: it is from the function factorial, but the doctest is actually calling factorial from functions/other.py. If you preface the doctests in arith.py with something like from sage.rings.arith import factorial, then doctests for that file pass. Maybe this should be done anyway?

Re the error in rings/polynomial/pbori.pyx: it's weird. If I put in some print statements, then I find that the problem comes from the line

        gb = groebner_basis(self.gens(), **kwds)

This calls the function groebner_basis in polybori.gbcore, which ends when it hits a massive "return" statement (in the nested function call_algorithm). I can change that statement from

return  ...blah...

to

XX = ...blah...
print XX
return XX

and it successfully prints XX, but then it crashes.

Re the error in plot/text.py: this one can be reproduced in pure Python as follows:

$ sage -python
Python 2.6.2 (r262:71600, Dec 10 2009, 21:33:56) 
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib.pyplot as plt
>>> plt.plot([1,2,3])
[<matplotlib.lines.Line2D object at 0x101eb2490>]
>>> plt.savefig('a.pdf')
/Applications/sage/local/bin/sage-sage: line 356: 62648 Abort trap              python "$@"

This seems interesting; does it help to have a failure coming from pure Python?

Most of the others seem to call bool and perhaps are related to ginac?

[sagetrac-drkirkby]

comment:17

There's a lot of patches to the python .spkg file. Some look to me like they might be old and perhaps not needed any more. (There's things done because they caused problems in python 2.5).

It might be worth updating to 2.6.4 (which is only a bug-fix upgrade), then looking at seeing what python patches are actually necessary, so making a clearer python package.

It could be one of the patches which is causing the failures.

[jhpalmieri]

comment:18

See #7022 for perhaps the same matplotlib issue.

[jhpalmieri]

comment:19

Replying to @sagetrac-drkirkby:

There's a lot of patches to the python .spkg file. Some look to me like they might be old and perhaps not needed any more. (There's things done because they caused problems in python 2.5).

It might be worth updating to 2.6.4 (which is only a bug-fix upgrade), then looking at seeing what python patches are actually necessary, so making a clearer python package.

It could be one of the patches which is causing the failures.

I tried building a new Python 2.6.4 spkg (not for general distribution, just for working on this ticket). In doing so, I got rid of all of the patches except for those to two files: pickle.py and cPickle.c. It looks like a lot of the other issues have been fixed, but I'm not sure. In any case, I still get the same doctest failures, and I don't see how the two remaining patches could be causing this.

[williamstein]

comment:20

I'm declaring a total feature freeze on sage-4.3. Also, this is clearly not a blocker for making a release so I changed it to critical.

[rlmill]

comment:48

Oh! I didn't realize you upgraded Python also, I just saw the p*'s.

[sagetrac-GeorgSWeber]

comment:49

Good work so far, but it seems that at least the numpy spkg also needs to be updated, e.g. by adding the lines:

    if [ `uname` = "Darwin" ]; then
        # set MACOSX_DEPLOYMENT_TARGET -- if set incorrectly, this can
        # cause lots of random "Abort trap" issues on OSX. see trac
        # #7095 for an example.
        MACOSX_VERSION=`uname -r | awk -F. '{print $1}'`
        MACOSX_DEPLOYMENT_TARGET=10.$[$MACOSX_VERSION-4]
        export MACOSX_DEPLOYMENT_TARGET
    fi

from the python spkg-install also to the numpy spg-install.

More precisely, on my MacIntel with OS X 10.4, I just tried to build sage-4.3.1.rc0 from scratch, with only the two new spkgs from this ticket thrown in. The build choked on numpy with:

...
gcc -L/Users/Shared/sage/test/sage-4.3.1.rc0_7095/sage-4.3.1.rc0/local/lib -bundle -undefined dynamic_lookup build/temp.macosx-10.4-i386-2.6/numpy/random/mtrand/mtrand.o build/temp.macosx-10.4-i386-2.6/numpy/random/mtrand/randomkit.o build/temp.macosx-10.4-i386-2.6/numpy/random/mtrand/initarray.o build/temp.macosx-10.4-i386-2.6/numpy/random/mtrand/distributions.o -Lbuild/temp.macosx-10.4-i386-2.6 -o build/lib.macosx-10.4-i386-2.6/numpy/random/mtrand.so
running scons
running build_scripts
creating build/scripts.macosx-10.4-i386-2.6
Creating build/scripts.macosx-10.4-i386-2.6/f2py
  adding 'build/scripts.macosx-10.4-i386-2.6/f2py' to scripts
changing mode of build/scripts.macosx-10.4-i386-2.6/f2py from 644 to 755
error: Can't install when cross-compiling
Error building numpy.

real	1m5.507s
user	0m55.447s
sys	0m5.804s
sage: An error occurred while installing numpy-1.3.0.p2
...

But after setting "export MACOSX_DEPLOYMENT_TARGET=10.4" manually on the command line, then issuing another "make", the build would run OK for numpy and continue (not finished yet, currently at the singular spkg).

[jhpalmieri]

comment:50

    if [ `uname` = "Darwin" ]; then
        # set MACOSX_DEPLOYMENT_TARGET -- if set incorrectly, this can
        # cause lots of random "Abort trap" issues on OSX. see trac
        # #7095 for an example.
        MACOSX_VERSION=`uname -r | awk -F. '{print $1}'`
        MACOSX_DEPLOYMENT_TARGET=10.$[$MACOSX_VERSION-4]
        export MACOSX_DEPLOYMENT_TARGET
    fi

Should we just add these lines to sage-env?

[sagetrac-GeorgSWeber]

comment:51

Replying to @jhpalmieri:

    if [ `uname` = "Darwin" ]; then
        # set MACOSX_DEPLOYMENT_TARGET -- if set incorrectly, this can
        # cause lots of random "Abort trap" issues on OSX. see trac
        # #7095 for an example.
        MACOSX_VERSION=`uname -r | awk -F. '{print $1}'`
        MACOSX_DEPLOYMENT_TARGET=10.$[$MACOSX_VERSION-4]
        export MACOSX_DEPLOYMENT_TARGET
    fi

Should we just add these lines to sage-env?

Yes, this is probably the best solution. Any more votes?

[sagetrac-GeorgSWeber]

comment:52

Update: The scipy spkg has the same problem (with the same solution) as the numpy spkg.

[craigcitro]

comment:53

Replying to @jhpalmieri:

Should we just add these lines to sage-env?

I'm +1 on this. Has someone rolled a new patch/spkg, or should I do it tonight?

[sagetrac-drkirkby]

comment:54

Since I'd made a major update to sage-env which went wrong, as setting CFLAGS had an inadvertant side effect, I'd like to change to do what it does now, but not export CFLAGS, CXXFLATS etc.

I can easily tac those on at the same time. And rolls that out as a refinement of #7817.

I can do that tonight if you want.

Dave

[craigcitro]

comment:55

Well, I'd like this to get into the next rc, and it sounds like #7818 may get pushed to the next release for more serious testing. So I'd like to go ahead and get something pushed for this ticket ...

[sagetrac-drkirkby]

comment:56

ok, go for it

[craigcitro]

comment:57

Attachment: sage-scripts_trac_7095.patch.gz

Apparently I just forgot to make these changes last night. Anyway, I've attached a patch for the $SAGE_ROOT/local/bin repo that puts the changes in sage-env, and I've posted a new python spkg (which simply removes those lines) here:

http://sage.math.washington.edu/home/craigcitro/sage/SPKGs/python-2.6.4.p4.spkg

This should do it ... famous last words. :) The only problem would be if sage-env isn't sourced before we build the python spkg, but I don't think that's the case. (Reality will correct me if I'm wrong.)

[jhpalmieri]

comment:58

Hi Craig,

Could you also post a copy of the modified sage-env? There are two copies in sage, one in SAGE_ROOT/local/bin which is part of sage_scripts, and another one (which should be identical) in SAGE_ROOT/spkg/base -- I think this one gets run at the start of the make process. Does it ever get run later? Anyway, we should make sure that the two copies are the same, and the release manager might need sage-env posted here for that.

[craigcitro]

comment:59

No sweat -- I actually didn't know there was a second copy. (Does sdist and/or bdist do something where it copies the one from local/bin into spkg/base when making a tarball? If not, someone should file an enhancement ticket ...)

[craigcitro]

Attachment: sage-env.gz

Copy of sage-env that results from the attached sage-scripts patch

[sagetrac-drkirkby]

comment:60

The sage-env in local/bin does not start there - it gets copied there from spkg/base

Dave

[sagetrac-GeorgSWeber]

Changed reviewer from David Kirkby to David Kirkby, Georg S. Weber

[sagetrac-GeorgSWeber]

comment:61

Looks OK to me.

[rlmill]

Merged: sage-4.3.1.rc1

[rlmill]

comment:62

Replying to @jhpalmieri:

Hi Craig,

Could you also post a copy of the modified sage-env? There are two copies in sage, one in SAGE_ROOT/local/bin which is part of sage_scripts, and another one (which should be identical) in SAGE_ROOT/spkg/base -- I think this one gets run at the start of the make process. Does it ever get run later? Anyway, we should make sure that the two copies are the same, and the release manager might need sage-env posted here for that.

There is actually an hg repo in spkg/base, so you can just submit patches... I've checked in the changes recommended here, so don't worry about it this time.

Updated python and matplotlib spkgs, applied trac_7095.patch, trac_7095-ref.patch, and sage-scripts_trac_7095.patch, and then copied the new sage-env file to spkg/base (checking in the changes there).

.....If sage-env is copied from spkg/base to local/bin, then why is it under revision control in both places!? This should probably be fixed, and documented better.