cvxopt: fix illegal BLAS call and fix Solaris build #12011

jhpalmieri · 2011-11-11T06:40:38Z

On OS X Lion, if you set SAGE_CHECK=yes and build cvxopt, some tests fail and then it hangs before completing the tests. See http://sage.math.washington.edu/home/palmieri/misc/cvxopt-1.1.3.log for the install log.

CVXOPT devs claimed that Apple shipped a broken BLAS. But it turned out that they cannot provide valid examples showing this; they did not adhere to a de facto BLAS standard, and they agreed to fix the corresponding CVXOPT bug quickly.

This spkg also fixes an issue building with GCC on Solaris 10. There already was a fix for this in the spkg, but it was only enabled for GCC versions < 4.5 (without any mention of why). Enable this fix for all GCC versions.

New spkg: http://boxen.math.washington.edu/home/jdemeyer/spkg/cvxopt-1.1.4.p1.spkg

Depends on #12519

Upstream: Fixed upstream, in a later stable release.

CC: @dimpase @vbraun

Component: packages: standard

Keywords: Lion Darwin cvxopt atlas blas Solaris

Author: Jeroen Demeyer, John Palmieri

Reviewer: Jeroen Demeyer, John Palmieri

Merged: sage-5.0.beta13

Issue created by migration from https://trac.sagemath.org/ticket/12011

The text was updated successfully, but these errors were encountered:

jhpalmieri · 2012-02-01T06:41:45Z

comment:2

In a related matter, we should patch spkg-check to exit with a nonzero return value if tests fail. We can either exit immediately once there's a failure, or we can run all tests and then exit. I'm not sure which of these is better. In general I would suggest the latter, but since it's actually hanging on OS X Lion (after at least one failed test), I might lean toward the former...

Also, should we upgrade to version 1.1.4?

jdemeyer · 2012-02-16T11:09:38Z

comment:3

See #12519 for a new cvxopt spkg. I fixed spkg-check such that it now actually exists when there is a failure. However, it still fails on OS X Lion.

jhpalmieri · 2012-02-20T04:36:37Z

comment:4

Dima: any ideas about the self-test failures?

dimpase · 2012-02-20T10:35:11Z

comment:5

Replying to @jhpalmieri:

Dima: any ideas about the self-test failures?

I just asked upstream here.
Is there a way to get access to 10.7 for me?

Dima

dimpase · 2012-02-20T10:35:11Z

Upstream: Reported upstream. Little or no feedback.

dimpase · 2012-02-20T12:42:05Z

comment:6

Replying to @dimpase:

Replying to @jhpalmieri:

Dima: any ideas about the self-test failures?

I just asked upstream here.
Is there a way to get access to 10.7 for me?

Dima

here it is written that it's probably a bug in Apple's blas/lapack. A workaround is to use Atlas. I have no way to check this at the moment, but it looks very reasonable to me.

It would be great if you tested this with the newest Xcode...

jhpalmieri · 2012-02-20T23:19:59Z

comment:7

Replying to @dimpase:

here it is written that it's probably a bug in Apple's blas/lapack. A workaround is to use Atlas. I have no way to check this at the moment, but it looks very reasonable to me.

Do you mean Apple's ATLAS? We don't build ATLAS on Mac OS X, we just use the system's libraries. What should I do to try this out? Can I just make some changes in the spkg-install script?

It would be great if you tested this with the newest Xcode...

I've tried this with various versions of Xcode, including the newly released 4.3, and I get failures in all of them.

jhpalmieri · 2012-02-20T23:36:03Z

comment:8

By the way, it might make sense to base further discussion on the new cvxopt package at #12519.

dimpase · 2012-02-21T03:15:20Z

comment:9

Replying to @jhpalmieri:

Replying to @dimpase:

here it is written that it's probably a bug in Apple's blas/lapack. A workaround is to use Atlas. I have no way to check this at the moment, but it looks very reasonable to me.

Do you mean Apple's ATLAS? We don't build ATLAS on Mac OS X, we just use the system's libraries. What should I do to try this out? Can I just make some changes in the spkg-install script?

Well, no. We need to build ATLAS spkg (or get some other non-Apple lapack/blas somehow) and use it.
Actually, if I recall right, this (native lapack/blas) is an artefact of the PPC support. We can perfectly well use Atlas
on x86, right?

It would be great if you tested this with the newest Xcode...

I've tried this with various versions of Xcode, including the newly released 4.3, and I get failures in all of them.

jdemeyer · 2012-03-06T10:46:26Z

Changed keywords from Lion Darwin cvxopt to Lion Darwin cvxopt atlas blas

dimpase · 2012-03-06T11:29:24Z

comment:12

doesnt work on OSX10.6:

Calling sage-spkg on 'http://boxen.math.washington.edu/home/jdemeyer/spkg/atlas-3.8.4.p2.spkg'
atlas-3.8.4.p2
Machine:
Darwin nash 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun  7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
Deleting directories from past builds of previous/current versions of atlas-3.8.4.p2
/usr/local/src/sage/sage-4.8.alpha4/local/bin/sage-spkg: file atlas-3.8.4.p2 does not exist
Attempting to download it.
http://boxen.math.washington.edu/home/jdemeyer/spkg/atlas-3.8.4.p2.spkg --> atlas-3.8.4.p2.spkg
[..................................................]
Extracting package /usr/local/src/sage/sage-4.8.alpha4/spkg/optional/atlas-3.8.4.p2.spkg ...
-rw-r--r--  1 dima  staff  2866339 Mar  6 19:27 /usr/local/src/sage/sage-4.8.alpha4/spkg/optional/atlas-3.8.4.p2.spkg
Finished extraction
****************************************************
Host system
uname -a:
Darwin nash 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun  7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
****************************************************
****************************************************
CC Version
gcc -v
Using built-in specs.
Target: i686-apple-darwin10
Configured with: /var/tmp/gcc/gcc-5666.3~6/src/configure --disable-checking --enable-werror --prefix=/usr --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin10 --program-prefix=i686-apple-darwin10- --host=x86_64-apple-darwin10 --target=i686-apple-darwin10 --with-gxx-include-dir=/include/c++/4.2.1
Thread model: posix
gcc version 4.2.1 (Apple Inc. build 5666) (dot 3)
****************************************************
Traceback (most recent call last):
  File "./spkg-install", line 13, in <module>
    from configuration import conf, cp, try_run, edit_in_place
  File "/usr/local/src/sage/sage-4.8.alpha4/spkg/build/atlas-3.8.4.p2/configuration.py", line 108, in <module>
    conf['release'] = subprocess.check_output(['uname', '-r'])
AttributeError: 'module' object has no attribute 'check_output'

real	0m0.092s
user	0m0.038s
sys	0m0.044s
************************************************************************
Error installing package atlas-3.8.4.p2
************************************************************************
Please email sage-devel (http://groups.google.com/group/sage-devel)
explaining the problem and including the relevant part of the log file
  /usr/local/src/sage/sage-4.8.alpha4/spkg/logs/atlas-3.8.4.p2.log
Describe your computer, operating system, etc.
If you want to try to fix the problem yourself, *don't* just cd to
/usr/local/src/sage/sage-4.8.alpha4/spkg/build/atlas-3.8.4.p2 and type 'make' or whatever is appropriate.
Instead, the following commands setup all environment variables
correctly and load a subshell for you to debug the error:
  (cd '/usr/local/src/sage/sage-4.8.alpha4/spkg/build/atlas-3.8.4.p2' && '/usr/local/src/sage/sage-4.8.alpha4/sage' -sh)
When you are done debugging, you can type "exit" to leave the subshell.
************************************************************************
Error: Failed to install package 'atlas-3.8.4.p2'.
nash:sage-5.0.beta6 dima$

jdemeyer · 2012-03-06T11:32:32Z

comment:13

You're using a too old version of Sage.

dimpase · 2012-03-06T11:59:26Z

comment:14

Replying to @jdemeyer:

You're using a too old version of Sage.

/me bangs head on the kbd...

kcrisman · 2012-03-06T14:44:33Z

comment:15

Does this only build ATLAS in Lion, or all Mac? Seems silly to do it from scratch on others, maybe we can check Dima's program mentioned at #12519 on older versions...

jdemeyer · 2012-03-06T14:46:14Z

comment:16

Replying to @kcrisman:

Does this only build ATLAS in Lion, or all Mac?

Only on Lion.

jdemeyer · 2012-03-06T16:05:08Z

comment:17

After more than 3 hours, it finally managed to fail to build on sqrt5:

ATLAS install complete.  Examine
ATLAS/bin/<arch>/INSTALL_LOG/SUMMARY.LOG for details.
make -j1 clean
rm -f *.o x* config?.out *core*
rm: xarchinfo_freebsd.dSYM: is a directory
rm: xarchinfo_x86.dSYM: is a directory
rm: xconfig.dSYM: is a directory
rm: xprobe_gas_x8632.dSYM: is a directory
rm: xprobe_gas_x8664.dSYM: is a directory
rm: xspew.dSYM: is a directory
make[1]: *** [clean] Error 1
make: *** [build] Error 2
ATLAS failed to build, possibly because of a loaded system.
Traceback (most recent call last):
  File "./spkg-install", line 473, in <module>
    assert rc==0, 'Failed to build ATLAS.'
AssertionError: Failed to build ATLAS.

real    202m1.081s
user    182m33.861s
sys     10m48.195s

Weird that it reported that the build succeeded, but still it failed somehow.

jdemeyer · 2012-03-06T16:24:24Z

comment:18

Replace a "rm -f" by "rm -rf" probably will solve this problem.

This explain the dSYM directories: http://stackoverflow.com/questions/584825/dsym-directories-while-compiling-c-code-in-macos

kcrisman · 2012-03-06T16:25:12Z

comment:19

Aah, you beat me by five seconds! I concur.

jdemeyer · 2012-03-06T16:52:27Z

comment:20

I put up a new version hopefully fixing this, currently building on sqrt5...

jdemeyer · 2012-03-06T17:02:41Z

comment:21

:-(

ATLAS install complete.  Examine
ATLAS/bin/<arch>/INSTALL_LOG/SUMMARY.LOG for details.
make -j1 clean
# Xcode (OS X) may create directories like xspew.dSYM.
# Add -r option to delete also these.
rm -rf *.o x* config?.out *core*
Finished building ATLAS core.
Finished building ATLAS core.
Running make -j1 shared cshared
rm -f libatlas.so liblapack.so
make -j1 libatlas.so liblapack.so libf77blas.so libcblas.so liblapack.so
ld -melf_x86_64 -shared -soname libatlas.so -o libatlas.so \
        --whole-archive libatlas.a --no-whole-archive -lc -lpthread -lm
ld: unknown option: -melf_x86_64
make[1]: *** [libatlas.so] Error 1
make: *** [shared] Error 2
Traceback (most recent call last):
  File "./spkg-install", line 480, in <module>
    assert rc==0, 'Building shared ATLAS library failed.'
AssertionError: Building shared ATLAS library failed.

jdemeyer · 2012-03-10T10:06:53Z

Dependencies: #12519

jdemeyer · 2012-03-28T08:19:05Z

comment:55

Any news from upstream CVXOPT?

dimpase · 2012-03-31T06:52:33Z

Changed upstream from Reported upstream. Developers acknowledge bug. to Fixed upstream, in a later stable release.

dimpase · 2012-03-31T06:52:33Z

comment:56

Replying to @jdemeyer:

Any news from upstream CVXOPT?

They have released 1.1.5 fixing this issue.
Any takers to update the cvxopt spkg ? I, right now, have too many other things to do.

jhpalmieri · 2012-03-31T18:41:02Z

comment:57

Replying to @dimpase:

They have released 1.1.5 fixing this issue.
Any takers to update the cvxopt spkg ? I, right now, have too many other things to do.

I'm working on it. More details in a few minutes.

jhpalmieri · 2012-03-31T18:55:44Z

comment:58

I'm posting a new spkg, but not a perfect one. The flaw is that it is version 1.1.4 rather than 1.1.5. I looked at 1.1.4 and 1.1.5, and (using diff -r ...) I found two changes in the code: the file src/C/dense.c was modified, and I imported that change into the 1.1.4 spkg. With this change, it builds and self-tests pass on OS X Lion, and also on sage.math and OpenSolaris. So that's the version I'm posting here.

The other change was to setup.py, and I don't understand enough of the changes in our patch to modify it. That is, setup.py in version 1.1.4 was almost identical to the version in 1.1.3, so it was easy to rebase our patch, but I couldn't figure out how to rebase to 1.1.5. C'est la vie.

jhpalmieri · 2012-03-31T18:55:44Z

Changed author from Jeroen Demeyer to Jeroen Demeyer, John Palmieri

jhpalmieri · 2012-03-31T18:56:19Z

patch for cvxopt spkg; for review only

jdemeyer · 2012-04-03T13:01:23Z

comment:59

Attachment: trac_12011-cvxopt.patch.gz

Unfortunately, I don't have access anymore to a OS X 10.7 machine.

jdemeyer · 2012-04-03T13:12:29Z

comment:60

This spkg is included in http://boxen.math.washington.edu/home/jdemeyer/release/sage-5.0.beta12-gcc/ for testing.

jdemeyer · 2012-04-03T13:53:15Z

Diff for the cvxopt spkg (p0->p1). For review only

jdemeyer · 2012-04-03T14:00:26Z

Reviewer: Jeroen Demeyer

jdemeyer · 2012-04-03T14:00:26Z

Changed keywords from Lion Darwin cvxopt atlas blas to Lion Darwin cvxopt atlas blas Solaris

jdemeyer · 2012-04-03T14:00:26Z

comment:61

Attachment: cvxopt-1.1.4.p1.diff.gz

Reviewer spkg: http://boxen.math.washington.edu/home/jdemeyer/spkg/cvxopt-1.1.4.p1.spkg

Changes: attachment: cvxopt-1.1.4.p1.diff

John, if you confirm that my spkg builds with SAGE_CHECK=yes on OS X Lion and you agree with my changes, you may set this to positive review.

jhpalmieri · 2012-04-03T18:44:30Z

Changed reviewer from Jeroen Demeyer to Jeroen Demeyer, John Palmieri

jhpalmieri · 2012-04-03T18:44:30Z

comment:62

Changes look good and this passes tests on OS X Lion (and OpenSolaris, which I tried just for fun).

jdemeyer · 2012-04-04T19:27:35Z

Merged: sage-5.0.beta13

jhpalmieri added this to the sage-4.8 milestone Nov 11, 2011

jhpalmieri added c: packages: standard labels Nov 11, 2011

jdemeyer modified the milestones: sage-4.8, sage-5.0 Jan 2, 2012

dimpase added the s: needs info label Feb 20, 2012

This comment has been minimized.

Sign in to view

jdemeyer changed the title ~~OS X Lion: cvxopt fails self tests~~ Build ATLAS on OS X Lion Mar 6, 2012

This comment has been minimized.

Sign in to view

jhpalmieri added s: needs review and removed s: needs info labels Mar 31, 2012

This comment has been minimized.

Sign in to view

jdemeyer changed the title ~~Fix illegal BLAS call in cvxopt~~ cvxopt: fix illegal BLAS call and fix Solaris build Apr 3, 2012

jdemeyer added the c: porting: solaris label Apr 3, 2012

jhpalmieri added s: positive review and removed s: needs review labels Apr 3, 2012

jdemeyer removed the s: positive review label Apr 4, 2012

jdemeyer closed this as completed Apr 4, 2012

This was referenced May 3, 2012

Metaticket: build Sage on OS X 10.7 Lion #11881

Closed

OS X Lion: pari fails self tests #12315

Closed

dimpase mentioned this issue Mar 9, 2012

cvxopt should not add -lcblas and -latlas on Darwin #12519

Closed

cvxopt: fix illegal BLAS call and fix Solaris build #12011

cvxopt: fix illegal BLAS call and fix Solaris build #12011

Comments

jhpalmieri commented Nov 11, 2011

jhpalmieri commented Feb 1, 2012

jdemeyer commented Feb 16, 2012

jhpalmieri commented Feb 20, 2012

dimpase commented Feb 20, 2012

dimpase commented Feb 20, 2012

dimpase commented Feb 20, 2012

jhpalmieri commented Feb 20, 2012

jhpalmieri commented Feb 20, 2012

dimpase commented Feb 21, 2012

jdemeyer commented Mar 6, 2012

This comment has been minimized.

This comment has been minimized.

dimpase commented Mar 6, 2012

jdemeyer commented Mar 6, 2012

dimpase commented Mar 6, 2012

kcrisman commented Mar 6, 2012

jdemeyer commented Mar 6, 2012

jdemeyer commented Mar 6, 2012

jdemeyer commented Mar 6, 2012

kcrisman commented Mar 6, 2012

jdemeyer commented Mar 6, 2012

jdemeyer commented Mar 6, 2012

jdemeyer commented Mar 10, 2012

jdemeyer commented Mar 28, 2012

dimpase commented Mar 31, 2012

dimpase commented Mar 31, 2012

jhpalmieri commented Mar 31, 2012

jhpalmieri commented Mar 31, 2012

jhpalmieri commented Mar 31, 2012

This comment has been minimized.

jhpalmieri commented Mar 31, 2012

jdemeyer commented Apr 3, 2012

jdemeyer commented Apr 3, 2012

jdemeyer commented Apr 3, 2012

jdemeyer commented Apr 3, 2012

This comment has been minimized.

jdemeyer commented Apr 3, 2012

jdemeyer commented Apr 3, 2012

jhpalmieri commented Apr 3, 2012

jhpalmieri commented Apr 3, 2012

jdemeyer commented Apr 4, 2012