Get libs.mwrank to compile on OS X 10.4

[kcrisman]

gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -DNTL_ALL -I/Users/student/Desktop/sage-6.2.beta4/local/include -I/Users/student/Desktop/sage-6.2.beta4/local/include/csage -I/Users/student/Desktop/sage-6.2.beta4/src -I/Users/student/Desktop/sage-6.2.beta4/src/sage/ext -I/Users/student/Desktop/sage-6.2.beta4/local/include -I/Users/student/Desktop/sage-6.2.beta4/local/include/csage -I/Users/student/Desktop/sage-6.2.beta4/src -I/Users/student/Desktop/sage-6.2.beta4/src/sage/ext -I/Users/student/Desktop/sage-6.2.beta4/local/include/python2.7 -c sage/libs/mwrank/mwrank.c -o build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.o -fno-strict-aliasing -w
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -DNTL_ALL -I/Users/student/Desktop/sage-6.2.beta4/local/include -I/Users/student/Desktop/sage-6.2.beta4/local/include/csage -I/Users/student/Desktop/sage-6.2.beta4/src -I/Users/student/Desktop/sage-6.2.beta4/src/sage/ext -I/Users/student/Desktop/sage-6.2.beta4/local/include -I/Users/student/Desktop/sage-6.2.beta4/local/include/csage -I/Users/student/Desktop/sage-6.2.beta4/src -I/Users/student/Desktop/sage-6.2.beta4/src/sage/ext -I/Users/student/Desktop/sage-6.2.beta4/local/include/python2.7 -c sage/libs/mwrank/wrap.cc -o build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/wrap.o -fno-strict-aliasing -w
g++ -bundle -undefined dynamic_lookup -L/Users/student/Desktop/sage-6.2.beta4/local/lib build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.o build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/wrap.o -L/Users/student/Desktop/sage-6.2.beta4/local/lib -lcsage -lec -lntl -lpari -lgmp -lgmpxx -lstdc++ -lm -lstdc++ -lntl -o build/lib.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.so
/usr/bin/ld: warning can't open dynamic library: libflint.dylib referenced from: /Users/student/Desktop/sage-6.2.beta4/local/lib/libec.dylib (checking for undefined symbols may be affected) (No such file or directory, errno = 2)
/usr/bin/ld: Undefined symbols:
_nmod_mat_clear referenced from libec expected to be defined in libflint.dylib
_nmod_mat_init referenced from libec expected to be defined in libflint.dylib
_nmod_mat_rref referenced from libec expected to be defined in libflint.dylib
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1

See lots of discussion at this sage-release thread. Best guesses are either this Flint bug and leif's idea that there is a missing linker flag on Darwin (see the sage-release thread).

---

This is not eclib's fault, but due to incomplete / wrong "install_names" (meant to be absolute paths) in libflint.dylib (and also libpari-gmp.dylib).

On Darwin 10 (MacOS X 10.6) and later, apparently the linker is smarter (or more tolerant), and does find the indirectly used libraries in $SAGE_LOCAL/lib/, despite their wrong install_names.

We can solve this by upgrading flint to 2.4.3 which includes the necessary fix.

New sources:

• http://www.flintlib.org/flint-2.4.3.tar.gz

CC: @nexttime @vbraun @jpflori

Component: packages: standard

Keywords: Darwin 8 linker error install_name libflint.dylib

Author: François Bissey

Branch/Commit: 6c25897

Reviewer: Karl-Dieter Crisman, Volker Braun

Issue created by migration from https://trac.sagemath.org/ticket/16017

[kcrisman]

comment:1

Leif's idea seems to have been right, though I don't know that setting the DYLD_ whatever path or rpath is any easier that what seemed to do it for me - just adding an explicit dependency in src/module_list.py, as we often do for Cygwin, which is more finicky. From the thread:

---

But the problem is that somehow the linking isn't happening properly in sage/libs/mwrank, and indeed -lflint isn't in the line

g++ -bundle -undefined dynamic_lookup -L/Users/student/Desktop/sage-6.2.beta4/local/lib build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.o build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/wrap.o -L/Users/student/Desktop/sage-6.2.beta4/local/lib -lcsage -lec -lntl -lpari -lgmp -lgmpxx -lstdc++ -lm -lstdc++ -lntl -o build/lib.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.so

Maybe could I add something to src/module_list.py or whatever the appropriate file is now to force -lflint as well? On Cygwin we often had to be more explicit because of something analogous, if I recall that correctly. I'm going to try this just to see...

It seems to work! At least, things continue compiling. On Cygwin that was always the sign of victory, though :)

So Leif's idea was probably the right one. I'll see if things finish up first and then run tests - John, any files I should run tests on first to make sure this did indeed compile and link correctly?

However, I don't know the right fix. I don't mind including extra things in module_list.py since probably Cygwin likes them too, but perhaps there is a better fix, or maybe there's a reason that's a bad fix, or something.

[kcrisman]

comment:2

Also required for sage.libs.cremona extensions. Given that on Cygwin we will probably need this too, I'm suggesting that this is the right solution now.

[nexttime]

comment:3

I'm pretty curious whether -- with the "superfluous" -lflint added -- you're still getting the weird

/usr/bin/ld: warning can't open dynamic library: libflint.dylib referenced from: /Users/student/Desktop/sage-6.2.beta4/local/lib/libec.dylib (checking for undefined symbols may be affected) (No such file or directory, errno = 2)

The addition is IMHO just a work-around (on Darwin 8; still probably necessary on Cygwin); it doesn't address the root of the problem (provided we can trust the linker messages). (Cf. my recent comment on the Sage 6.2.beta4 thread on sage-release.)

[nexttime]

comment:4

... in other words:

Where the heck does the linker try to find the libflint.dylib libec.dylib refers to, and does libec.dylib really refer to $SAGE_ROOT/local/lib/libflint.dylib (modulo symlinks to versioned versions of the libraries)?

Could Apple's otool enlighten us? (Not sure whether passing -t to the Darwin linker would.)

[kcrisman]

comment:5

No, no warnings if I use that. I really think it just doesn't know where to find it. This happened on Cygwin all the time, like I said. Yes, it's certainly just a workaround, but one that I can't really see a downside to. I have no idea where it is looking, and I have pretty much run out of energy on this now that I found what I think it is an acceptable solution. Not that I don't want a better one, but my sense is there may not be one that doesn't involve special casing a -lflint for old Macs for no particular reason, or that doesn't involve asking John to do something annoying with eclib.

[kiwifb]

comment:6

Try using https://github.com/cschwan/sage-on-gentoo/blob/master/sci-mathematics/flint/files/flint-2.4-makefile.patch in the patches folder for flint. Then in spkg-install

export SHARE_FLAGS="-install_name ${SAGE_LOCAL}/lib/libflint.dylib"

before $MAKE
Rebuild flint, eclib and see if that works. If it does we'll put that in a better shape.

[kcrisman]

comment:7

Thanks, that sounds fine too if it works, but I won't be able to try this until Monday.

Since we have several workarounds now, and since (for now) this is still a supported platform, I'm making this a blocker for 6.2.

[nexttime]

comment:8

leif@bsd:~/Sage/sage-6.2.beta5$ otool -DL local/lib/libec.dylib 
local/lib/libec.dylib:
	/Users/leif/Sage/sage-6.2.beta5/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	libflint.dylib (compatibility version 0.0.0, current version 0.0.0)
	/Users/leif/Sage/sage-6.2.beta5/local/lib/libntl.2.dylib (compatibility version 3.0.0, current version 3.0.0)
	/Users/leif/Sage/sage-6.2.beta5/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib (compatibility version 2.5.0, current version 2.5.5)
	/Users/leif/Sage/sage-6.2.beta5/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.17.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)
	/Users/leif/Sage/sage-6.2.beta5/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

Interesting... (Note the recorded path to the PARI library!)

But we (also) already have -lpari on the command line for linking the mwrank Python extension module (which I think by the way shouldn't be necessary either, as it isn't directly used IIRC):

g++ -bundle -undefined dynamic_lookup -L/Users/leif/Sage/sage-6.2.beta5/local/lib build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.o build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/wrap.o -L/Users/leif/Sage/sage-6.2.beta5/local/lib -lcsage -lec -lntl -lpari -lgmp -lgmpxx -lstdc++ -lm -lstdc++ -lntl -o build/lib.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.so

And FWIW, in the build on bsd.math, libtool also drops the -rpath given to it when linking libec.dylib (or, more precisely, libec.0.dylib).

[nexttime]

comment:9

For completeness, this is how libec.0.dylib gets linked on bsd.math (looks the same as on Karl-Dieter's MacOS X 10.4 PPC box):

/bin/sh ../libtool --tag=CXX   --mode=link g++  -g -O3  -no-undefined -L/Users/leif/Sage/sage-6.2.beta5/local/lib -L/Users/leif/Sage/sage-6.2.beta5/local/lib -L/Users/leif/Sage/sage-6.2.beta5/local/lib   -o libec.la -rpath /Users/leif/Sage/sage-6.2.beta5/local/lib interface.lo unimod.lo arith.lo marith.lo gpslave.lo compproc.lo vector.lo matrix.lo subspace.lo kbessel.lo mvector.lo mmatrix.lo msubspace.lo svector.lo smatrix.lo smatrix_elim.lo xsplit.lo conic.lo legendre.lo quadratic.lo illl.lo hilbert.lo timer.lo cubic.lo polys.lo realroots.lo parifact.lo p2points.lo gf.lo xsplit_data.lo threadpool.lo logger.lo curve.lo curvedata.lo curvered.lo points.lo cperiods.lo isogs.lo heights.lo mwprocs.lo lambda.lo sifter.lo sieve_search.lo htconst.lo egr.lo saturate.lo divpol.lo pointsmod.lo curvemod.lo ffmod.lo tlss.lo elog.lo getcurve.lo mequiv.lo mrank1.lo mlocsol.lo mglobsol.lo mquartic.lo mrank2.lo qc.lo sqfdiv.lo version.lo minim.lo reduce.lo transform.lo desc2.lo bitspace.lo GetOpt.lo twoadic.lo descent.lo newforms.lo symb.lo homspace.lo cusp.lo oldforms.lo fixc6.lo periods.lo moddata.lo pcprocs.lo nfd.lo curvesort.lo  -lflint -lntl -lpari
libtool: link: g++ -dynamiclib  -o .libs/libec.0.dylib  .libs/interface.o .libs/unimod.o .libs/arith.o .libs/marith.o .libs/gpslave.o .libs/compproc.o .libs/vector.o .libs/matrix.o .libs/subspace.o .libs/kbessel.o .libs/mvector.o .libs/mmatrix.o .libs/msubspace.o .libs/svector.o .libs/smatrix.o .libs/smatrix_elim.o .libs/xsplit.o .libs/conic.o .libs/legendre.o .libs/quadratic.o .libs/illl.o .libs/hilbert.o .libs/timer.o .libs/cubic.o .libs/polys.o .libs/realroots.o .libs/parifact.o .libs/p2points.o .libs/gf.o .libs/xsplit_data.o .libs/threadpool.o .libs/logger.o .libs/curve.o .libs/curvedata.o .libs/curvered.o .libs/points.o .libs/cperiods.o .libs/isogs.o .libs/heights.o .libs/mwprocs.o .libs/lambda.o .libs/sifter.o .libs/sieve_search.o .libs/htconst.o .libs/egr.o .libs/saturate.o .libs/divpol.o .libs/pointsmod.o .libs/curvemod.o .libs/ffmod.o .libs/tlss.o .libs/elog.o .libs/getcurve.o .libs/mequiv.o .libs/mrank1.o .libs/mlocsol.o .libs/mglobsol.o .libs/mquartic.o .libs/mrank2.o .libs/qc.o .libs/sqfdiv.o .libs/version.o .libs/minim.o .libs/reduce.o .libs/transform.o .libs/desc2.o .libs/bitspace.o .libs/GetOpt.o .libs/twoadic.o .libs/descent.o .libs/newforms.o .libs/symb.o .libs/homspace.o .libs/cusp.o .libs/oldforms.o .libs/fixc6.o .libs/periods.o .libs/moddata.o .libs/pcprocs.o .libs/nfd.o .libs/curvesort.o   -L/Users/leif/Sage/sage-6.2.beta5/local/lib -lflint /Users/leif/Sage/sage-6.2.beta5/local/lib/libntl.dylib -L/Users/leif/Sage/sage-6.2.beta5/local/var/tmp/sage/build/gcc-4.7.3.p1/gcc-build/x86_64-apple-darwin10.8.0/libstdc++-v3/src -L/Users/leif/Sage/sage-6.2.beta5/local/var/tmp/sage/build/gcc-4.7.3.p1/gcc-build/x86_64-apple-darwin10.8.0/libstdc++-v3/src/.libs -lpari  -O3   -install_name  /Users/leif/Sage/sage-6.2.beta5/local/lib/libec.0.dylib -compatibility_version 1 -current_version 1.0 -Wl,-single_module

So indeed the (missing or weird) install_names seem to be the problem. (Or libtool... ;-) )

[nexttime]

comment:10

Replying to @nexttime:

(Note the recorded path to the PARI library!)

But we (also) already have -lpari on the command line for linking the mwrank Python extension module (which I think by the way shouldn't be necessary either, as it isn't directly used IIRC) [...]

And if I remove pari from the allegedly needed libraries of the mwrank Python extension module (in module_list.py), this also works on bsd.math:

leif@bsd:~/Sage/sage-6.2.beta5$ touch src/sage/libs/mwrank/mwrank.pyx 
leif@bsd:~/Sage/sage-6.2.beta5$ ./sage -b
scons: `install' is up to date.
Updating Cython code....
Compiling sage/libs/mwrank/mwrank.pyx because it changed.
Cythonizing sage/libs/mwrank/mwrank.pyx
Finished compiling Cython code (time = 4.62841916084 seconds)
running install
running build
running build_py
running build_ext
building 'sage.libs.mwrank.mwrank' extension
Executing 1 command (using 1 thread)
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -DNTL_ALL -I/Users/leif/Sage/sage-6.2.beta5/local/include -I/Users/leif/Sage/sage-6.2.beta5/local/include/csage -I/Users/leif/Sage/sage-6.2.beta5/src -I/Users/leif/Sage/sage-6.2.beta5/src/sage/ext -I/Users/leif/Sage/sage-6.2.beta5/local/include -I/Users/leif/Sage/sage-6.2.beta5/local/include/csage -I/Users/leif/Sage/sage-6.2.beta5/src -I/Users/leif/Sage/sage-6.2.beta5/src/sage/ext -I/Users/leif/Sage/sage-6.2.beta5/local/include/python2.7 -c sage/libs/mwrank/mwrank.c -o build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.o -fno-strict-aliasing -w
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -DNTL_ALL -I/Users/leif/Sage/sage-6.2.beta5/local/include -I/Users/leif/Sage/sage-6.2.beta5/local/include/csage -I/Users/leif/Sage/sage-6.2.beta5/src -I/Users/leif/Sage/sage-6.2.beta5/src/sage/ext -I/Users/leif/Sage/sage-6.2.beta5/local/include -I/Users/leif/Sage/sage-6.2.beta5/local/include/csage -I/Users/leif/Sage/sage-6.2.beta5/src -I/Users/leif/Sage/sage-6.2.beta5/src/sage/ext -I/Users/leif/Sage/sage-6.2.beta5/local/include/python2.7 -c sage/libs/mwrank/wrap.cc -o build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/wrap.o -fno-strict-aliasing -w
g++ -bundle -undefined dynamic_lookup -L/Users/leif/Sage/sage-6.2.beta5/local/lib build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.o build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/wrap.o -L/Users/leif/Sage/sage-6.2.beta5/local/lib -lcsage -lec -lntl -lgmp -lgmpxx -lstdc++ -lm -lstdc++ -lntl -o build/lib.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.so
Time to execute 1 command: 4.69843101501 seconds
Total time spent compiling C/C++ extensions:  4.88040590286 seconds.

I guess doing so would trigger another "no such file or directory" warning on Karl-Dieter's machine. MacOS X 10.6's linker is simply "smarter"...

[kiwifb]

comment:11

So we may also want to look at pari because it put the build time location rather than final location in install name. Not a problem if you tell sage to keep the build stuff I guess.

[kiwifb]

comment:12

I actually have a patch for pari's installname that I made for pari 2.5.0 and it is still in the gentoo and sage-on-gentoo trees, I guess I should try to it merged in pari proper.
https://github.com/cschwan/sage-on-gentoo/blob/master/sci-mathematics/pari/files/pari-2.5.0-macos.patch
As for Karl-Dieter's problem we should be able to see if fixing the installname solves his problem without rebuilding flint and eclib.
we can use

install_name_tool -change "oldname" "newname" filename

We can alter libflint.dylib and libec.dylib with the appropriate values and see if sage compile.

[nexttime]

comment:13

Replying to @kiwifb:

I actually have a patch for pari's installname that I made for pari 2.5.0 and it is still in the gentoo and sage-on-gentoo trees, I guess I should try to it merged in pari proper.
https://github.com/cschwan/sage-on-gentoo/blob/master/sci-mathematics/pari/files/pari-2.5.0-macos.patch
As for Karl-Dieter's problem we should be able to see if fixing the installname solves his problem without rebuilding flint and eclib.
we can use

install_name_tool -change "oldname" "newname" filename

We can alter libflint.dylib and libec.dylib with the appropriate values and see if sage compile.

Yep, but he'd have to rebuild eclib (after changing libflint's [and perhaps libpari's] install_name) such that the proper dependency name(s) get(s) recorded in libec.0.dylib; I don't think you can change these with install_name_tool.

Any idea why libtool ignores -rpath (on both MacOS X 10.4 and 10.6)?

[kiwifb]

comment:14

First yes you can.

Mirage:lib fbissey$ otool -L libec.0.dylib 
libec.0.dylib:
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	libflint.dylib (compatibility version 0.0.0, current version 0.0.0)
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libntl.2.dylib (compatibility version 3.0.0, current version 3.0.0)
	/Users/fbissey/build/sage-6.2.beta4/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib (compatibility version 2.5.0, current version 2.5.5)
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.17.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
Mirage:lib fbissey$ install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libec.0.dylib 
Mirage:lib fbissey$ otool -L libec.0.dylib 
libec.0.dylib:
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib (compatibility version 0.0.0, current version 0.0.0)
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libntl.2.dylib (compatibility version 3.0.0, current version 3.0.0)
	/Users/fbissey/build/sage-6.2.beta4/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib (compatibility version 2.5.0, current version 2.5.5)
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.17.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
Mirage:lib fbissey$ 

About libtool, not sure. I always felt rpath was redundant with the install_name mechanism but that's not really an answer to your question.

[nexttime]

comment:15

Replying to @kiwifb:

First yes you can.

Cool. \o/

About libtool, not sure. I always felt rpath was redundant with the install_name mechanism but that's not really an answer to your question.

Well, by default install_name is an absolute path. You can prepend @rpath/ (and other flavours) to the install_name to make it relative to paths set by -rpath, similar to $ORIGIN in an ELF object's DT_RUNPATH. But I'm not a Darwin linker (nor Mach-O) expert; I'd expect libtool to know how to do it though. On the other hand, libtool did convert some of the -L/path/to/foo/ -lfoo to absolute filenames on the linker command line (when linking libec.0.dylib), presumably exactly [to] those with a correct, absolute install_name.

[kcrisman]

comment:16

Yikes! As long as I have a specific thing to try out on Monday I'll be happy - if you guys can agree on what I should try, I don't have infinite time with that machine ;-)

[kiwifb]

comment:17

Try the equivalent for your machine of:

install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libec.0.dylib 
install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libflint.dylib 

in $SAGE_LOCAL/lib. Don't forget to replace the path for SAGE_LOCAL with what's correct for you.
Once that's done you can try to build sage again and see if it will go past that point.
If it does we'll make the appropriate patch.

[nexttime]

comment:18

Replying to @kcrisman:

Yikes! As long as I have a specific thing to try out on Monday I'll be happy - if you guys can agree on what I should try, I don't have infinite time with that machine ;-)

There are plenty of things you could try... :-)

But I think kiwifb and me agree on that you could first try to:

• Restore module_list.py (at least remove flint, optionally also pari) from the "needed" libraries of the libs.mwrank Python extension module -- to test whether it works for one of the affected modules (you had to change about five IIRC)

• Make sure libflint's install_name is still just libflint.dylib (otherwise install_name_tool will give an error):

$ cd $SAGE_ROOT
$ otool -D local/lib/libflint.dylib
local/lib/libflint.dylib:
libflint.dylib

• Change the install_name (here called "id") to the library's absolute path:

$ install_name_tool -id `pwd`/local/lib/libflint.dylib local/lib/libflint.dylib

• Then either rebuild/reinstall eclib (which should now record libflint's absolute path), or -- simpler -- just change the recorded name (for FLINT) in libec.0.dylib manually:

$ otool -L local/lib/libec.0.dylib  # just to check libflint's name there
local/lib/libec.0.dylib:
	<path to>/sage-6.2.beta5/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	libflint.dylib (compatibility version 0.0.0, current version 0.0.0)
        ...

$ install_name_tool -change libflint.dylib `pwd`/local/lib/libflint.dylib local/lib/libec.0.dylib  # now really change the name

$ otool -L local/lib/libec.0.dylib  # just to verify it changed
local/lib/libec.0.dylib:
	<path to>/sage-6.2.beta5/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	<path to>/sage-6.2.beta5/local/lib/libflint.dylib (compatibility version 0.0.0, current version 0.0.0)
        ...

• Pretend file modification to trigger the rebuild of the module(s):

$ touch src/sage/libs/mwrank/mwrank.pyx  # and more if you like

$ ./sage -b  # and now rebuild

The linker should now be able to locate libflint.dylib referenced by libec.dylib.

Optionally, repeat the steps analogously for pari (which in contrast is originally listed as a needed lib in module_list.py, so you'd have to really remove it rather than restoring the state of vanilla beta4 or beta5); just libpari's original install_name is a bit more complicated (with a non-existing temporary path):

 $ otool -D local/lib/libpari-gmp.dylib 
local/lib/libpari-gmp.dylib:
<path to>/sage-6.2.beta5/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib

(libpari.dylib is a symbolic link to libpari-gmp.dylib.)

When you change the name (again in libec.0.dylib), you have to specify the full original (invalid) path:

$ install_name_tool -change /Users/leif/Sage/sage-6.2.beta5/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib `pwd`/local/lib/libpari-gmp.dylib local/lib/libec.0.dylib

(Just as an example -- note that the invalid path will differ on a PowerPC, and of course your $SAGE_ROOT.)

[nexttime]

comment:19

Just for the record: I started writing before François replied... 8)

But we (still) agree on what you could try.

[kiwifb]

comment:20

Yes we do. Just using install_name_tool means that we can test that theory without rebuilding flint and eclib, which would take some time on that machine.

[nexttime]

comment:21

Replying to @kiwifb:

Try the equivalent for your machine of:

install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libec.0.dylib 
install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libflint.dylib 

in $SAGE_LOCAL/lib. Don't forget to replace the path for SAGE_LOCAL with what's correct for you.

Note that the second line wouldn't work (it's a no-op).

If you want to change the install_name of the library itself (as opposed to the recorded install names of libraries it uses), you have to use the -id option of install_name_tool, which takes just the new name (plus the filename of the library to change of course).

[kiwifb]

comment:22

Well spotted. I was a bit too eager there and I had completely forgotten about that important detail.

[vbraun]

comment:23

For the record, @rpath is OSX 10.5 and later only afaik. So absolute paths it is.

[kcrisman]

comment:24

Ok, I'm trying this. Thank you leif for the VERY explicit steps, which start to help me understand even what these files even are.

The otool output was fine, and I decided to have it rebuild eclib just to make sure that worked fine. I already reverted module_list.py and touched the files so I'll let you know what happens when it's done. I'm not messing with the pari issue, though :)

[kcrisman]

comment:45

Replying to @kiwifb:

Other libs without a proper install_name:

• libLfunction.dylib (lcalc)
• libcliquer.dylib
• polybori's libraries are strange, possibly bad substitution
• libzn_poly.dylib (zn_poly)
• libcsage.dylib :) moving to autotools would solve that

And apparently the R dylibs as well - see #16044, and of course fbissey and leif discovered these too :)

[kiwifb]

Branch: u/fbissey/flint2.4.3

[sagetrac-git]

Commit: d86fecf

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

d86fecf

trivial version bump for trac 16017

[kiwifb]

Author: François Bissey

[kcrisman]

Reviewer: Karl-Dieter Crisman

[kcrisman]

comment:49

From my point of view this is fine - upstream does seem to have a couple other changes since the previous release so it seems good to test this on a few other machines just in case there is some problem. But presumably those problems would also be problems elsewhere.

[kiwifb]

comment:50

Right. I haven't run doctests against it yet so I will. I am putting this back to "need work" because I forgot to generate the checksum :(

[kcrisman]

comment:51

So... will this fix be non-movable? (I.e., on any platform, on this one...) See leif's comment on #16044.

[sagetrac-git]

Changed commit from d86fecf to 6c25897

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

6c25897

checksums!

[kiwifb]

comment:53

Replying to @kcrisman:

So... will this fix be non-movable? (I.e., on any platform, on this one...) See leif's comment on #16044.

Basically yes, unless we find variables that overrides the install_names effectively. Note that once your install is moved, if we find variables overrides to make it work, you'll probably be screwed if you try to upgrade in the new location.

On linux LD_LIBRARY_PATH is a very effective override to anything you put in an elf, so regardless of Volker's comment about dangling rpath they are screened by LD_LIBRARY_PATH. Even on linux upgrading in the new location is probably leaving a mess behind, but LD_LIBRARY_PATH is a good safety blanket most of the time.

Putting this for review now that I have put the checksums.

[kcrisman]

comment:54

Hmm, I don't know if I like that. I'd at least want to see what happened here. How will relocation issues be a problem with people downloading binaries? (I assume this change in flint would cause problems for all Mac platforms, not just ours, since the extra flag and hence full name comes in on all Darwin, not just this Darwin 8, based on the patch.)

[nexttime]

comment:55

Replying to @kcrisman:

How will relocation issues be a problem with people downloading binaries?
(I assume this change in flint would cause problems for all Mac platforms, not just ours, since the extra flag and hence full name comes in on all Darwin, not just this Darwin 8, based on the patch.)

For FLINT and w.r.t. relocating Sage, the patch would only (effectively) make a difference on MacOS X 10.4.

If relocation is (at least partially) broken on Darwin anyway, the change doesn't make a difference to the others, i.e., it doesn't really worsen the situation.

On Darwin, we have absolute paths in (most of) all other libraries already; fixing that is subject of another ticket I'd say.

Disclaimer: I don't know at all what the Mac app currently does; but if it behaves differently, I'd rather expect it to be smarter w.r.t. "relocation".

[vbraun]

comment:56

According to http://wiki.sagemath.org/SupportedPlatforms the Mac App doesn't work at all on OSX 10.4

[nexttime]

comment:57

Replying to @nexttime:

If relocation is (at least partially) broken on Darwin anyway, the change doesn't make a difference to the others, i.e., it doesn't really worsen the situation.

On Darwin, we have absolute paths in (most of) all other libraries already; fixing that is subject of another ticket I'd say.

Ways to go (not excluding each other):

• Putting Sage's library folders into DYLD_FALLBACK_LIBRARY_PATH; rather a work-around (not to say hack).

• Rewriting all dylibs in sage-location (adjusting all absolute paths / install_names). (Raises the question whether install_name_tool is available "by default", i.e., at least shipped with something (e.g. XCode?) Sage requires anyway, for ordinary as opposed to development use. And the lengths of the new filenames might be an issue.)

• Using relative paths in install_names (presumably @loader_path/...). This could be done (just) in sage-location or some other post-inst hook as well, in the long run (also) changing build scripts (or just settings) and presumably patching a couple of upstream packages.

---

*Ads by G!*gle

Buy the compiler wrapper NOW!!!

[kcrisman]

comment:58

According to http://wiki.sagemath.org/SupportedPlatforms the Mac App doesn't work at all on OSX 10.4

Huh, that surprises me. There is one built, anyway, and distributed via the mirrors for 5.13. I can check that later - it should still work okay.

[vbraun]

Changed reviewer from Karl-Dieter Crisman to Karl-Dieter Crisman, Volker Braun

[vbraun]

Changed branch from u/fbissey/flint2.4.3 to 6c25897