libnest3.so not compiling on Mac OSX 10.6.8

[jtlz2]

Hi Johannes,

I'm trying to compile libnest3.so from multinest 2.18 so I can install the multinest bridge, but

make libnest3.so WITHOUT_MPI=1

fails thus:

localhost:MultiNest_v2.18 jtlz2$ make libnest3.so
ld -shared -o libnest3.so utils.o utils1.o priors.o kmeans_clstr.o xmeans_clstr.o posterior.o nested.o
ld: unknown option: -shared
make: *** [libnest3.so] Error 1

If I remove -shared I get:

[snip]
___nested_MOD_gen_initial_live in nested.o
___nested_MOD_nestsample in nested.o
___nested_MOD_nestsample in nested.o
___nested_MOD_nestsample in nested.o
___nested_MOD_nestsample in nested.o
___nested_MOD_nestrun in nested.o
___nested_MOD_nestrun in nested.o
___nested_MOD_nestrun in nested.o
___nested_MOD_nestrun in nested.o
___nested_MOD_nestrun in nested.o
"dsyevr", referenced from:
___utils1_MOD_diagonalize in utils1.o
___utils1_MOD_diagonalize in utils1.o
"_tan", referenced from:
___priors_MOD_cauchyprior in priors.o
ld: symbol(s) not found for inferred architecture i386

If I remove -m64 from my FC (see below) I still get:

localhost:MultiNest_v2.18 jtlz2$ make libnest3.so
gfortran -ffree-line-length-none -O3 -c -o utils.o utils.f90
gfortran -ffree-line-length-none -O3 -c -o utils1.o utils1.f90
gfortran -ffree-line-length-none -O3 -c -o priors.o priors.f90
gfortran -ffree-line-length-none -O3 -c -o kmeans_clstr.o kmeans_clstr.f90
gfortran -ffree-line-length-none -O3 -c -o xmeans_clstr.o xmeans_clstr.f90
gfortran -ffree-line-length-none -O3 -c -o posterior.o posterior.F90
gfortran -ffree-line-length-none -O3 -c -o nested.o nested.F90
ld -shared -o libnest3.so utils.o utils1.o priors.o kmeans_clstr.o xmeans_clstr.o posterior.o nested.o
ld: unknown option: -shared
make: *** [libnest3.so] Error 1

My makefile is modified from the supplied version:

< #FC = mpif90
< FC = gfortran -ffree-line-length-none -m64
< #CC = mpicc
< CC = gcc
< #CXX = mpiCC
< CXX = gcc
< FFLAGS += -O3 #-DMPI

< CFLAGS += -O3 #-DMPI

FC = mpif90
CC = mpicc
CXX = mpiCC
FFLAGS += -O3 -DMPI
CFLAGS += -O3 -DMPI

and multinest itself compiles and the examples (and indeed my own lhoods) have been running fine.

Any ideas? Thanks in advance for a speedy reply.

[JohannesBuchner]

Macs are distributed with LLVM/clang, not gcc. That's why the ld options don't work.

Either install and use gcc/gfortran, or run
$ clang -shared -o libnest3.so utils.o utils1.o priors.o kmeans_clstr.o xmeans_clstr.o posterior.o nested.o
(from http://stackoverflow.com/questions/9170000/how-use-llvm-linker)

[jtlz2]

Thanks so much for getting back to me so quickly. Still having problems.
The top of my makefile is:

#FC = mpif90
FC = gfortran -ffree-line-length-none -m64
#CC = mpicc
CC = gcc
#CXX = mpiCC
CXX = gcc
FFLAGS += -O3
#-DMPI
CFLAGS += -O3
#-DMPI

LAPACKLIB = -llapack

NESTLIBDIR = ./

export FC CC CXX FFLAGS CFLAGS LAPACKLIB

AR = ar r
LINKLIB = clang -shared

i.e. I'm using gfortran/gcc already. Here's what I get at the moment:

make
[snip - is OK]

make libest3.so
[snip]
___nested_MOD_clusterednest in nested.o
___nested_MOD_clusterednest in nested.o
___nested_MOD_clusterednest in nested.o
"dsyevr", referenced from:
___utils1_MOD_diagonalize in utils1.o
___utils1_MOD_diagonalize in utils1.o
ld: symbol(s) not found
clang: error: linker command failed with exit code 1 (use -v to see
invocation)
make: *** [libnest3.so] Error 1

Thanks again.

[JohannesBuchner]

Can this be resolved by passing -llapack?

It can be problematic to combine gfortran/gcc compiled binaries with a clang linker. Either compile with LLVM (which would be interesting), or use the gnu linker.
How did you install gcc? Try
$ which gcc
to find where gcc is installed, and check if there is a ld executable next to it. Presumably it is not the one called referred to in
$ which ld
Perhaps it is called gld.

[jtlz2]

Pass -llapack to which?

So:

which gcc
/usr/bin/gcc

which ld
/usr/bin/ld

Here's the gcc info:

gcc -v
Using built-in specs.
Target: i686-apple-darwin10
Configured with: /var/tmp/gcc/gcc-5664~89/src/configure --disable-checking
--enable-werror --prefix=/usr --mandir=/share/man
--enable-languages=c,objc,c++,obj-c++
--program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib
--build=i686-apple-darwin10 --program-prefix=i686-apple-darwin10-
--host=x86_64-apple-darwin10 --target=i686-apple-darwin10
--with-gxx-include-dir=/include/c++/4.2.1
Thread model: posix
gcc version 4.2.1 (Apple Inc. build 5664)

[JohannesBuchner]

To ld

[JohannesBuchner]

Huh, shared libraries are quite difficult in MacOS, judging from leto/math--gsl#4 and http://sourceforge.net/p/math-atlas/support-requests/536/

Sorry I can't test this, I don't have a Mac. Try these:
$ clang -dynamiclib -llapack -o libnest3.so utils.o utils1.o priors.o kmeans_clstr.o xmeans_clstr.o posterior.o nested.o

$ ld -llapack -dynamic -dylib -single_module -dead_strip -x -all_load -dylib_install_name -o libnest3.so utils.o utils1.o priors.o kmeans_clstr.o xmeans_clstr.o posterior.o nested.o
or a subset of these ld parameters

[jtlz2]

Making progress - I stuck with ld and libnest3.so was successfully generated with:

ld -o libnest3.so -llapack -lpthread -L. -lnest3 -dynamic -dylib -single_module -dead_strip -x -arch x86_64

(I've been doing 64 bit compilation. Haven't yet seen whether the line above can be trimmed down a bit.)

My modified makefile is at the bottom of this post.

And the next problem is (I think you know how to solve this though?):

make -C multinest_bridge libcnest.so WITHOUT_MPI=1

MULTINEST set to /Users/jtlz2/src/MultiNest_v2.18.

If you didn't set this to the directory where MultiNest

is compiled, you will receive see this error later:

OSError: libcnest.so: undefined symbol: __nested_MOD_nestrun

Checking if libnest3.so is in $MULTINEST

Found.

gfortran -m64 -shared -I. -O3 -std=c99 -fPIC -DMULTINEST_CALL=__nested_MOD_nestrun cnest.c -o libcnest.so -L/Users/jtlz2/src/MultiNest_v2.18 -lnest3 -llapack -lpthread
Undefined symbols:
"___nested_MOD_nestrun", referenced from:
_run in cc9RHcQV.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
make: *** [libcnest.so] Error 1

Thanks!

Here's the modified multinest makefile (most of the changes are, I suspect, redundant - again will prune at some point):

#FC = mpif90
FC = gfortran -ffree-line-length-none -m64 -lpthread
#CC = mpicc
CC = gcc -arch x86_64 -lpthread
#CXX = mpiCC
CXX = gcc -arch x86_64 -lpthread
FFLAGS += -O3 #-DMPI
CFLAGS += -O3 #-DMPI

LAPACKLIB = -llapack

NESTLIBDIR = ./

export FC CC CXX FFLAGS CFLAGS LAPACKLIB

AR = ar r
LINKLIB = ld -llapack -arch x86_64

NSOBJECTS = utils.o utils1.o priors.o kmeans_clstr.o xmeans_clstr.o posterior.o nested.o

%.o: %.f90
$(FC) $(FFLAGS) -c -o $@ $^

%.o: %.F90
$(FC) $(FFLAGS) -c -o $@ $^

default: libnest3.a

all: libnest3.a obj_detect eggboxC eggboxC++ gaussian gauss_shell
rosenbrock himmelblau ackley

libnest3.so: $(NSOBJECTS)
$(LINKLIB) -lpthread -o $(LIBS) $@ $^

libnest3.a: $(NSOBJECTS)
$(AR) $@ $^

[JohannesBuchner]

Great!
To get the right MULTINEST_CALL (every fortran compiler unfortunately names things differently, there is no convention), run
$ readelf -s libnest3.so | grep nestrun
For me, this prints out "__nested_MOD_nestrun". Then use this name
$ make -C multinest_bridge libcnest.so WITHOUT_MPI=1 MULTINEST_CALL=__nested_MOD_nestrun

[JohannesBuchner]

Hmm, readelf is a tool for linux binaries.
http://stackoverflow.com/questions/3286675/readelf-like-tool-for-mac-os-x suggests a couple of tools to use for MacOS, such as gobjdump (from macports):
$ gobjdump -plibnest3.so | grep nestrun

[jtlz2]

Hmm. I get nothing for:

gobjdump -plibnest3.so | grep nestrun

whereas -plibnest3.a does give some output (but not containing 'nestrun'). Also:

nm libnest3.a | grep nestrun
0000000000021020 T ___nested_MOD_nestrun

But for the .so, nm (not that I know what that does(!)) just gives:

nm libnest3.so
U dyld_stub_binder

So I wonder if something's wrong with the .so file, which I compiled with:

ld -o libnest3.so -llapack -lpthread -L. -lnest3 -dynamic -dylib -single_module -dead_strip -x -arch x86_64

I tried adding -L/Users/jtlz2/src/MultiNest_v2.18 but it made no difference.

Any ideas? :-S

[JohannesBuchner]

That's not good. Try without the -dylib -single_module -dead_strip -x options. I think what you have now is a stripped dylib file, not actually a .so. file will tell you.

If you get really frustrated with the MacOS compilation, you could set up a ubuntu virtual machine using Virtualbox. However figuring this native compilation out will be useful for others.

[jtlz2]

So file gives:

file libnest3.so
libnest3.so: Mach-O 64-bit dynamically linked shared library x86_64

I think you're right about the stripped dylib.. Here's the ld output now
though:

localhost:MultiNest_v2.18 jtlz2$ ld -llapack -dynamic -all_load
-dylib_install_name -L. -lnest3 -arch x86_64 -o libnest3.so -v
@(#)PROGRAM:ld PROJECT:ld64-97.17
Library search paths:
.
/usr/lib
/usr/local/lib
Framework search paths:
/Library/Frameworks/
/System/Library/Frameworks/
ld: could not find entry point "start" (perhaps missing crt1.o)

I do quite want to get this working on my mac....

[JohannesBuchner]

what does
$ ld -llapack -dynamic -dylib -all_load -dylib_install_name -L. -lnest3 -arch x86_64 -o libnest3.dylib -v
$ file libnest3.dylib
$ nm libnest3.dylib
do?

[jtlz2]

ld -llapack -dynamic -dylib -all_load -dylib_install_name -L. -lnest3 -arch
x86_64 -o libnest3.dylib -v
@(#)PROGRAM:ld PROJECT:ld64-97.17
Library search paths:
.
/usr/lib
/usr/local/lib
Framework search paths:
/Library/Frameworks/
/System/Library/Frameworks/

file libnest3.dylib
libnest3.dylib: Mach-O 64-bit dynamically linked shared library x86_64

nm libnest3.dylib
0000000000000000 t __mh_dylib_header

What do you get for your search paths (or is that a non-issue?)?

[jtlz2]

http://stackoverflow.com/questions/12601302/creating-shared-object-from-static-library-whose-object-files-were-linked-with

?

[JohannesBuchner]

For me it is:
$ ld --verbose|grep SEARCH_DIR
SEARCH_DIR("/usr/x86_64-pc-linux-gnu/lib64"); SEARCH_DIR("/usr/lib64/binutils/x86_64-pc-linux-gnu/2.2264"); SEARCH_DIR("/usr/local/lib64"); SEARCH_DIR("/lib64"); SEARCH_DIR("/usr/lib64"); SEARCH_DIR("/usr/x86_64-pc-linux-gnu/lib"); SEARCH_DIR("/usr/lib64/binutils/x86_64-pc-linux-gnu/2.22"); SEARCH_DIR("/usr/local/lib"); SEARCH_DIR("/lib"); SEARCH_DIR("/usr/lib");

and nm libnest3.so prints a lot of gfortran, _xmeans, _nested_MOD, etc. lines.
By the way, I don't think "-dylib_install_name" helps us at all, and "-L. -lnest3 " will not do anything, because you are not trying to load anything from the current directory, and you are not trying to load the nest3 library, you are actually trying to build it.
I only see now that you don't specify the .o files anymore.

Try this:
$ ld -llapack -arch x86_64 -dynamic -dylib -all_load -o libnest3.dylib utils.o utils1.o priors.o kmeans_clstr.o xmeans_clstr.o posterior.o nested.o

Yes, could try what you linked to, but then you have to know and specify each symbol.
How about http://stackoverflow.com/questions/3532589/how-to-build-a-dylib-from-several-o-in-mac-os-x-using-gcc

$ g++ -dynamiclib -undefined suppress -flat_namespace utils.o utils1.o priors.o kmeans_clstr.o xmeans_clstr.o posterior.o nested.o -o libnest3.dylib

[jtlz2]

Good news!

g++ -dynamiclib -undefined suppress -flat_namespace utils.o utils1.o
priors.o kmeans_clstr.o xmeans_clstr.o posterior.o nested.o -o
libnest3.dylib

works. nm is also happy now:

nm libnest3.so | grep nestrun
00000000000716a0 T ___nested_MOD_nestrun

once I had renamed libnest3.dylib -> libnest3.so

make -C multinest_bridge libcnest.so WITHOUT_MPI=1

also works without complaining :)

Now I have but a humble image issue:

getinetlaptop:PyMultiNest-master jtlz2$ python -c 'import pymultinest'
dlopen(libcnest.so, 6): Library not loaded: libnest3.dylib
Referenced from:
/Users/jtlz2/src/PyMultiNest-master/multinest_bridge/libcnest.so
Reason: image not found

Do you know what could be up there?

Thanks!

[jtlz2]

This also fails in the same way if I delete libnest.dylib (it then picks up
the .so).

[JohannesBuchner]

It is better to not use .so extensions, and stick to .dylib, because this is what you are using now and tools might get confused.
That means you also have to modify the Makefile in multinest_bridge (the multinest_bridge compile is quite stupid, it only says "I will use this library", but it does not check anything about it) and the 3rd line of pymultinest/run.py where it tries to load the library.

[jtlz2]

OK.

I'm still getting:

python -c 'import pymultinest'dlopen(libcnest.dylib, 6): Library not
loaded: libnest3.dylib
Referenced from:
/Users/jtlz2/src/PyMultiNest-master/multinest_bridge/libcnest.dylib
Reason: image not found

having done a search and replace .so -> .dylib in the Makefile in
multinest_bridge and pymultinest/run.py thus:

    lib = cdll.LoadLibrary('libcnest.so')

->

    lib = cdll.LoadLibrary('libcnest.dylib')

Getting there...

[JohannesBuchner]

You can be lazy and just put a
cdll.LoadLibrary('libnest3.dylib')
in front of it if automatic loading does not work.
I assume your DYLD_LIBRARY_PATH / LD_LIBRARY_PATH includes both the MultiNest and multinest_bridge directory?

[jtlz2]

Yes I was just setting

export
$DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/Users/jtlz2/src/PyMultiNest-master/multinest_bridge

which did the trick - massive thanks for all your help today, and for
writing the wrapper in the first place!

Do you want to update the documentation/makefiles or shall I? ;)

Thanks again!

[JohannesBuchner]

If you can update it it would be greatly appreciated, perhaps in form of a Makefile.MacOS that is minimally different from the Makefile.
So it sounds like the naming of the library was not the problem, they could be called .so? If so, that would save me changing the python file, and the multinest_bridge Makefile, and all that is needed is to correct the ld call in MultiNest?

[jtlz2]

OK I'll try and prepare a makefile since I have to explain how to compile
to a colleague anyway. I'll strip it down and check out the .dylib v .so
issue. Yes I think the only issue in the end was that ld line.

Thanks again!

[JohannesBuchner]

I put a if clause into https://github.com/JohannesBuchner/MultiNest/blob/master/Makefile
Is this sufficient to run on Mac OS?

[vallis]

Dear Johannes and Jonathan:

Wow, that was quite an effort, congratulations on getting it working. For completeness, I will describe how I got PyMultiNest to work on OS X 10.6-10.8.

First of all, I found homebrew (http://mxcl.github.com/homebrew/) very reliable to get tools that integrate with Apple's (i.e., XCode's gcc). With homebrew, getting a functional Fortran and MPI with fortran is a matter of doing

$ brew install gfortran
$ brew install open-mpi

Then, in the multinest Makefile (I am using 2.18 currently), I replace the lines

LAPACKLIB = -framework vecLib
LINKLIB = mpif90 -bundle -undefined dynamic_lookup

and I make libnest.a ONLY (not libnest.so). [So maybe the -bundle -undefined dynamic_lookup flags, which equate to -shared, are not even needed... oh well.]

Then I export MULTINEST={my multinest source directory), and proceed to PyMultiNest/multinest_bridge. That Makefile is modified thus (without bothering to show you where each line goes):

MULTINEST_CALL := __nested_MOD_nestrun
LINKLIB = ${FC} -bundle -undefined dynamic_lookup
test -e ${MULTINEST}/libnest3.a

The MULTINEST_CALL may depend on the exact version of the compiler, and the mangled name can be found with "nm libnest3.a | grep nestrun".

The resulting libcnest.so contains a full compiled version of the multinest library, but that's not a big price to pay IMHO. I was not able to make this work with both libnest3 and libcnest as shared libraries. To work correctly, libcnest.so must be moved to your Python's site-packages, or to the directory where you run code.

[JohannesBuchner]

Grand! If you can, please also provide the output of make so everyone is clear on how the compiler is actually called. Unless you modified the Makefile in other places, LAPACKLIB and LINKLIB will not used at all.
I'm still unsure whether I should recommend the dynamic or the static code variant. @vallis, would the dynamic variant fail for you (e.g. using https://github.com/JohannesBuchner/MultiNest/blob/master/Makefile ?)
You can also set the DYLD_LIBRARY_PATH variable instead of copying the libraries into specific directories.
Thanks for playing around with this!

[jtlz2]

Partly for my own benefit, I just wanted to record my steps for compiling with MultiNEST v3.2 on a Mac running 10.6.8 in 64-bit mode with MPI:

1. I couldn't get pymultinest to compile with the CMake version of v3.2. Are you able to liaise with the author to get this working? That would be awesome...

So, proceeding with the main version:

1. export MULTINEST='~/src/MultiNest_v3.2'
2. cd $MULTINEST
3. Update the makefile to work with 64 bits and MPI:

FC = mpif90 -ffree-line-length-none -lmpi -m64 -lpthread
CC = mpicc -arch x86_64 -lpthread
CXX = mpicc -arch x86_64 -lpthread
libnest3.so: $(NSOBJECTS)
g++ -dynamiclib -undefined suppress -flat_namespace utils.o utils1.o priors.o kmeans_clstr.o xmeans_clstr.o posterior.o nested.o -lpthread -o libnest3.so

1. make

Note that 'make all' fails with the current settings because the C and C++ versions of eggbox cannot be compiled, but the fortran versions do compile.

1. make libnest3.so

.so is actually a dynamic library, but I use the .so extension to avoid having to hack pymultinest (see discussion above).

1. Check contents of libnest3.so to see if the linking has worked:

nm libnest3.so | grep nestrun
0000000000072b40 T ___nested_MOD_nestrun

1. cd ~/src/PyMultiNest-master/
2. python setup.py install
3. Update the bridge makefile to work with 64 bits and MPI:

MNESTLIBNAME := nest3
CC = mpicc
FC = mpif90
FFLAGS += -I. -O3 -w -Wno-unused-parameter -fPIC -m64
CFLAGS += -I. -O3 -std=c99 -Wall -Wextra -Wno-unused-parameter -fPIC -arch x86_64
LINKLIB = ${FC} -bundle -undefined dynamic_lookup

1. make -C multinest_bridge libcnest.so

export MULTINEST=/src/MultiNest_v3.2
export DYLD_LIBRARY_PATH=$MULTINEST:/src/PyMultiNest-master/multinest_bridge:/src/MultiNest_v3.2_CMake/lib:$DYLD_LIBRARY_PATH

1. python

import pymultinest
help(pymultinest.run)

Success!

Is there any chance you could add a pymultinest.version or add this to the help command so one can keep track?

I wonder if you can add any of the above to the makefile? I'm not sure how to replace the g++ line in step (3) without asking Farhan to intervene. @vallis?

[JohannesBuchner]

Thanks for documenting. I use Linux exclusively, and I always use the CMake version, so I am not sure how I can help. What is your error compiling with cmake? Ultimately, you will have to communicate with Farhan if MultiNest does not compile for you on MacOS.

I am not sure why you need to specify -arch and -m64. By default, the compiler should build for your architecture automatically, so 64 bit binaries should come out.
So I am not sure you need to change the Makefile in the bridge at all, especially if you change -o libnest3.so to -o libmultinest.so, or create a symlink. I tried to incorporate the linklib changes in commit 70b43e3, otherwise I'm unsure how to ease installation.

What information would pymultinest.version hold? What would you use it for?

[jtlz2]

Hey.

In fact the cmake version is now working for me which is great:

export MULTINEST='~/src/MultiNest_v3.2_CMake/multinest/lib/'
cd $MULTINEST
ln -s libmultinest_mpi.dylib libmultinest_mpi.so

Then the bridge installation proceeds fine with your new linklib commit - thanks.

Ah don't worry about version - it just would've helped to check I was picking up the right version at my end. Since you use commits rather than specific versions numbers you could label using that.

Thanks!