linux installation without root?

[swadey]

I'm on a linux box where I don't have root or sudo privileges. Is there a way to install prereq libraries so that Pkg.add() doesn't fail for HDF5?

[timholy]

Interesting question. Without asking users to set LD_LIBRARY_PATH ahead of time, I'm not sure there's a good way to keep it from "failing" upon initial installation. But what we could do is include HDF5/deps in the search path for the library, so that it can find the lib if you manually install it there (you should have write permission there). That would prevent errors on Pkg.update().

Can I ask you to do some testing for me? Does it work to

• download and unpack one of the binaries
• start julia from inside the HDF5/test directory
• push!(DL_LOAD_PATH, path_to_library) where obviously you need to replace path_to_library with wherever you put libhdf5.so
• say include("runtests.jl")?

[swadey]

OK, I tried this. It fails to find libhdf5 when calling dlopen() from src/plain.jl

So I tried to do the dlopen() from the REPL and got the same issue:

julia>  push!(DL_LOAD_PATH, "/home/swade/projects/hdf5-1.8.13-linux-x86_64-shared/lib")
1-element Array{Union(ASCIIString,UTF8String),1}:
"/home/swade/projects/hdf5-1.8.13-linux-x86_64-shared/lib"

julia> dlopen("libhdf5")
ERROR: could not load module libhdf5: libhdf5: cannot open shared object file: No such file or directory
in dlopen at c.jl:19

julia> filesize(DL_LOAD_PATH[1] * "/libhdf5.so")
3107232

I’m guessing that the problem is that dlopen() can’t find libsz which causes the failure.

On May 24, 2014, at 6:59 AM, Tim Holy notifications@github.com wrote:

Interesting question. Without asking users to set LD_LIBRARY_PATH ahead of time, I'm not sure there's a good way to keep it from "failing" upon initial installation. But what we could do is include HDF5/deps in the search path for the library, so that it can find the lib if you manually install it there (you should have write permission there). That would prevent errors on Pkg.update().

Can I ask you to do some testing for me? Does it work to

• download and unpack one of the binaries
• start julia from inside the HDF5/test directory
• push!(DL_LOAD_PATH, path_to_library) where obviously you need to replace path_to_library with wherever you put libhdf5.so
• say include("runtests.jl")?
—
Reply to this email directly or view it on GitHub.

[timholy]

Is libsz installed? If so, you may need to add its path, too.

[swadey]

yes, it's in the same directory (it's part of the distribution for libhdf5)

[swadey]

Further testing: I think DL_LOAD_PATH may be broken? This is odd:

julia> dlopen("/home/swade/projects/hdf5-1.8.13-linux-x86_64-shared/lib/libhdf5.so")
ERROR: could not load module /home/swade/projects/hdf5-1.8.13-linux-x86_64-shared/lib/libhdf5.so: libsz.so.2: cannot open shared object file: No such file or directory
 in dlopen at c.jl:19

[timholy]

It's not surprising that, when given the complete path, it would complain about not being able to find a different library in the same directory. But I would have though that DL_LOAD_PATH would have worked. What if you cd to that directory?

[swadey]

Same error message:

julia> dlopen("libhdf5")
ERROR: could not load module libhdf5: libhdf5: cannot open shared object file: No such file or directory
in dlopen at c.jl:19

julia> pwd()
"/home/swade/projects/hdf5-1.8.13-linux-x86_64-shared/lib"

[timholy]

That's a different error message; this time it didn't find libhdf5, the previous time it complained about libsz.so.2. Try adding the .so to libhdf5 while you're in the directory that contains both.

[swadey]

I tried adding ".so":

julia> dlopen("libhdf5.so")
ERROR: could not load module libhdf5.so: libhdf5.so: cannot open shared object file: No such file or directory
in dlopen at c.jl:19

[timholy]

I thought that dlopen would search in the current directory, but I guess that's not true. I'm stumped about why DL_LOAD_PATH is not working.

I think your best hope is to use gdb or insert some printfs into https://github.com/JuliaLang/julia/blob/master/src/dlload.c#L114.

[swadey]

I'm baffled too. Here's what GDB says:

(gdb) 
126                         error = jl_uv_dlopen(path, handle, flags);
(gdb) p path
$6 = "/home/swade/projects/hdf5-1.8.13-linux-x86_64-shared/lib/libhdf5.so\000so\000\000\000\000\000\000\000\000\000\000P\005j\000\000\000\000\000TS`\367\377\177\000\000\320\366\314\367\377\177\000\000\001", '\000' <repeats 15 times>"\374, \377\377\377\377\377\377\377\000\000\000\000\000\000\000\000\021\000\000\000\000\000\000\000\320\366\314\367\377\177\000\000\001", '\000' <repeats 15 times>"\370, \377\377\377\377\377\377\377\000\000\000\000\000\000\000\000 \233\367\366\377\177\000\000еh\000\000\000\000\000TS`\367\377\177\000\000\320\366\314\367\377\177\000\000\001", '\000' <repeats 15 times>"\374, \377\377\377\377\377\377\377\000\000\000\000\000\000\000\000\004\000\000\000\000\000\000\000\320\366\314\367\377\177\000\000\001", '\000' <repeats 15 times>"\370"...
(gdb) n
127                         if (!error) goto done;
(gdb) p error
$7 = -1

It definitely forms the right path and it definitely tries to load it with the right file name extension.

So then I tried tracking into jl_uv_dlopen(). It gets a /full pathname/ and it fails, but interestingly the error message is (note the filename):

(gdb) p lib->errmsg
$3 = 0x258d100 "libsz.so.2: cannot open shared object file: No such file or directory"
(gdb) quit

This file is in the same directory:

lrwxrwxrwx 1 swade cdrom      14 May  6 14:17 libsz.so.2 -> libsz.so.2.0.0
-rwxr-xr-x 1 swade cdrom   50518 May  6 14:17 libsz.so.2.0.0

I think it's not using the path of the original file to load subsequent shared objects that are required by libhdf5?

[timholy]

I would say you're best off reporting this as a bug in Julia (you can point back to this issue, of course). I haven't really waded into libuv-related stuff. Sorry not to be able to help more. You've done a very nice job of isolating the problem!

One last thought, though: does it fix it if you launch julia as LD_LIBRARY_PATH=/home/swade/projects/hdf5-1.8.13-linux-x86_64-shared/lib julia?

[swadey]

That works fine.
On May 27, 2014, at 8:36 PM, Tim Holy notifications@github.com wrote:

LD_LIBRARY_PATH=/home/swade/projects/hdf5-1.8.13-linux-x86_64-shared/lib julia

[timholy]

OK, it looks like if you're going to do this, you basically have to set LD_LIBRARY_PATH. In such cases, I'm not certain there's much that can be profitably changed in this package. If you disagree, let me know.

I've at least documented the steps needed to make this work in the README---it would be great if you could look that over and edit it for any necessary improvements.

[timholy]

@swadey, any interest in trying @vtnash's suggestion about loading the libraries manually (e.g., loading libsz before hdf5)? If that works, then presumably we could set this up to work automatically, as long as the user places the libraries in a standard location like JULIA_PKGDIR/HDF5/deps/lib.

[swadey]

Sorry about the delay. Sure. I gave that a run and it works. I just added dlopen("libsz") prior to loading libhdf5

[timholy]

OK, we can fix this to be more convenient, then. Reopening.

[timholy]

Can you try checking out the teh/locallibinstall branch, putting the libraries (or a symlink) in JULIA_PKGDIR/HDF5/deps/usr/lib? In case you don't know how to do this, it's just

git fetch
git checkout teh/locallibinstall

When we're sure this works and you get to the point of wanting to follow official releases again, just say Pkg.free("HDF5") from inside julia.

[timholy]

In fact, let's be more ambitious: after you check out that branch, from within Julia please say Pkg.build("HDF5"). If I've done everything right, it should download and install the binaries in the proper directory for you.

[timholy]

Although I should say that I'm not certain this will work: presumably it will first try the installation via the package manager, and if that fails (which it will for you) I'm not certain it will try the Binaries install option. Anyway, let's see what happens.

[swadey]

So it fails due to some errors in build.jl. I fixed these but then doesn't actually to download the binaries, it still tries to apt-get install them. Then I tried commenting out apt-get and yum providers at which the build complains that none of the providers can install.

Here's what I have changed in build.jl:

hdf5 = library_dependency("libhdf5")                                                                                                                                              
    #provides(AptGet, "hdf5-tools", hdf5)                                                                                                                                             
    #provides(Yum, "hdf5", hdf5)                                                                                                                                                      
    if WORD_SIZE == 32                                                                                                                                                                
        provides(Binaries, URI("http://www.hdfgroup.org/ftp/HDF5/current/bin/linux/hdf5-1.8.13-linux-shared.tar.gz"), hdf5, unpacked_dir=Pkg.dir("HDF5/deps/usr/lib/"))               
    else                                                                                                                                                                              
        provides(Binaries, URI("http://www.hdfgroup.org/ftp/HDF5/current/bin/linux-x86_64/hdf5-1.8.13-linux-x86_64-shared.tar.gz"), hdf5, unpacked_dir=Pkg.dir("HDF5/deps/usr/lib/")) 
        println(hdf5.providers)                                                                                                                                                       
    end                           

[timholy]

And then it all works, right?

@loladiro & @staticfloat, any insight on whether BinDeps can be configured to use a fallback binary install if apt/yum fail? Here this issue is, how to handle library dependencies for a user who does not have root access?

[swadey]

sorry, yes it does work if I manually put the libraries into .julia/v0.3/HDF5/deps/usr/lib but it does not work when I comment out the provides calls for AptGet and Yum.

[staticfloat]

Hmmm.... I'm not sure why it wouldn't be working. That all looks right to me. :P

[timholy]

@swadey, it might be good to show us the errors that happen with Pkg.build("HDF5"), with the apt and yum providers commented out.

[swadey]

sure:

julia> Pkg.build("HDF5")
INFO: Building HDF5
(DependencyProvider,Dict{Symbol,Any})[(Binaries,[:unpacked_dir=>"/home/swade/.julia/v0.3/HDF5/deps/usr/lib/"])]
================================[ ERROR: HDF5 ]=================================

None of the selected providers can install dependency libhdf5
while loading /home/swade/.julia/v0.3/HDF5/deps/build.jl, in expression starting on line 36

================================================================================

================================[ BUILD ERRORS ]================================

WARNING: HDF5 had build errors.

 - packages with build errors remain installed in /home/swade/.julia/v0.3
 - build a package and all its dependencies with `Pkg.build(pkg)`
 - build a single package by running its `deps/build.jl` script

===============================================================================

[timholy]

'Doh! I closed some parens incorrectly. Can you git pull from that branch and try again? (Comment out the Apt/Yum again.)

[swadey]

I'm pretty sure I fixed those before trying it, but I can do this tomorrow
in the am. I initially had errors associated with uri.

thanks,
wade
On Jun 1, 2014 11:29 PM, "Tim Holy" notifications@github.com wrote:

'Doh! I closed some parens incorrectly. Can you git pull from that branch
and try again? (Comment out the Apt/Yum again.)

Reply to this email directly or view it on GitHub
#97 (comment).

[swadey]

@timholy yes, these fixes were the ones that I had made myself (so they are good)! that said, it still results in the same error.

[roncoursera]

I do have root permission but had to build the libraries from source and ran into similar issues. I resolved them with help from this thread and the new README and am just documenting the resolution.

Amazon EC2
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 3

Built and installed SZIP 2.1 from here
http://www.hdfgroup.org/ftp/lib-external/szip/2.1/src/szip-2.1.tar.gz
With configure option
configure --prefix=/usr

Built and installed HDF5 from here
http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.13.tar.gz
With configure option
configure --prefix=/usr

In the julia console,

Pkg.build("HDF5")
gave me the error
None of the selected providers can install dependency libhdf5

The resolution was

push!(Sys.DL_LOAD_PATH, "/usr/lib")
Pkg.build("HDF5")

[jstrube]

Not wanting to create another ticket, since this one is still open.
Had the same problem on 0.4.0-rc2. I have non-root access to a linux machine with hdf5 installed.
Interesting problem:
Pkg.build("HDF5") failed because libhdf5 could not be found, but BinUtils.debug("HDF5") found all dependencies.
I may misinterpret the BinUtils docs, but explicitly calling library_dependency("hdf5") found nothing, while library_dependency("libhdf5") was successful. From the BinUtils docs, I got the impression that the former should succeed.
Anyway, back to HDF5:
I solved it by:

• renaming hdf5=library_dependency... to libhdf5=library_dependency
• removing the windows_only and macosx_only macro blocks, and only keeping the linux_only block, without the macro.
• removing the compat line.

[jstrube]

Sorry, that only fixed the Pkg.build problem. Still working on "using HDF5", which is still complaining about libhdf5 not being defined

[jstrube]

Somehow it's working now, after a couple of build attempts. I noticed, however, that I don't have a clean checkout. Somehow Pkg.rm fails to cleanly uninstall the package.
Please ignore for now, sorry for the noise. Will report back if it's still a problem after a manual cleanup.

[chrisstolk]

Hello, I am using Julia 0.6.2 on a linux cluster. When running Pkg.build("HDF5") I get the following error message

E: Opening /etc/apt/sources.list.d/debian.list - ifstream::ifstream (13: Permission denied)
E: The list of sources could not be read.
==============================================[ ERROR: HDF5 ]===============================================

LoadError: failed process: Process(apt-cache showpkg hdf5-tools, ProcessExited(100)) [100]
while loading /home/cstolk/.julia/v0.6/HDF5/deps/build.jl, in expression starting on line 40

The build command for HDF5 fails.

Apparently some julia package (perhaps BinDeps) call "apt-cache showpkg hdf5-tools" which leads to these errors. This apt-cache command also generates an error when called from the command line.

On this system various choice of libhdf5_blabla.so files are available, which I can choose from using a module system. But following the suggestion in the README doesn't solve the problem.

[musm]

I'm guessing fixing #471 would fix this

[chrisstolk]

It seems to be a problem in BinDeps. BinDeps first calls "apt-cache -v" to test apt-cache. This test passes on my system. Then it calls "apt-cache showpkg hdf5-tools" which fails and causes the build command to fail. I commented out the line
provides(AptGet, "hdf5-tools", hdf5, os=:Linux)
in deps/build.jl . After doing this the package built without error messages and I could save and load some arrays.

[emcangi]

I have a similar problem as @swadey--I'm using RHEL and don't have sudoing privileges. I read the information in the readme, but my problem is I already have to preload the library folder just to get PyPlot working in Julia 0.6.2. The method I'm using to make PyPlot work is as described here, although my library folder was different. Specifically:

# following two lines go in .juliarc.jl
using PyCall
pygui(:tk)

# following line goes in .bashrc
alias julia="LD_PRELOAD=${HOME}/anaconda3/lib/libz.so julia"

How can I get HDF5 to work as well? I downloaded the binaries for HDF5, thinking I could put the library files into the anaconda3/lib folder, but I don't know if (e.g.) libz.so in the binary is the same as the one I already have in the anaconda library, and I don't want to overwrite anything that will then break PyPlot.. or something else.

EDIT:

The solution from a similar thread on ZMQ worked for me. Specifically, comment out the line provides(Yum, "hdf5", hdf5, os=:Linux)
in ~/.julia/v0.5/HDF5/deps/build.jl which will force it to download and build from source and not try to use yum.

[adowling2]

I am in a similar situation. I am using Julia on a centrally maintained cluster and I do not have root access. For some reason, the already installed hdf5 libraries are not being used.

Following the previous advise, I just commented out all of the provides lines in ~/.julia/v0.6/HDF5/deps/build.jl. The package now builds (although it compiles HDF5 from source.)

[musm]

should be fixed by the latest version when it goes through