To enable or not to enable parallel [in HDF5]? That is the question.

[serbinsh]

./configure --prefix=/data/software/hdf5/1.8.13 --with-zlib=/data/software/zlib --enable-fortran

Should I also: --enable-parallel ?

I know that when I compile without using that switch configure uses mpi* compilers. I just can not remember if this is specifically needed, particularly with a mpirun/gridded run.

I am having some down-stream issues with not being able to make the latest netCDF:
/usr/bin/ld: /data/software/hdf5/1.8.13/lib/libhdf5_hl.a(H5DS.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/data/software/hdf5/1.8.13/lib/libhdf5_hl.a: could not read symbols: Bad value

And I think this may mean that I need to use the -fPIC compile flag for zlib, HDF4/5 and netCDF which requires a recompile.

[rgknox]

The implementation of parallel HDF5 has not been tested recently and should
definitely be viewed as at-your-own risk exploratory type stuff, if you
want to try make sure to turn:

USE_COLLECTIVE_MPIO=0

in the include file.

On Tue, Nov 4, 2014 at 4:34 AM, Shawn P. Serbin notifications@github.com
wrote:

./configure --prefix=/data/software/hdf5/1.8.13
--with-zlib=/data/software/zlib --enable-fortran

Should I also: --enable-parallel ?

I know that when I compile without using that switch configure uses mpi*
compilers. I just can not remember if this is specifically needed,
particularly with a mpirun/gridded run.

I am having some down-stream issues with not being able to make the latest
netCDF:
/usr/bin/ld: /data/software/hdf5/1.8.13/lib/libhdf5_hl.a(H5DS.o):
relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making
a shared object; recompile with -fPIC
/data/software/hdf5/1.8.13/lib/libhdf5_hl.a: could not read symbols: Bad
value

And I think this may mean that I need to use the -fPIC compile flag for
zlib, HDF4/5 and netCDF which requires a recompile.

—
Reply to this email directly or view it on GitHub
#10.

[rgknox]

Your question about symbol mangling in HDF5 is a tough one. I would not
had guessed you need to recompile zlib with FPIC also, but I could be
wrong. I would try compiling netcdf without the option that turns on the
need to link with hdf5, if you don't need it. The flag for that is
--disable-netcdf-4

--disable-netcdf-4

I also try to avoid dynamic libraries if possible, but it depends on your
intended usage. So for both netcdf and hdf5 you might be able to compile
with:

--disable-shared

I can compile both ED2 and CLM without shared libraries. This might help
the build complete successfully too.

On Tue, Nov 4, 2014 at 10:13 AM, Ryan Knox rgknox@lbl.gov wrote:

The implementation of parallel HDF5 has not been tested recently and
should definitely be viewed as at-your-own risk exploratory type stuff, if
you want to try make sure to turn:

USE_COLLECTIVE_MPIO=0

in the include file.

On Tue, Nov 4, 2014 at 4:34 AM, Shawn P. Serbin notifications@github.com
wrote:

./configure --prefix=/data/software/hdf5/1.8.13
--with-zlib=/data/software/zlib --enable-fortran

Should I also: --enable-parallel ?

I know that when I compile without using that switch configure uses mpi*
compilers. I just can not remember if this is specifically needed,
particularly with a mpirun/gridded run.

I am having some down-stream issues with not being able to make the
latest netCDF:
/usr/bin/ld: /data/software/hdf5/1.8.13/lib/libhdf5_hl.a(H5DS.o):
relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making
a shared object; recompile with -fPIC
/data/software/hdf5/1.8.13/lib/libhdf5_hl.a: could not read symbols: Bad
value

And I think this may mean that I need to use the -fPIC compile flag for
zlib, HDF4/5 and netCDF which requires a recompile.

—
Reply to this email directly or view it on GitHub
#10.

[rgknox]

doh! i had that backwards...

USE_COLLECTIVE_MPIO=0 if you DON'T want to do parallel IO.

[serbinsh]

Hi @rgknox

Thanks for the feedback. Here is how I ran the next round of ./configures:

Zlib
CFLAGS=-fPIC ./configure -s - -prefix=/data/software/zlib/1.2.8/

HDF4
CFLAGS=-fPIC ./configure - -with-zlib=/data/software/zlib/1.2.8 - -prefix=/data/software/hdf4/4.2.10

HDF5
CFLAGS=-fPIC./configure - -with-zlib=/data/software/zlib/1.2.8 - -enable-fortran - -prefix=/data/software/hdf5/1.8.13

netCDF
export CPPFLAGS="-I/data/software/hdf5/1.8.13/include -I/data/software/hdf4/4.2.10/include -I/data/software/zlib/1.2.8/include"
export LDFLAGS="-L/data/software/hdf5/1.8.13/include -L/data/software/hdf4/4.2.10/include -L/data/software/zlib/1.2.8/include"

./configure --disable-shared --disable-netcdf-4 --prefix=/data/software/netcdf/4.3.2

*Note I could not get make to work without invoking the --disable-shared --disable-netcdf-4 switches. Not sure why....but thanks for the suggestion!

[serbinsh]

Doh!
In R

install.packages("ncdf4")

...
ncdf4 REQUIRES the netcdf library be version 4 or above,
AND installed with HDF-5 support (i.e., the netcdf library must be
compiled with the --enable-netcdf-4 flag). If you don't want to install
the full version of netcdf-4 with HDF-5 support, then please install
the old, deprecated ncdf package instead.
...