Makefile for Jasmin

[SalahKouhen]

Hi there,

I am using the Jasmin hpc service in the UK. I don't really know how to write a system.mk file for it and would like some guidance if possible.

I used the Bluehive_open.mk. I ran into the following when I tried make Case_Files/coarse_grain.x, fatal error: netcdf_par.h: No such file or directory.

I tried getting a copy of this h file off github and put it in the main directory, the error became longer and included error: 'num_regions' not specified in enclosing 'parallel' and error: 'len_fields' not specified in enclosing 'parallel', so I can see that I am missing something here.

Salah

[bastorer]

Hi Salah,

I'm happy to help! netcdf_par.h will come from where / how your netcdf library is installed. Call nc-config --all should give a printout of your netcdf install settings. For example, on my system, I get something that lists

  --cxx4flags -> -I/usr/local/other/netcdf4/4.7.4/impi/19.1.3.304/intel-19.1.3.304_impi-19.1.3.304/include -DpgiFortran -D_FILE_OFFSET_BITS=64 -I/usr/local/other/netcdf4/4.7.4/impi/19.1.3.304/intel-19.1.3.304_impi-19.1.3.304/include -I/usr/local/other/hdf5/1.13.0/intel-19.1.3.304_impi-19.1.3.304/include -I/usr/local/intel/2020/compilers_and_libraries_2020.4.304/linux/mpi/intel64/include
  --cxx4libs  -> -L/usr/local/other/netcdf4/4.7.4/impi/19.1.3.304/intel-19.1.3.304_impi-19.1.3.304/lib -lnetcdf_c++4 -lnetcdf

and so in the system.mk file doing something like the following would tell the compiler where netcdf has been installed.

# Modules are automatically on lib dir
NETCDF_LIBS=`nc-config --cxx4libs`
NETCDF_INCS=`nc-config --cxx4flags`

LIB_DIRS:=${NETCDF_LIBS}
INC_DIRS:=${NETCDF_INCS}

I tried looking up the available modules for the Jasmin server, but had trouble finding a list.

For comparison, on the system that I typically use, I have a netcdf4 module that I load that then makes the library available.

Also, related to the second set of errors that you were getting, could you tell me what compiler you're using?

Regards,
Ben

[SalahKouhen]

Hi Ben,

I have

--cxx4flags     -> -I/opt/jaspy/include
--cxx4libs      -> -L/opt/jaspy//opt/jaspy/lib -lnetcdf-cxx4 -lnetcdf

so I copied your lines into my system.mk but still get fatal error: netcdf_par.h: No such file or directory.

Netcdf is part of the standard tools so I think they are available automatically but maybe I am missing some step like that.

For which compiler, do you want to see:

g++ --version
g++ (conda-forge gcc 12.1.0-16) 12.1.0 

or something else? Sorry, not that familiar with c++.

All the best,
Salah

[SalahKouhen]

On Jasmin the full output of nc-config --all is:

This netCDF 4.8.1 has been built with the following features: 

  --cc            -> x86_64-conda-linux-gnu-cc
  --cflags        -> -I/opt/jaspy/include
  --libs          -> -L/opt/jaspy/lib -lnetcdf
  --static        -> -lmfhdf -ldf -lhdf5_hl -lhdf5 -lm -lcurl -lzip

  --has-c++       -> no
  --cxx           -> 

  --has-c++4      -> yes
  --cxx4          -> /home/conda/feedstock_root/build_artifacts/netcdf-cxx4_1647186962680/_build_env/bin/x86_64-conda-linux-gnu-c++
  --cxx4flags     -> -I/opt/jaspy/include
  --cxx4libs      -> -L/opt/jaspy//opt/jaspy/lib -lnetcdf-cxx4 -lnetcdf

  --has-fortran   -> yes
  --fc            -> /home/conda/feedstock_root/build_artifacts/netcdf-fortran_1642696590650/_build_env/bin/x86_64-conda-linux-gnu-gfortran
  --fflags        -> -I/opt/jaspy/include -I/opt/jaspy/include
  --flibs         -> -L/opt/jaspy/lib -lnetcdff -lnetcdf -lnetcdf -lnetcdff_c
  --has-f90       -> TRUE
  --has-f03       -> yes

  --has-dap       -> yes
  --has-dap2      -> yes
  --has-dap4      -> yes
  --has-nc2       -> yes
  --has-nc4       -> yes
  --has-hdf5      -> yes
  --has-hdf4      -> yes
  --has-logging   -> no
  --has-pnetcdf   -> no
  --has-szlib     -> no
  --has-cdf5      -> yes
  --has-parallel4 -> no
  --has-parallel  -> no
  --has-nczarr    -> yes

  --prefix        -> /opt/jaspy
  --includedir    -> /opt/jaspy/include
  --libdir        -> /opt/jaspy/lib
  --version       -> netCDF 4.8.1

On my university cluster it is:

This netCDF 4.6.0 has been built with the following features:

  --cc        -> /usr/bin/cc
  --cflags    -> -I/usr/include -I/usr/include/hdf5/serial
  --libs      -> -L/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu/hdf5/serial -lnetcdf -lhdf5_hl -lhdf5 -lpthread -lsz -lz -ldl -lm -lcurl

  --has-c++   -> no
  --cxx       ->

  --has-c++4  -> yes
  --cxx4      -> g++
  --cxx4flags -> -I/usr/include -Wdate-time -D_FORTIFY_SOURCE=2
  --cxx4libs  -> -L/usr/lib/x86_64-linux-gnu -lnetcdf_c++4 -lnetcdf

  --has-fortran-> yes
  --fc        -> gfortran
  --fflags    -> -I/usr/include
  --flibs     -> -L/usr/lib -lnetcdff -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -lnetcdf -lnetcdf
  --has-f90   -> no
  --has-f03   -> yes

  --has-dap   -> yes
  --has-dap2  -> yes
  --has-dap4  -> yes
  --has-nc2   -> yes
  --has-nc4   -> yes
  --has-hdf5  -> yes
  --has-hdf4  -> no
  --has-logging-> no
  --has-pnetcdf-> no
  --has-szlib -> no
  --has-cdf5 -> no
  --has-parallel-> no

  --prefix    -> /usr
  --includedir-> /usr/include
  --libdir    -> /usr/lib/x86_64-linux-gnu
  --version   -> netCDF 4.6.0

I see the same error about netcdf_par on the uni cluster. I am trying to get the code to run on both but I am most interested in running it on my uni system.

All the best,

Salah

[bastorer]

Hi Salah,

Thanks for the notes! Sorry for the slow follow-up, Thanksgiving holidays were last week :-)

I will look into this more tomorrow and get back to you!

Regards,
Ben

[SalahKouhen]

Hi Ben,

Sorry, could I bump this please? Really looking forward to working with FlowSeive.

All the best,
Salah

[bastorer]

Hi Salah,

Thanks for the nudge! From the nc-config output, it looks like the netcdf library wasn't installed with parallel capabilities

  --has-parallel4 -> no
  --has-parallel  -> no

This would explain why netcdf_par.h unavailable (it's the header file for parallel netcdf), giving the fatal error: netcdf_par.h: No such file or directory.

The two options to solve this would be (if available) switching to a different netcdf module that was compiled with parallel capabilities, asking the cluster tech team to make a parallel netcdf module, or compiling netcdf yourself (this last one would be rather unpleasant, and avoided if possible haha).

Regards,
Ben

[yguo20ncsu]

Hi Ben,

I have a similar issue regarding netcdf_par.h when compiling coarse_grain.x on Cheyenne. Please see the erro info below. Is is also related to unparallel netcdf module on Cheyenne? (same nc-config output: --has-parallel4 -> no; --has-parallel -> no). Do you have any further insights/solutions on this? Thank you for your help!

Best,
Yiming

mpicxx -c -DDEBUG=0 -Wall -std=c++14 -O3 -fp-model fast=2 -ip -ipo -Wl,-Bstatic -lnetcdff -lnetcdf -lhdf5hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 -lsz -lz -Wl,-Bdynamic -lcurl -lm -ldl -o NETCDF_IO/write_integral_to_post.o NETCDF_IO/write_integral_to_post.cpp -lnetcdf -lhdf5_hl -lhdf5 -lz -lcurl -qopenmp -lm
In file included from NETCDF_IO/write_integral_to_post.cpp(5):
NETCDF_IO/../netcdf_io.hpp(12): catastrophic error: cannot open source file "netcdf_par.h"
#include "netcdf_par.h"
^

compilation aborted for NETCDF_IO/write_integral_to_post.cpp (code 4)
Makefile:78: recipe for target 'NETCDF_IO/write_integral_to_post.o' failed
make: *** [NETCDF_IO/write_integral_to_post.o] Error 4

[bastorer]

Hi Yiming,

Yes, it looks like the same error, the netcdf library that you're using doesn't have parallel features installed. I haven't looked in-depth, but it looks like Cheyenne does have parallel netcdf libraries available under the name netcdf-mpi: netcdf-mpi: netcdf-mpi/4.6.3, netcdf-mpi/4.7.1, netcdf-mpi/4.7.3,... (at least, I'm assuming from the names that those are netcdf libraries compiled with mpi compatibility).

I'm not sure what the dependencies are like, but those might be a good starting point.

The Cheyenne system file mentions using the following module commands:

$ module reset
$ module swap netcdf netcdf-mpi

Could you try using that and let me know if it works?

Regards,
Ben

[yguo20ncsu]

Hi Ben,

Thank you for your response! The compilation works well after swap to a parallel netcdf module on Cheyenne as you suggested. So far, so good.

Looking forward to working with this very cool algorithm! I appreciate your time and help.

Best,
Yiming

[bastorer]

We now have a working version on Jasmin!

Salah, if you could send me the system.mk file that you were using and the module load commands, I'll add a Jasmin system file to the set that's included in FlowSieve's Systems directory.

Beyond that, I'm going to go ahead and close this issue, but feel free to email me or open other issues with any other problems or questions :-)

[hmkhatri]

@bastorer @SalahKouhen
I want to use FlowSieve on Jasmin hpc system in the UK. Since you have already got it working, could you please share the system.mk file for Jasmin system? I couldn't find it on FlowSieve repo.
Thanks

[SalahKouhen]

Hi Hemant,

Unfortunately, I switched to using my university computing system instead midway through this thread, so the title is misleading. I never got FS to work on Jasmin!

Good luck!

[bastorer]

@hmkhatri If you want, we can arrange a zoom meeting for some time next week to see if we can try and get FlowSieve up and running on Jasmin :-) My schedule is pretty flexible, so feel free to email me to plan a time

[hmkhatri]

@SalahKouhen @bastorer Thanks. I too failed to compile on Jasmin system. Will check with their IT for a solution. Anyway, I ended u compiling on U. Liverpool Custer. It worked fine. Opened a pull request to add the system.mk file.