-
The C++ MPI library is called different things depending on platform and which implementation of MPI is being used - this needs to be specified by passing in the
MPICXX_LIBoption to the Makefile. Possible names include:MPICXX_LIB=-lmpi_cxxMPICXX_LIB=-lmpiCCMPICXX_LIB=-lmpichcxx
-
Device selection is done by choosing the devices starting at the index given by
opencl_devicein clover.in. For example, if there are 2 devices on a system andopencl_deviceis 0, rank 0 will take device 0 and rank 1 will take device 1.
Turn on opencl kernel use by putting use_opencl_kernels in tea.in.
opencl_vendorchooses the vendor - typicallynvidia,advanced(AMD), orintel, butanywill choose the first available platformopencl_typechooses device type - typicallygpu,cpu, oracceleratorbutallwill just choose the first device from the specified platform. Setting this tolistwill list all platforms and devices.opencl_devicespecifies the index of the device to use (0 indexed)opencl_usefirstmakes device selection ignore MPI ranks and make every process try to use device 0 in whatever platform it has.
- In many case just typing
makein the required software version will work.
If the MPI compilers have different names then the build process needs to
notified of this by defining two environment variables, MPI_COMPILER and
C_MPI_COMPILER.
For example on some Intel systems:
make MPI_COMPILER=mpiifort C_MPI_COMPILER=mpiicc
Or on Cray systems:
make MPI_COMPILER=ftn C_MPI_COMPILER=cc
All compilers use different arguments to invoke OpenMP compilation. A simple
call to make will invoke the compiler with -O3. This does not usually include
OpenMP by default. To build for OpenMP for a specific compiler a further
variable must be defined, COMPILER that will then select the correct option
for OpenMP compilation.
For example with the Intel compiler:
make COMPILER=INTEL
Which then append the -openmp to the build flags.
Other supported compiler that will be recognise are:-
- CRAY
- SUN
- GNU
- IBM
- PATHSCALE
- PGI
The default flags for each of these is show below:-
- INTEL: -O3 -ipo
- SUN: -fast
- GNU: -ipo
- XL: -O5
- PATHSCLE: -O3
- PGI: -O3 -Minline
- CRAY: -em Note: that by default the Cray compiler with pick the optimum options for performance.
The default compilation with the COMPILER flag set chooses the optimal performing set of flags for the specified compiler, but with no hardware specific options or IEEE compatability.
To produce a version that has IEEE compatiblity a further flag has to be set on the compiler line.
make COMPILER=INTEL IEEE=1
This flag has no effect if the compiler flag is not set because IEEE options are always compiler specific.
For each compiler the flags associated with IEEE are shown below:-
- INTEL: -fp-model strict –fp-model source –prec-div –prec-sqrt
- CRAY: -hpflex_mp=intolerant
- SUN: -fsimple=0 –fns=no
- GNU: -ffloat-store
- PGI: -Kieee
- PATHSCALE: -mieee-fp
- XL: -qstrict –qfloat=nomaf
Note that the MPI communications have been written to ensure bitwise identical answers independent of core count. However under some compilers this is not true unless the IEEE flags is set to be true. This is certainly true of the Intel and Cray compiler. Even with the IEEE options set, this is not guarantee that different compilers or platforms will produce the same answers. Indeed a Fortran run can give different answers from a C run with the same compiler, same options and same hardware.
Extra options can be added without modifying the makefile by adding two further
flags, OPTIONS and C_OPTIONS, one for the Fortran and one for the C options.
make COMPILER=INTEL OPTIONS=-xavx C_OPTIONS=-xavx
Finally, a DEBUG flag can be set to use debug options for a specific compiler.
make COMPILER=PGI DEBUG=1
These flags are also compiler specific, and so will depend on the COMPILER
environment variable.
So on a system without the standard MPI wrappers, for a build that requires OpenMP, IEEE and AVX this would look like so:-
make COMPILER=INTEL MPI_COMPILER=mpiifort C_MPI_COMPILER=mpiicc IEEE=1 \
OPTIONS="-xavx" C_OPTIONS="-xavx"