wave stress problem

[tdcwilliams]

• change interp method for wave stress to InterpMeshToMesh2d instead of InterpMeshToMeshQuick
• compiles and runs with foss compiler (2018) and nextsim compiles with intel (but not liboasis)
• some changes to makefiles
• dmean and dmax are constant 1000m inside ww3 (look fairly normal in nextsim though) so still some things to look at

Runs with these env vars for nextsim

#! /bin/bash

# Environment for compiling neXtSIM with FOSS compilers on fram
# and running neXtSIM

# Load the right modules
module restore system
ml load Boost/1.68.0-foss-2018b-Python-3.6.6
ml load netCDF-C++4/4.3.0-foss-2018b
ml load CMake/3.12.1

# Data
# Laurent's project
export CLUSTER_ROOT_DIR=/cluster/projects/nn2993k/sim
# SIM group project
export CLUSTER_ROOT_DIR1=/cluster/projects/nn9878k/sim
# SKD project
export CLUSTER_ROOT_DIR2=/cluster/projects/nn9624k
# Annette's project
export CLUSTER_ROOT_DIR3=/cluster/projects/nn9481k

# neXtSIM data paths
export NEXTSIM_MESH_DIR=$CLUSTER_ROOT_DIR1/mesh
export NEXTSIM_DATA_DIR=$USERWORK/nextsim_data_dir

# compilers
export CC=mpicc
export CXX=mpicxx
export CFLAGS="-O3 -fPIC"
export CXXFLAGS="-O3 -pthread -fPIC -fopenmp"
export CCFLAGS="$CFLAGS"

# Libraries
export MPI_INC_DIR=$EBROOTOPENMPI/include
export GMSH_DIR=$CLUSTER_ROOT_DIR1/packages/gmsh-foss2020/
export PATH=$PATH:$GMSH_DIR/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$GMSH_DIR/lib
#export PETSC_DIR=$CLUSTER_ROOT_DIR/packages/petsc/
export BOOST_INCDIR=$EBROOTBOOST/include
export BOOST_LIBDIR=$EBROOTBOOST/lib
export NETCDF_DIR=$EBROOTNETCDFMINCPLUSPLUS4

# User or use-case specific directories
NEXTSIM_REPOS=$HOME/src/nextsim-repos
export NEXTSIMDIR=$NEXTSIM_REPOS/nextsim-foss
export NEXTSIM_ENV_ROOT_DIR=$NEXTSIM_REPOS/nextsim-env
export XSIM_COMP_VERBOSE=no

# short-cuts
alias cdn="cd $NEXTSIMDIR"
alias cde="cd $NEXTSIM_ENV_ROOT_DIR"

And these for oasis/ww3:

#! /bin/bash
source ~/nextsim.foss2018.src
ml load netCDF-Fortran/4.4.4-foss-2018b

# OASIS
export USE_OASIS=true
# # Annette's oasis
# export OASIS_DIR=$CLUSTER_ROOT_DIR3/oasis3-mct/oa3-mct_gfortran
# Guillaume's oasis
#export OASIS_DIR=$CLUSTER_ROOT_DIR1/packages/oasis_guibou
# Guillaume's OASIS: foss 2020 compilers
# export OASIS_DIR=$CLUSTER_ROOT_DIR1/packages/oasis_2020
# Guillaume's OASIS: foss 2018 compilers
export OASIS_DIR=$CLUSTER_ROOT_DIR1/packages/oa3-mct_gfortran_ww3_2018
export FC=mpif90
export NETCDF_FOR_DIR=$EBROOTNETCDFMINFORTRAN
export MPI_LIB_DIR=$EBROOTOPENMPI/lib
LDFLAGS+=" -lgfortran"
LDFLAGS+=" -Wl,-rpath,$MPI_LIB_DIR -L $MPI_LIB_DIR -lmpi_mpifh"
export LDFLAGS

# WW3
export WW3_DIR=$HOME/src/WW3-nersc
alias cdw="cd $WW3_DIR/model/bin; "

# compilation
export OASISDIR=$OASIS_DIR
export WWATCH3_NETCDF=NC4
export NETCDF_CONFIG=$EBROOTNETCDFMINFORTRAN/bin/nf-config

[einola]

Looks fine, but you need to merge with Laurent's changes when they're ready - as discussed. That's why I'm "requesting changes"

Also, there seems to be a bug in FromMeshToMeshQuick. Can you make an issue, describe the problem briefly, and assign it to me? I'll see if I can get around to it. Your changes are fine, but From MeshToMeshQuick should be faster :)

[tdcwilliams]

Hi @einola, I did the merge but can't test it while fram is out of action (back Friday or Monday).
I made the issue about FromMeshToMeshQuick

[einola]

This looks fine - just one change I don't get.

Check that it runs on fram, and then we can merge.

Unfortunately, then this doesn't actually compile on docker - but that's Laurent's fault, so we should fix it separately.

[einola]

I don't think core depends on what is in modules - does it?

[tdcwilliams]

core/src/Makefile requires liboasis:

ifdef USE_OASIS
         CXXFLAGS += -DOASIS
         CXXFLAGS += -I$(NEXTSIMDIR)/modules/oasis/include
         LDFLAGS += -L$(NEXTSIMDIR)/lib -loasis
         CHAN = MPI1
endif

There's some stuff in environment.cpp that relates to OASIS:

environment.cpp:    //! -3) Initialise communicator and OASIS (if compiled in)
environment.cpp:#ifdef OASIS
environment.cpp:    // For OASIS we need to get the local communicator first
environment.cpp:    // Initialise OASIS
environment.cpp:    int ierror = OASIS3::init_comp(&comp_id, comp_name);
environment.cpp:        OASIS3::abort(comp_id, comp_name, "Problem calling OASIS3::init_comp");
environment.cpp:    ierror = OASIS3::get_localcomm(&localComm);
environment.cpp:        OASIS3::abort(comp_id, comp_name, "Problem calling OASIS3::get_localcomm");
environment.cpp:    // create a Nextsim::Communicator from the OASIS communicator
environment.cpp:    ierror = OASIS3::create_couplcomm(mpicomm.rank()==0, &localComm, &cplComm);
environment.cpp:        OASIS3::abort(comp_id, comp_name, "Problem calling OASIS3::create_couplcomm");

[einola]

Yes - very good!

[einola]

Unfortunately, then this doesn't actually compile on docker - but that's Laurent's fault, so we should fix it separately.

I've merged the fix for this into develop, so you should merge/rebase develop into this branch before merging to develop.

[tdcwilliams]

merged and checked that it compiles for

• docker
• foss v 2018b
• intel v 2018b (will go to 2018a sometime)