diff --git a/README.md b/README.md index 5b81ce44..84ca1cf8 100644 --- a/README.md +++ b/README.md @@ -13,18 +13,20 @@ module use -a (path) module load GEOSenv ``` -where `(path)` depends on the computer and operating system: +where `(path)` depends on the computing system; at NCCS, `(path)` also depends on the operating system (SLES12 on Skylake and Cascade Lake nodes; SLES15 on Milan nodes, as of Jan. 2024): | System | Path | | ------------- |---------------------------------------------------| -| NCCS | `/discover/swdev/gmao_SIteam/modulefiles-SLES12` | +| NCCS Discover | `/discover/swdev/gmao_SIteam/modulefiles-SLES12` | +| | `/discover/swdev/gmao_SIteam/modulefiles-SLES15` | | NAS | `/nobackup/gmao_SIteam/modulefiles` | | GMAO desktops | `/ford1/share/gmao_SIteam/modulefiles` | +Step 1 can be coded into the user's shell configuration file (e.g., `.bashrc` or `.cshrc`). See the [GEOSgcm Wiki](https://github.com/GEOS-ESM/GEOSgcm/wiki/) for sample shell configuration files. ### Step 2: Obtain the Model -For development work, clone the _entire_ repository and use the `develop` branch as your starting point (equivalent to the `UNSTABLE` tag in the old CVS repository): +For development work, clone the _entire_ repository and use the `develop` branch as your starting point: ``` git clone -b develop git@github.com:GEOS-ESM/GEOSldas.git ``` @@ -36,25 +38,32 @@ git clone -b v17.9.1 --single-branch git@github.com:GEOS-ESM/GEOSldas.git ### Step 3: Build the Model -To build the model in a single step, do the following: +To build the model in a single step, do the following from a head node: ``` cd ./GEOSldas parallel_build.csh ``` -from a head node. Doing so will check out all the external repositories of the model (albeit only on the first run, [see subsection on mepo below](#mepo)!) and build the model. When done, the resulting model build will be found in `build-SLES12/` and the installation will be found in `install-SLES12/`, with setup scripts like `ldas_setup` in `install-SLES12/bin`. +This checks out all the external repositories of the model (albeit only on the first run, [see subsection on mepo below](#mepo)!) and then builds and installs the model. -To obtain a build that is suitable for debugging, use `parallel_build.csh -debug`, which will build in `build-Debug-SLES12/` and install in `install-Debug-SLES12/`. There is also an option for aggressive optimization. For details, see [GEOSldas Wiki](https://github.com/GEOS-ESM/GEOSldas/wiki). +At **NCCS**, the default is to build GEOSldas on SLES12 (Skylake or Cascade Lake nodes); to build GEOSldas on SLES15 (Milan nodes), use `parallel_build.csh -mil`. -See below for how to build the model in multiple steps. +The resulting model build is found in `build[-SLESxx]/`, and the installation is found in `install[-SLESxx]/`, with setup scripts like `ldas_setup` in `install[-SLESxx]/bin`. + +To obtain a build that is suitable for debugging, use `parallel_build.csh -debug`, which builds in `build-Debug[-SLESxx]/` and installs in `install-Debug[-SLESxx]/`. There is also an option for aggressive optimization. For details, see the [GEOSldas Wiki](https://github.com/GEOS-ESM/GEOSldas/wiki). + +Instructions for building the model in multiple steps are provided below. --- ## How to Set Up (Configure) and Run GEOSldas -a) Set up the job as follows: + +a) At **NCCS**, GEOSldas must be built, configured, and run on the same operating system. To run GEOSldas on Milan nodes (SLES15), start with `ssh discover-mil`. + +b) Set up the job as follows: ``` -cd (build_path)/GEOSldas/install/bin +cd (build_path)/GEOSldas/install[-SLESxx]/bin source g5_modules [for bash or zsh: source g5_modules.[z]sh] ./ldas_setup setup [-v] (exp_path) ("exe"_input_filename) ("bat"_input_filename) ``` @@ -82,7 +91,7 @@ Edit these sample files following the examples and comments within the sample fi The ldas_setup script creates a run directory and other directories at: `[exp_path]/[exp_name]` -Configuration input files will be created at: +Configuration input files are created at: `[exp_path]/[exp_name]/run` For more options and documentation, use any of the following: @@ -92,16 +101,19 @@ ldas_setup sample -h ldas_setup setup -h ``` -b) Configure the experiment output by editing the ```./run/HISTORY.rc``` file as needed. +c) Configure the experiment output by editing the ```./run/HISTORY.rc``` file as needed. -c) Run the job: +d) Run the job: ``` cd [exp_path]/[exp_name]/run/ sbatch lenkf.j ``` -For more information, see the files in `./doc/`. -Moreover, descriptions of the configuration (resource) parameters are included in the sample "exeinp" and "batinp" files that can be generated using `ldas_setup`. +At **NCCS**, the appropriate SLURM directive `#SBATCH --constraint=[xxx]` is automatically added into `lenkf.j` depending on the operating system. + +For more information, see the files in `./doc/`. Moreover, descriptions of the configuration (resource) parameters are included in the sample "exeinp" and "batinp" files that can be generated using `ldas_setup`. + + ----------------------------------------------------------------------------------- @@ -138,7 +150,6 @@ We currently do not allow in-source builds of GEOSldas. So we must make a direct ``` mkdir build ``` -The advantages of this is that you can build both a Debug and Release version with the same clone if desired. #### Run CMake CMake generates the Makefiles needed to build the model. @@ -146,7 +157,7 @@ CMake generates the Makefiles needed to build the model. cd build cmake .. -DBASEDIR=$BASEDIR/Linux -DCMAKE_Fortran_COMPILER=ifort -DCMAKE_INSTALL_PREFIX=../install ``` -This will install to a directory parallel to your `build` directory. If you prefer to install elsewhere change the path in: +This installs into a directory parallel to your `build` directory. If you prefer to install elsewhere change the path in: ``` -DCMAKE_INSTALL_PREFIX= ``` diff --git a/src/Applications/LDAS_App/CMakeLists.txt b/src/Applications/LDAS_App/CMakeLists.txt index f2f5ed36..b3b5a4a1 100644 --- a/src/Applications/LDAS_App/CMakeLists.txt +++ b/src/Applications/LDAS_App/CMakeLists.txt @@ -20,7 +20,6 @@ ecbuild_add_executable ( LIBS GEOSlandassim_GridComp) set (scripts - ldas_setup process_hist.csh process_rst.py ens_forcing/average_ensemble_forcing.py @@ -35,6 +34,10 @@ install ( DESTINATION bin ) +set(file ldas_setup) +configure_file(${file} ${file} @ONLY) +install(PROGRAMS ${CMAKE_CURRENT_BINARY_DIR}/${file} DESTINATION bin) + file(GLOB rc_files GEOSldas_*rc) file(GLOB nml_files LDASsa_DEFAULT*nml) diff --git a/src/Applications/LDAS_App/ldas_setup b/src/Applications/LDAS_App/ldas_setup index f8860681..12a72d27 100755 --- a/src/Applications/LDAS_App/ldas_setup +++ b/src/Applications/LDAS_App/ldas_setup @@ -51,6 +51,13 @@ class LDASsetup: 'MINLON','MAXLON','MINLAT','MAXLAT','EXCLUDE_FILE','INCLUDE_FILE','MWRTM_PATH','GRIDNAME', 'ADAS_EXPDIR', 'BCS_RESOLUTION' ] + # if built on sles15, BUILT_ON_SLES15 is "TRUE", else empty "" + BUILT_ON_SLES15 = "@BUILT_ON_SLES15@" + + if BUILT_ON_SLES15 == "TRUE": + self.BUILT_ON_SLES15 = True + else: + self.BUILT_ON_SLES15 = False # ------ # Required resource manager input fields @@ -693,7 +700,11 @@ class LDASsetup: print ('\nCorrect the tile file if it is an old EASE tile format... \n') EASEtile=self.bcsdir+'/MAPL_'+short_tile cmd = './preprocess_ldas.x correctease '+ tile + ' '+ EASEtile - print ("cmd: " + cmd) + if self.BUILT_ON_SLES15 : + print ("Executables were built on SLES15 and must be run on SLES15: " + cmd) + else: + print ("cmd: " + cmd) + sp.call(shlex.split(cmd)) if os.path.isfile(EASEtile) : @@ -1333,8 +1344,13 @@ class LDASsetup: elif 'MY_NODES' in line : line_ = line.replace('MY_NODES',str(self.optRmInp['nodes'])) fout.write(line_.replace('MY_NTASKS_PER_NODE',str(self.rqdRmInp['ntasks-per-node']))) - if int(self.rqdRmInp['ntasks-per-node']) > 40: - fout.write("#SBATCH --constraint=cas\n") + + if self.BUILT_ON_SLES15 : + fout.write("#SBATCH --constraint=mil\n") + else: + assert int(self.rqdRmInp['ntasks-per-node']) <= 46, 'ntasks-per-node should be <=46 for cas' + fout.write("#SBATCH --constraint=cas\n") + elif 'MY_OSERVER_NODES' in line : fout.write(line.replace('MY_OSERVER_NODES',str(self.optRmInp['oserver_nodes']))) elif 'MY_WRITERS_NPES' in line : @@ -1365,8 +1381,6 @@ class LDASsetup: elif 'MY_ADAS_EXPDIR' in line : if self.ladas_coupling > 0: fout.write(line.replace('MY_ADAS_EXPDIR', self.rqdExeInp['ADAS_EXPDIR'])) - - else : fout.write(line.replace('MY_EXPDIR',self.exphome+'/$EXPID')) diff --git a/src/Applications/LDAS_App/lenkf.j.template b/src/Applications/LDAS_App/lenkf.j.template index 11a1ddfb..6f77c5c2 100644 --- a/src/Applications/LDAS_App/lenkf.j.template +++ b/src/Applications/LDAS_App/lenkf.j.template @@ -40,7 +40,23 @@ setenv argv source $GEOSBIN/g5_modules -setenv I_MPI_DAPL_UD enable +# OPENMPI flags +# Turn off warning about TMPDIR on NFS +setenv OMPI_MCA_shmem_mmap_enable_nfs_warning 0 +# pre-connect MPI procs on mpi_init +setenv OMPI_MCA_mpi_preconnect_all 1 +setenv OMPI_MCA_coll_tuned_bcast_algorithm 7 +setenv OMPI_MCA_coll_tuned_scatter_algorithm 2 +setenv OMPI_MCA_coll_tuned_reduce_scatter_algorithm 3 +setenv OMPI_MCA_coll_tuned_allreduce_algorithm 3 +setenv OMPI_MCA_coll_tuned_allgather_algorithm 4 +setenv OMPI_MCA_coll_tuned_allgatherv_algorithm 3 +setenv OMPI_MCA_coll_tuned_gather_algorithm 1 +setenv OMPI_MCA_coll_tuned_barrier_algorithm 0 +# required for a tuned flag to be effective +setenv OMPI_MCA_coll_tuned_use_dynamic_rules 1 +# disable file locks +setenv OMPI_MCA_sharedfp "^lockedfile,individual" # By default, ensure 0-diff across processor architecture by limiting MKL's freedom to pick algorithms. # As of June 2021, MKL_CBWR=AVX2 is fastest setting that works for both haswell and skylake at NCCS. @@ -53,11 +69,8 @@ setenv MKL_CBWR "AVX2" # reversed sequence for LADAS_COUPLING (Sep 2020) (needed when coupling with ADAS using different BASEDIR) setenv LD_LIBRARY_PATH ${BASEDIR}/${ARCH}/lib:${ESMADIR}/lib:${LD_LIBRARY_PATH} -if ( -e /etc/os-release ) then - module load nco/4.8.1 -else - module load other/nco-4.6.8-gcc-5.3-sp3 -endif +module load nco + setenv RUN_CMD "$GEOSBIN/esma_mpirun -np " #######################################################################