Merge branch 'lilac_cap'

Add LILAC Add LILAC: The Lightweight Infrastructure for Land-Atmosphere Coupling. This infrastructure consists of two major pieces: (1) A lightweight coupling infrastructure built on top of ESMF that makes it easier for atmosphere models to call CTSM directly, rather than using the hub-and-spoke architecture that is used by CESM. (2) A set of python-based tools for building CTSM and creating its runtime inputs when running in an atmosphere model via LILAC. Although these tools are built on top of cime, details of the create_newcase / case.setup / case.build process are hidden from the user, because many of the aspects of this workflow don't make sense in the LILAC context. So far we have used LILAC to couple CTSM to WRF. There are plans to use the same infrastructure to couple CTSM to other regional atmosphere models. Documentation of LILAC is provided in https://escomp.github.io/ctsm-docs/versions/master/html/lilac/index.html (though there are still some missing sections), as well as in various presentations on the wiki (https://github.com/ESCOMP/CTSM/wiki/Presentations). There have been many contributors besides myself to the development, testing and documentation of LILAC; chief among them being Mariana Vertenstein, Negin Sobhani, Joe Hamman, Sam Levis, Mike Barlage and Dave Lawrence.
ESCOMP · Jul 6, 2020 · d143fc8 · d143fc8
2 parents 488860d + cb94fb4
commit d143fc8
Show file tree

Hide file tree

Showing 123 changed files with 15,355 additions and 491 deletions.
diff --git a/README.lilac b/README.lilac
@@ -0,0 +1,110 @@
+========================================================================
+I. Building a CTSM / LILAC library for inclusion in an atmosphere model
+========================================================================
+
+1) check out the code (ctsm and lilac are now bundled together) and built as one library
+
+   > git clone https://github.com/ESCOMP/ctsm.git
+   > git checkout lilac_cap
+   > ./manage_externals/checkout_externals -v
+
+2) set the following environment variables
+   SRCROOT is where ctsm is checked out 
+
+   > export SRCROOT=`pwd`
+   > export CASEDIR=/glade/scratch/$USER/test_lilac
+
+3) build the ctsm/lilac library using a CIME case
+
+   > cd $SRCROOT/cime/scripts
+   > ./create_newcase --case $CASEDIR --compset I2000Clm50SpRsGs --res f45_f45_mg37 --run-unsupported --driver nuopc
+   >  cd $CASEDIR
+   > ./xmlchange LILAC_MODE=on
+   > ./xmlchange DEBUG=TRUE
+   > ./case.setup
+   > ./case.build --sharedlib-only
+
+========================================================================
+II. Building and running the test atmosphere driver
+========================================================================
+
+After following the above instructions for building a CTSM / LILAC
+library (I), do the following:
+
+1) To build the atm_driver executable on cheyenne (***CTSM_MKFILE IS CRITICAL for the operation of the atm_driver makefile)
+
+   > export CTSM_MKFILE=$CASEDIR/bld/ctsm.mk
+   > cd $SRCROOT/lilac/atm_driver
+   > $SRCROOT/cime/tools/configure --comp-interface nuopc --macros-format Makefile --clean
+   > make clean
+   > source ./.env_mach_specific.sh
+   > export DEBUG=TRUE
+   > make COMPILER=intel atm_driver
+
+2) to generate the input namelists
+
+  - to customize the generated namelist - edit the file ctsm.cfg (in this directory)
+  - to create the ctsm namelist FROM THIS DIRECTORY:
+
+    > $SRCROOT/lilac_config/buildnml
+
+  - this will now create the files lnd_in and clm.input_data_list in this directory
+    THIS ONLY NEEDS TO BE DONE ONCE 
+    to futher customize the lnd_in (say to adjust the ctsm history output) edit the generated lnd_in in this directory
+
+3) run the atm_driver on cheyenne
+
+  > qsub cheyenne.sub
+
+4) compare with latest baselines
+
+   use something like this to compare the last clm and last cpl hist files:
+
+   > basedir=/glade/p/cgd/tss/ctsm_baselines/lilac_20191202
+   > cprnc test_lilac.clm2.h0.2000-01-03-00000.nc $basedir/test_lilac.clm2.h0.2000-01-03-00000.nc | tail -30
+   > cprnc test_lilac.lilac.hi.2000-01-02-81000.nc $basedir/test_lilac.lilac.hi.2000-01-02-81000.nc | tail -30
+   > cprnc -m test_lilac.atm.h0.0001-01.nc $basedir/test_lilac.atm.h0.0001-01.nc | tail -30
+
+5) if there are differences, and those are intentional, then create new
+   baselines
+
+   copy all *.nc files, plus ctsm.cfg, lilac_in and lnd_in to the
+   baseline directory
+
+========================================================================
+III. Linking the CTSM / LILAC library into another atmosphere model
+========================================================================
+
+After following the above instructions for building a CTSM / LILAC
+library (I), you should do the following, assuming that the atmosphere
+model is built using a makefile:
+
+1) Set some environment variable (e.g., CTSM_MKFILE) to point to the
+   ctsm.mk file generated in CTSM's bld directory.
+
+2) Modify the atmosphere model's makefile to include the file given by
+   the environment variable $CTSM_MAKEFILE.
+
+3) In the compilation line for the atmosphere model, add
+   $(CTSM_INCLUDES)
+
+4) In the link line for the atmosphere model, add $(CTSM_LIBS)
+
+========================================================================
+IV. Running CTSM / LILAC from another atmosphere model
+========================================================================
+
+After (III), the following steps are needed to stage the inputs needed
+for running the atmosphere model
+
+1) Generate the input namelists following the instructions given in part
+   (II).
+
+2) Copy the following files from $SRCROOT/lilac/atm_driver into the
+   directory from which the atmosphere model will be run:
+
+   - lilac_in
+   - lnd_in
+   - lnd_modelio.nml
+
+3) Run the atmosphere model
diff --git a/bld/CLMBuildNamelist.pm b/bld/CLMBuildNamelist.pm
@@ -35,7 +35,7 @@ use File::Basename qw(dirname);
 use English;
 use Getopt::Long;
 use IO::File;
-use File::Glob ':glob';
+use File::Glob ':bsd_glob';
 
 #-------------------------------------------------------------------------------
 #
@@ -189,7 +189,6 @@ OPTIONS
                                form \$CASEDIR/user_nl_clm/user_nl_clm_????)
      -inputdata "filepath"    Writes out a list containing pathnames for required input datasets in
                               file specified.
-     -l_ncpl "LND_NCPL"       Number of CLM coupling time-steps in a day.
      -lnd_tuning_mode "value" Use the parameters tuned for the given configuration (CLM version and atmospheric forcing)
      -mask "landmask"         Type of land-mask (default, navy, gx3v5, gx1v5 etc.)
                               "-mask list" to list valid land masks.
@@ -255,7 +254,6 @@ sub process_commandline {
                help                  => 0,
                glc_nec               => "default",
                light_res             => "default",
-               l_ncpl                => undef,
                lnd_tuning_mode       => "default",
                lnd_frac              => undef,
                dir                   => "$cwd",
@@ -307,7 +305,6 @@ sub process_commandline {
              "infile=s"                  => \$opts{'infile'},
              "lnd_frac=s"                => \$opts{'lnd_frac'},
              "lnd_tuning_mode=s"         => \$opts{'lnd_tuning_mode'},
-             "l_ncpl=i"                  => \$opts{'l_ncpl'},
              "inputdata=s"               => \$opts{'inputdata'},
              "mask=s"                    => \$opts{'mask'},
              "namelist=s"                => \$opts{'namelist'},
@@ -477,7 +474,7 @@ sub read_envxml_case_files {
   my %envxml = ();
   if ( defined($opts->{'envxml_dir'}) ) {
       (-d $opts->{'envxml_dir'})  or  $log->fatal_error( "envxml_dir is not a directory" );
-      my @files = glob( $opts->{'envxml_dir'}."/env_*xml" );
+      my @files = bsd_glob( $opts->{'envxml_dir'}."/env_*xml" );
       ($#files >= 0)              or  $log->fatal_error( "there are no env_*xml files in the envxml_dir" );
       foreach my $file (@files) {
           $log->verbose_message( "Open env.xml file: $file" );
@@ -1394,9 +1391,6 @@ sub process_namelist_commandline_clm_usr_name {
         $nvars++;
       }
     }
-    if ( $nvars == 0 ) {
-      $log->message("setting clm_usr_name -- but did NOT find any user datasets: $opts->{'clm_usr_name'}", $opts);
-    }
     # Go through all variables and expand any XML env settings in them
     expand_xml_variables_in_namelist( $nl_usrfile, $envxml_ref );
     # Merge input values into namelist.  Previously specified values have higher precedence
@@ -1438,7 +1432,7 @@ sub process_namelist_commandline_use_case {
       my $val = $uc_defaults->get_value($var, \%settings );
 
       if ( defined($val) ) {
-        $log->message("CLM adding use_case $opts->{'use_case'} defaults for var '$var' with val '$val'");
+        $log->verbose_message("CLM adding use_case $opts->{'use_case'} defaults for var '$var' with val '$val'");
 
         add_default($opts,  $nl_flags->{'inputdata_rootdir'}, $definition, $defaults, $nl_usecase, $var, 'val'=>$val);
       }
@@ -1486,7 +1480,6 @@ sub process_namelist_inline_logic {
   setup_logic_co2_type($opts, $nl_flags, $definition, $defaults, $nl);
   setup_logic_irrigate($opts, $nl_flags, $definition, $defaults, $nl);
   setup_logic_start_type($opts, $nl_flags, $nl);
-  setup_logic_delta_time($opts, $nl_flags, $definition, $defaults, $nl);
   setup_logic_decomp_performance($opts,  $nl_flags, $definition, $defaults, $nl);
   setup_logic_snow($opts, $nl_flags, $definition, $defaults, $nl);
   setup_logic_glacier($opts, $nl_flags, $definition, $defaults, $nl,  $envxml_ref);
@@ -1862,28 +1855,6 @@ sub setup_logic_start_type {
 
 #-------------------------------------------------------------------------------
 
-sub setup_logic_delta_time {
-  my ($opts, $nl_flags, $definition, $defaults, $nl) = @_;
-
-  if ( defined($opts->{'l_ncpl'}) ) {
-    my $l_ncpl = $opts->{'l_ncpl'};
-    if ( $l_ncpl <= 0 ) {
-      $log->fatal_error("bad value for -l_ncpl option.");
-    }
-    my $val = ( 3600 * 24 ) / $l_ncpl;
-    my $dtime = $nl->get_value('dtime');
-    if ( ! defined($dtime)  ) {
-      add_default($opts,  $nl_flags->{'inputdata_rootdir'}, $definition, $defaults, $nl, 'dtime', 'val'=>$val);
-    } elsif ( $dtime ne $val ) {
-      $log->fatal_error("can NOT set both -l_ncpl option (via LND_NCPL env variable) AND dtime namelist variable.");
-    }
-  } else {
-    add_default($opts,  $nl_flags->{'inputdata_rootdir'}, $definition, $defaults, $nl, 'dtime', 'hgrid'=>$nl_flags->{'res'});
-  }
-}
-
-#-------------------------------------------------------------------------------
-
 sub setup_logic_decomp_performance {
   my ($opts, $nl_flags, $definition, $defaults, $nl) = @_;
 
@@ -4203,7 +4174,7 @@ sub validate_options {
            # create the @expect array by listing the files in $use_case_dir
            # and strip off the ".xml" part of the filename
            @expect = ();
-           my @files = glob("$opts->{'use_case_dir'}/*.xml");
+           my @files = bsd_glob("$opts->{'use_case_dir'}/*.xml");
            foreach my $file (@files) {
                $file =~ m{.*/(.*)\.xml};
                &check_use_case_name( $1 );
@@ -4217,7 +4188,7 @@ sub validate_options {
         } else {
            print "Use cases are:...\n\n";
            my @ucases;
-           foreach my $file( sort( glob($opts->{'use_case_dir'}."/*.xml") ) ) {
+           foreach my $file( sort( bsd_glob($opts->{'use_case_dir'}."/*.xml") ) ) {
               my $use_case;
               if ( $file =~ /\/([^\/]+)\.xml$/ ) {
                  &check_use_case_name( $1 );

diff --git a/bld/namelist_files/namelist_defaults_ctsm.xml b/bld/namelist_files/namelist_defaults_ctsm.xml
@@ -19,9 +19,6 @@ attributes from the config_cache.xml file (with keys converted to upper-case).
 
 <sim_year>2000</sim_year>
 
-<!-- Default time step -->
-<dtime>1800</dtime>
-
 <!-- CO2 volume mixing ratio -->
 <co2_ppmv sim_year="1000"      >379.0</co2_ppmv>
 <co2_ppmv sim_year="2000"      >379.0</co2_ppmv>

diff --git a/bld/namelist_files/namelist_definition_ctsm.xml b/bld/namelist_files/namelist_definition_ctsm.xml
@@ -619,11 +619,6 @@ The maximum value to use for zeta under stable conditions
 baseline proportion of nitrogen allocated for electron transport (J)
 </entry>
 
-<entry id="dtime" type="real"  category="clm_physics" 
-       group="clm_inparm" valid_values="">
-Time step (seconds)
-</entry>
-
 <entry id="use_fates" type="logical" category="physics"
         group="clm_inparm" valid_values="" value=".false.">
 Toggle to turn on the FATES model

diff --git a/bld/unit_testers/build-namelist_test.pl b/bld/unit_testers/build-namelist_test.pl
@@ -138,9 +138,9 @@ sub make_config_cache {
 #
 # Figure out number of tests that will run
 #
-my $ntests = 842;
+my $ntests = 834;
 if ( defined($opts{'compare'}) ) {
-   $ntests += 510;
+   $ntests += 507;
 }
 plan( tests=>$ntests );
 
@@ -287,15 +287,15 @@ sub make_config_cache {
 &make_config_cache($phys);
 
 print "\n===============================================================================\n";
-print "Test configuration, structure, irrigate, verbose, clm_demand, ssp_rcp, test, sim_year, use_case, l_ncpl\n";
+print "Test configuration, structure, irrigate, verbose, clm_demand, ssp_rcp, test, sim_year, use_case\n";
 print "=================================================================================\n";
 
-# configuration, structure, irrigate, verbose, clm_demand, ssp_rcp, test, sim_year, use_case, l_ncpl
+# configuration, structure, irrigate, verbose, clm_demand, ssp_rcp, test, sim_year, use_case
 my $startfile = "clmrun.clm2.r.1964-05-27-00000.nc";
 foreach my $options ( "-configuration nwp",
                       "-structure fast",
                       "-namelist '&a irrigate=.true./'", "-verbose", "-ssp_rcp SSP1-2.6", "-test", "-sim_year 1850",
-                      "-use_case 1850_control", "-l_ncpl 1", 
+                      "-use_case 1850_control",
                       "-clm_start_type startup", "-namelist '&a irrigate=.false./' -crop -bgc bgc",
                       "-envxml_dir . -infile myuser_nl_clm", 
                       "-ignore_ic_date -clm_start_type branch -namelist '&a nrevsn=\"thing.nc\"/' -bgc bgc -crop",
@@ -309,10 +309,7 @@ sub make_config_cache {
    $cfiles->checkfilesexist( "$options", $mode );
    $cfiles->shownmldiff( "default", $mode );
    my $finidat = `grep finidat lnd_in`;
-   if (      $options eq "-l_ncpl 1" ) {
-      my $dtime = `grep dtime lnd_in`;
-      like( $dtime, "/ 86400\$/", "$options" );
-   } elsif ( $options =~ /myuser_nl_clm/ ) {
+   if ( $options =~ /myuser_nl_clm/ ) {
       my $fsurdat =  `grep fsurdat lnd_in`;
       like( $fsurdat, "/MYDINLOCROOT/lnd/clm2/PTCLMmydatafiles/1x1pt_US-UMB/surfdata_1x1pt_US-UMB_simyr2000_clm4_5_c131122.nc/", "$options" );
    }
@@ -391,21 +388,6 @@ sub make_config_cache {
                                      GLC_TWO_WAY_COUPLING=>"FALSE",
                                      phys=>"clm5_0",
                                    },
-     "l_ncpl is zero"            =>{ options=>"-l_ncpl 0 -envxml_dir .",
-                                     namelst=>"",
-                                     GLC_TWO_WAY_COUPLING=>"FALSE",
-                                     phys=>"clm5_0",
-                                   },
-     "l_ncpl not integer"        =>{ options=>"-l_ncpl 1.0 -envxml_dir .",
-                                     namelst=>"",
-                                     GLC_TWO_WAY_COUPLING=>"FALSE",
-                                     phys=>"clm5_0",
-                                   },
-     "both l_ncpl and dtime"     =>{ options=>"-l_ncpl 24 -envxml_dir .",
-                                     namelst=>"dtime=1800",
-                                     GLC_TWO_WAY_COUPLING=>"FALSE",
-                                     phys=>"clm5_0",
-                                   },
      "use_crop without -crop"    =>{ options=>" -envxml_dir .",
                                      namelst=>"use_crop=.true.",
                                      GLC_TWO_WAY_COUPLING=>"FALSE",