Write non-HISTORY output into ./scratch (get_io_filename)

[gmao-rreichle]

Problem statement:

In the SMAP L4_SM assimilation use case (i.e., the global/assim test), the following
sets of GEOSldas output files appear to be written directly into their final year/month
directories by GEOSldas.x via function get_io_filename():

  ana/ensXXXX/Yyyyy/Mmm/*.ldas_ObsFcstAna.*.bin
  ana/ensXXXX/Yyyyy/Mmm/*.ldas_tile_inst_smapL4SMaup.*.bin

  rc_out/Yyyyy/Mmm/*.ldas_domdecomp.*.txt

  rc_out/Yyyyy/Mmm/*.ldas_ensprop_inputs.*.nml
  rc_out/Yyyyy/Mmm/*.ldas_ensupd_inputs.*.nml

  rc_out/Yyyyy/Mmm/*.ldas_log.*.txt

  rc_out/Yyyyy/Mmm/*.ldas_obslog.*.txt
  rc_out/Yyyyy/Mmm/*.ldas_obsparam.*.txt

  rc_out/Yyyyy/Mmm/*.ldas_smapL4SMlmc.*.bin

This fails when an existing simulation is extended into a new month without
running ldas_setup again, which is a common and useful approach. In this case,
the year/month subdirectories are not created in pre-processing.

There is currently a patch in GEOSldas such that GEOSldas.x creates year/month
directories in ana/ensXXXX/ using Execute_command_line(). This call fails
occasionally when running an executable built with the Gnu compiler.

I'm not sure how the year/month directories are created in rc_out/, and why
the above fix seems sufficient. Possibly the files that end up in
the year/month directories in rc_out/ are already written into scratch?

---

Tentative solution:

Start from branch "feature/RR_getiofilename" (synced w/ develop on 2 May 2020).

All of the above files should be written into the scratch/ directory and then
moved into their final year/month directories during post-processing.

Using the branch "feature/RR_getiofilename", the files can be written directly
into ./scratch by changing the calls to get_io_filename() as follows (pseudo-code):

scratch_path = './'

fname = get_io_filename( scratch_path, exp_id, file_tag, date_time=date_time, &
          dir_name=dir_name, ens_id=ens_id, option=option, no_subdirs=.true. )

where all arguments other than "scratch_path" and "no_subdirs" depend on the
specific instance and remain as before.

The function get_io_filename() is used in the following files:

Applications/LDAS_App/preprocess_ldas.F90

--> No change needed. Only runs in pre-processing.

Components/GEOSldas_GridComp/GEOS_LdasGridComp.F90

--> Write "ldas_domdecomp" file to ./scratch instead of "OUT_PATH"

Components/GEOSldas_GridComp/GEOSlandassim_GridComp/clsm_adapt_routines.F90
Components/GEOSldas_GridComp/GEOSlandassim_GridComp/clsm_bias_routines.F90

--> No change needed. Read only for "ldas_catbias_inputs" file. Not currently
used otherwise.

Components/GEOSldas_GridComp/GEOSlandassim_GridComp/clsm_ensdrv_out_routines.F90

--> Not sure about "ldas_log" file. May require changes to get_io_filename()

--> Not sure about "ldas_smapL4SMlmc" file. May require changes to get_io_filename()

Components/GEOSldas_GridComp/GEOSlandassim_GridComp/clsm_ensupd_enkf_update.F90

--> Write "ldas_ObsFcstAna" files to ./scratch. Remove call to Execute_command_line().

--> Write "ldas_tile_inst_smapL4SMaup" files to ./scratch.

Components/GEOSldas_GridComp/GEOSlandassim_GridComp/clsm_ensupd_upd_routines.F90

--> Write "ldas_ensupd_inputs" file to ./scratch?

--> Write "ldas_obsparam" file to ./scratch?

--> Write "ldas_obslog" file to ./scratch?

Components/GEOSldas_GridComp/GEOSlandpert_GridComp/LDAS_PertRoutines.F90

--> Write "ldas_ensprop_inputs" file to ./scratch?

--> No change needed for "pert_ldas_rst" files (read only).

Components/GEOSldas_GridComp/Shared/LDAS_ensdrv_init_routines.F90

--> No change needed for "ldas_domain", "ldas_tilecoord" and "ldas_tilegrids" files.
These files are not time-stamped and are not written into year/month directories.

---

[weiyuan-jiang]

This will need to change lenkf.j.template. Should we approve #202 first?

[gmao-rreichle]

Yes, #202 should be finalized first, but #202 is not quite ready for approval.

[weiyuan-jiang]

quote: "I'm not sure how the year/month directories are created in rc_out/" . These are created during the ldas_setup and post processing. The YYYY/MM would be there for starting and continuation

[gmao-rreichle]

As discusssed, the txt, nml and bin files in ./rc_out/year/month can be written to this location directly and do not need to go through ./scratch. These files only need the initial year/month directory, which already exists.
Only the .ldas_ObsFcstAna.* and .ldas_tile_inst_smapL4SMaup. files should go into ./scratch and be moved into the ana/ensXXXX/year/month directory during post-processing (create directories as needed during post-processing).

[gmao-rreichle]

addressed with pull request #202