start concatenation of subdaily-to-daily nc4 files for each day

[gmao-rreichle]

The concatenation of subdaily-to-daily nc4 files is kicked off only at the end of each month.
Some SMAP runs require (concatenated) daily nc4 files before the end of the month.
@weiyuan-jiang : Please change the script so that the subdaily-to-daily nc4 concatenation is kicked off for each completed day, even if the month is not yet complete.

cc: @gmao-qliu @smahanam

[gmao-qliu]

the concatenation of subdaily-to-daily nc4 files should kick off at the end of each segment after the output files were moved from scratch/ to output/ . Modifying the "if" statement for executing the concatenation should be sufficient.

[smahanam]

@weiyuan-jiang I haven't deleted the experiment I set up to debug that concatenation issue, let me know if you need me to do anything.

Like @weiyuan-jiang said , issue 175, must be misinterpretting a different error. lenkf.j.template was set up to add directorirs right from the start, even in the very first tutorial we said, if one needs to extend, just change END_DATE in CAP.rc and submit.

It is also possible there could be a broken shell programming rule - like the concattenating sub-daily issue. Otherwise the line that Weiyuan mentioned is in the script and it creates directories if needed

[weiyuan-jiang]

@smahanam Do you have any good idea to concatenate the sub-daily? I guess we need double loop. First get the daily from sub-daily , then get monthly.

[smahanam]

@weiyuan-jiang
Please take a look at /discover/nobackup/smahanam/lenkf.j.template

Do not copy that file to the repository, just diff the relevant section and copy that section where NCO post processes nc4 files.
It worked with Qing's M04 2 files, and worked again, when I made some dummy files and processed April 2 too.

[weiyuan-jiang]

@smahanam . Thanks. I will take your version. I made a minor change . The second cut -d'.' to cut -d'_'
from
set time_steps = ls -1 $EXPID.$ThisCol.${YYYY}${MM}* | rev | cut -d'.' -f2 | cut -d'.' -f1 | rev
to
set time_steps = ls -1 $EXPID.$ThisCol.${YYYY}${MM}* | rev | cut -d'.' -f2 | cut -d'_' -f1 | rev

[smahanam]

I think it is wrong @weiyuan-jiang when the script gets to that line, the directory does not have "_HHMMz.nc4" files, what it has is YYYYMMDD.nc4 - either dailies or concatenated subdailies.

I still have the set up, let me check a full month

[weiyuan-jiang]

@smahanam After the first cut -d'.', there is no '.' for you to cut any more. We need to make changes to that command. ( My change may be wrong though )

[smahanam]

The 2nd cut is not needed anymore, basically it cuts nothing the string contains only YYYYMMDD when the script gets to that line

[smahanam]

ohh ok we both said the same

[gmao-rreichle]

Addressed with pull request #181

[gmao-rreichle]

A bug was discovered in the above fix.
Reopened issue.
To be addressed with pull request #202

[gmao-rreichle]

Addressed with pull request #202