gdaspostanl should report failure if files do not show up before job ends

[WalterKolczynski-NOAA]

If the loop waiting for output files to show up exhausts all iterations before files are available, the job should register as failed, not succeeded.

One GDAS cycle (2020083018) has several analysis files missing. There was a failure in the gdasanalcalc step, but upon rerun it completed successfully. However, the gdaspostanl step timed out in the meantime and skipped processing the analysis. There was no failure recorded for this task:

[WalterKolczynski-NOAA]

@lgannoaa please submit it as a PR to the post repo

[lgannoaa]

Created issue 443 in UPP because script in question exgdas_atmos_nceppost.sh and exgfs_atmos_nceppost.sh is belong to UPP repo.

[lgannoaa]

UPP CM has merged my fix PR. Close this issue.

[WalterKolczynski-NOAA]

This needs to stay open until we update our post version in the checkout script.

[lgannoaa]

@WenMeng-NOAA , there is a PR #444 that addressed this issue. Looks like you merged that in. The Global Workflow develop currently checkout upp_v10.0.11. This version does not contain this fix. Would you please provide us another UPP version that included this fix for us to checkout?
Thanks
Lin

[WenMeng-NOAA]

@lgannoaa I assume you are meaning the UPP PR #444. You may checkout the latest UPP develop branch revision 5f60e60.

[WalterKolczynski-NOAA]

I'm using the current UPP develop in moving all the scripts to global-workflow (Issue #769), so this issue will be fixed by that as well.

[WenMeng-NOAA]

@WalterKolczynski-NOAA Before you take further actions of moving out all gfs post processing scripts from UPP repository to global-workflow, some coordination efforts are needed between managements of VPB and EIB, like GFS product user support, GFS post processing operation support handoff. @HuiyaChuang-NOAA may chime in.

[HuiyaChuang-NOAA]

@WalterKolczynski-NOAA @arunchawla-NOAA

Hi, Walter, I just spoke to Arun. We're supportive of EIB taking over GFS post processing scripts but given Wen is pretty busy and also her first check indicated your new scripts have inadvertently removed existing operational GFS products, I would
like to propose the followings going forward:

1. we will not remove any post processing scripts from UPP repository until EIB takes over all post processing scripts and both post processing team and EIB verify all operational capabilities are being replicated by EIB's new post processing scripts.
2. with that said, please proceed to transition all gfs post processing scripts to EIB workflow and when you've completed all transition, please reach out to us, and we will check the final scripts and products.

Again, I recommend that we try to replicate operational GFS post processing capabilities first. Thank you. I will also send this as an email to a bigger group.

[WalterKolczynski-NOAA]

@HuiyaChuang-NOAA @WenMeng-NOAA we should probably move this conversation over to Issue #769. The matter was only incidentally mentioned here.

[WalterKolczynski-NOAA]

When we moved all the UPP scripts into workflow, we updated to the tip of the UPP develop, so this issue should be satisfied.