updates for RRFS_A real-time

[hu5970]

Several bug fixes and updates from RRFS_A real- time tests:

1. Change default value to 1 for:
   DA_CYCLE_INTERV
   RESTART_INTERVAL
   RESTART_INTERVAL_LONG

2. When run GSI hybrid with RRFS ensemble, only retrospective experiment
   needs to check if RRFS ensembles exist. The real-time run does not need to check.

3. bug fix for running FVCOM on wcoss2

4. tune resources used for run model, UPP on wcoss2

5. change default version to 0.3.6

[daviddowellNOAA]

If RDAS ensemble DA is proposed to be part of the RRFSv1 implementation then I disagree with the strategy of not waiting for the RDAS ensemble to be available for the DA. Instead, we should be working to make the ensemble available for as many hours as possible as we set up the prototype system. Having the check for ensemble availability will be helpful during the early stages of development. Later, we might be able to refine the logic based on what relative timing of the ensemble and control seems feasible.

[hu5970]

@daviddowellNOAA I should make the statement more clear, the real-time control runs will not wait for the late arrival of the RDAS ensemble. In real-time run, we assume the RDAS will arrive before certain time and the control run GSI will start at that time to check if RDAS exist or not. If not, use the global ensemble. If most of the RDAS arrives late, we need to improve the wall-time for RDAS cycles or delay the start of the control runs. Strict dependence is applied only for retrospective runs.

[daviddowellNOAA]

Ming, thanks for clarifying. I'm still concerned about the possibility of the control member running for long periods of time without using the RDAS ensemble for DA. Thus, we would be providing output that is unrepresentative of what we are proposing for RRFSv1*. If we handle this issue in the workflow, then I would propose a time we're willing to wait for the regional ensemble. It seems that the dependencies on START_TIME_CONVENTIONAL and PREP_CYC define an optimum time to start the analysis. But presumably there is an additional amount of time (several minutes?) that we're willing to wait for the regional ensemble and still have the opportunity to catch up in future cycles.

If we don't handle this issue in the workflow, then I think we need some way to be notified when we're not using the regional ensemble, e.g., a web-based monitoring tool and/or an email that gets sent out each time we miss using the regional ensemble.

*Curtis or Jacob, can you comment on the current thinking for RRFSv1? Will it include the regional (RDAS) ensemble?

[hu5970]

David: We can add a check in the analysis scripts to wait for regional ensemble, maybe 2-5 minutes if needed. We will decide if we need this based on how stable ensemble DA runs on WCOSS2. We can also add a message in cycle.log to document the ensemble forecast used for the GSI hybrid. This change is to make the best effort to use the regional ensemble while not impact the requirement for a stable parallel real-time system runs.

[daviddowellNOAA]

Thanks, Ming.

[hu5970]

@daviddowellNOAA I added a message to document the GSI analysis type in cycle.log. We will decide if we need to add some scripts to hold the GSI for late regional ensembles when we run the RRFS_A with regional ensembles in real-time.

[daviddowellNOAA]

Excellent, thanks Ming.