Add the Multigrid Beta Filter (MGBF) for ensemble localization (#699)

[shoyokota]

DUE DATE for merger of this PR into develop is 3/29/2024 (six weeks after PR creation).

Description

Resolves #699

This PR is to add the option to apply Multigrid Beta Filter (MGBF; Purser et al. 2022) for ensemble localization instead of Recursive Filter (RF). This work includes to add an initial version of the MGBF as a subdirectory in GSI.

To apply the MGBF, set "l_mgbf_loc=true" in the namelist and additionally input "mgbf_loc01.nml". (In Scale/Variable-Dependent Localization, input also "mgbf_locXX.nml" (XX=02,03,...) with the same number of grid points.)

How to set MGBF parameters in mgbf_locXX.nml

An example of mgbf_locXX.nml is as follows:

&PARAMETERS_MGBETA
  mg_ampl01=1.125,            ! length of vertical beta filter (standard deviation; filter grid unit)
  mg_ampl02=2.4,              ! length of horizontal beta filter (standard deviation; filter grid unit)
  mg_ampl03=0.85,             ! length of 3D beta filter (standard deviation; filter grid unit)
  mg_weig1=0.,                ! weight of generation 1
  mg_weig2=0.,                ! weight of generation 2
  mg_weig3=0.,                ! weight of generation 3
  mg_weig4=1.,                ! weight of generation 4
  hx=5,                       ! number of halo grid points in x-direction
  hy=5,                       ! number of halo grid points in y-direction
  hz=3,                       ! number of halo grid points in z-direction
  p=2,                        ! beta filter exponent
  mgbf_line=.false.,          ! set false except for mgbf_proc=2,4,7
  mgbf_proc=8,                ! 1-2: 3D filter; 3-5: 2D filter for static B; 6-8: 2D filter for localization (1,3,6: radial filter; 2,4,7: line filter; 5,8: isotropic line filter)
  lm_a=65,                    ! number of vertical layers in analysis grid
  lm=33,                      ! number of vertical layers in filter grid
  km2=0,                      ! number of 2D variables (set 0 for localization)
  km3=1,                      ! number of 3D variables (set 1 for localization)
  n_ens=30,                   ! ensemble size
  l_loc=.true.,               ! set true in localization
  l_filt_g1=.false.,          ! set false in skipping generation 1
  l_lin_vertical=.true.,      ! set true in applying linear vertical interpolation for analysis-filter mapping
  l_lin_horizontal=.true.,    ! set true in applying linear horizontal interpolation for analysis-filter mapping
  l_quad_horizontal=.false.,  ! set true in applying quadratic horizontal interpolation for analysis-filter mapping
  l_new_map=.true.,           ! set true in applying efficient vertical interpolation for analysis-filter mapping
  l_vertical_filter=.true.,   ! set true in applying vertical beta filter outside 2D filter
  ldelta=.false.,             ! (not used)
  lquart=.false.,             ! set true in applying quadratic horizontal interpolation for up/down-sending
  lhelm=.false.,              ! set true in applying Helmholtz differential operator for weighting
  nm0=1975,                   ! number of analysis grid points in x-direction
  mm0=1350,                   ! number of analysis grid points in y-direction
  gm_max=4,                   ! highest generation (max: 4)
  nxPE=79,                    ! number of MPI processors in x-direction
  nyPE=54,                    ! number of MPI processors in y-direction
  im_filt=8,                  ! number of filter grid points in each MPI processor in x-direction
  jm_filt=8,                  ! number of filter grid points in each MPI processor in y-direction
 /

Here, to make the result of MGBF-based localization similar to RF-based one, we can set the beta filter length ( mg_ampl0[12] ) from the recursive filter length in the GSI namelist ( s_ens_[vh] ) as:

• $\text{mg\_ampl01} = \left[\text{s\_ens\_v (grid unit)} * \frac{1}{\sqrt{2}} * \frac{\text{lm}-1}{\text{lm\_a}-1} \right]^2$
• $\text{mg\_ampl02} = \left[\frac{\text{s\_ens\_h (km)}}{\text{analysis grid interval (km)}} * \frac{1}{\sqrt{2}} * \frac{\text{im\_filt} * \text{nxPE}}{\text{nm0}} * \frac{1}{2} * \frac{1}{2} * \frac{1}{2} \right]^2$ (in case mg_weig[1-4]=[0,0,0,1])

Please note there are some limitations for the other MGBF parameters such as:

• The number of MPI processors input in GSI should be nxPE x nyPE
• (nm0, mm0, lm_a) should be the same as the GSI analysis grid
• nm0 should be divisible by nxPE
• mm0 should be divisible by nyPE
• nm0 / nxPE = mm0 / nyPE

How to run RRFS regression tests with MGBF-based localization

Change settings in regression/ as follows, and run Test#3 (rrfs_3denvar_glbens)

diff --git a/regression/regression_namelists.sh b/regression/regression_namelists.sh
index 7ca183ef3..671d028ff 100755
--- a/regression/regression_namelists.sh
+++ b/regression/regression_namelists.sh
@@ -457,7 +457,7 @@ OBS_INPUT::
    beta_s0=0.15,s_ens_h=110,s_ens_v=3,
    regional_ensemble_option=1,
    pseudo_hybens = .false.,
-   grid_ratio_ens = 3,
+   grid_ratio_ens = 5.1,
    l_ens_in_diff_time=.true.,
    ensemble_path='',
    i_en_perts_io=1,
@@ -465,6 +465,7 @@ OBS_INPUT::
    fv3sar_bg_opt=0,
    readin_localization=.true.,
    ens_fast_read=.false.,
+   l_mgbf_loc=.true.,
  /
  &RAPIDREFRESH_CLDSURF
    dfi_radar_latent_heat_time_period=20.0,

diff --git a/regression/regression_namelists_db.sh b/regression/regression_namelists_db.sh
index e03917e88..36b8b6a22 100755
--- a/regression/regression_namelists_db.sh
+++ b/regression/regression_namelists_db.sh
@@ -438,7 +438,7 @@ OBS_INPUT::
    beta_s0=0.15,s_ens_h=110,s_ens_v=3,
    regional_ensemble_option=1,
    pseudo_hybens = .false.,
-   grid_ratio_ens = 3,
+   grid_ratio_ens = 5.1,
    l_ens_in_diff_time=.true.,
    ensemble_path='',
    i_en_perts_io=1,
@@ -446,6 +446,7 @@ OBS_INPUT::
    fv3sar_bg_opt=0,
    readin_localization=.true.,
    ens_fast_read=.false.,
+   l_mgbf_loc=.true.,
  /
  &RAPIDREFRESH_CLDSURF
    dfi_radar_latent_heat_time_period=20.0,

diff --git a/regression/regression_param.sh b/regression/regression_param.sh
index 2ac615fc4..6186acdbb 100755
--- a/regression/regression_param.sh
+++ b/regression/regression_param.sh
@@ -87,23 +87,23 @@ case $regtest in
     rrfs_3denvar_glbens)

         if [[ "$machine" = "Hera" ]]; then
-           topts[1]="0:15:00" ; popts[1]="20/1/"  ; ropts[1]="/1"
-           topts[2]="0:15:00" ; popts[2]="20/2/"  ; ropts[2]="/1"
+           topts[1]="0:15:00" ; popts[1]="30/2/"  ; ropts[1]="/1"
+           topts[2]="0:15:00" ; popts[2]="15/4/"  ; ropts[2]="/1"
         elif [[ "$machine" = "Orion" ]]; then
-           topts[1]="0:15:00" ; popts[1]="20/1/" ; ropts[1]="/1"
-           topts[2]="0:15:00" ; popts[2]="20/2/" ; ropts[2]="/2"
+           topts[1]="0:15:00" ; popts[1]="30/2/"  ; ropts[1]="/1"
+           topts[2]="0:15:00" ; popts[2]="15/4/"  ; ropts[2]="/2"
         elif [[ "$machine" = "Hercules" ]]; then
-           topts[1]="0:15:00" ; popts[1]="20/1/" ; ropts[1]="/1"
-           topts[2]="0:15:00" ; popts[2]="20/2/" ; ropts[2]="/2"
+           topts[1]="0:15:00" ; popts[1]="30/2/"  ; ropts[1]="/1"
+           topts[2]="0:15:00" ; popts[2]="15/4/"  ; ropts[2]="/2"
         elif [[ "$machine" = "Jet" ]]; then
-           topts[1]="0:15:00" ; popts[1]="20/1/"  ; ropts[1]="/1"
-           topts[2]="0:15:00" ; popts[2]="20/2/"  ; ropts[2]="/1"
+           topts[1]="0:15:00" ; popts[1]="15/4/"  ; ropts[1]="/1"
+           topts[2]="0:15:00" ; popts[2]="10/6/"  ; ropts[2]="/1"
         elif [[ "$machine" = "Gaea" ]]; then
-           topts[1]="0:15:00" ; popts[1]="18/1/"  ; ropts[1]="/1"
-           topts[2]="0:15:00" ; popts[2]="18/2/"  ; ropts[2]="/1"
+           topts[1]="0:15:00" ; popts[1]="15/4/"  ; ropts[1]="/1"
+           topts[2]="0:15:00" ; popts[2]="10/6/"  ; ropts[2]="/1"
         elif [[ "$machine" = "wcoss2" ]]; then
-           topts[1]="0:15:00" ; popts[1]="64/1/" ; ropts[1]="/1"
-           topts[2]="0:15:00" ; popts[2]="128/2/" ; ropts[2]="/1"
+           topts[1]="0:15:00" ; popts[1]="60/1/"  ; ropts[1]="/1"
+           topts[2]="0:15:00" ; popts[2]="30/2/"  ; ropts[2]="/1"
         fi

         if [ "$debug" = ".true." ] ; then

diff --git a/regression/rrfs_3denvar_glbens.sh b/regression/rrfs_3denvar_glbens.sh
index af5da5117..04fd73d57 100755
--- a/regression/rrfs_3denvar_glbens.sh
+++ b/regression/rrfs_3denvar_glbens.sh
@@ -272,6 +272,46 @@ $gsi_namelist

 EOF

+cat << EOF > mgbf_loc01.nml
+&PARAMETERS_MGBETA
+  mg_ampl01=1.125,            ! length of vertical beta filter (standard deviation; filter grid unit)
+  mg_ampl02=1.615,            ! length of horizontal beta filter (standard deviation; filter grid unit)
+  mg_ampl03=0.85,             ! length of 3D beta filter (standard deviation; filter grid unit)
+  mg_weig1=0.,                ! weight of generation 1
+  mg_weig2=1.,                ! weight of generation 2
+  mg_weig3=0.,                ! weight of generation 3
+  mg_weig4=0.,                ! weight of generation 4
+  hx=4,                       ! number of halo grid points in x-direction
+  hy=4,                       ! number of halo grid points in y-direction
+  hz=3,                       ! number of halo grid points in z-direction
+  p=2,                        ! beta filter exponent
+  mgbf_line=.false.,          ! set false except for mgbf_proc=2,4,7
+  mgbf_proc=8,                ! 1-2: 3D filter; 3-5: 2D filter for static B; 6-8: 2D filter for localization (1,3,6: radial filter; 2,4,7: line filter; 5,8: isotropic line filter)
+  lm_a=65,                    ! number of vertical layers in analysis grid
+  lm=33,                      ! number of vertical layers in filter grid
+  km2=0,                      ! number of 2D variables (set 0 for localization)
+  km3=1,                      ! number of 3D variables (set 1 for localization)
+  n_ens=10,                   ! ensemble size
+  l_loc=.true.,               ! set true in localization
+  l_filt_g1=.false.,          ! set false in skipping generation 1
+  l_lin_vertical=.true.,      ! set true in applying linear vertical interpolation for analysis-filter mapping
+  l_lin_horizontal=.true.,    ! set true in applying linear horizontal interpolation for analysis-filter mapping
+  l_quad_horizontal=.false.,  ! set true in applying quadratic horizontal interpolation for analysis-filter mapping
+  l_new_map=.true.,           ! set true in applying efficient vertical interpolation for analysis-filter mapping
+  l_vertical_filter=.true.,   ! set true in applying vertical beta filter outside 2D filter
+  ldelta=.false.,             ! (not used)
+  lquart=.false.,             ! set true in applying quadratic horizontal interpolation for up/down-sending
+  lhelm=.false.,              ! set true in applying Helmholtz differential operator for weighting
+  nm0=40,                     ! number of analysis grid points in x-direction
+  mm0=24,                     ! number of analysis grid points in y-direction
+  gm_max=2,                   ! highest generation (max: 4)
+  nxPE=10,                    ! number of MPI processors in x-direction
+  nyPE=6,                     ! number of MPI processors in y-direction
+  im_filt=4,                  ! number of filter grid points in each MPI processor in x-direction
+  jm_filt=4,                  ! number of filter grid points in each MPI processor in y-direction
+ /
+EOF
+
 # Copy executable and fixed files to $tmpdir
 if [[ $exp == *"updat"* ]]; then
    $ncp $gsiexec_updat  ./gsi.x

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

EnVar for NA-domain RRFS was tested with "mgbf_locXX.nml" (XX=01) shown above on Orion. The resulting analysis increment was similar to the original and the computation time for localization became short.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

[shoyokota]

@ShunLiu-NOAA Thank you for managing this PR. Could you additionally request review from @MiodragRancic-NOAA? This PR is mainly based on his code.

[shoyokota]

@ShunLiu-NOAA Sorry, this unassignment was my mistake... Could you assign me again?

[TingLei-NOAA]

@shoyokota As we discussed off-line, now I have two suggestions: 1) re-organize to let the mgbf lib as a separate lib. The availability of such a form is one of the benefit for such a oops-style standing alone mgbf lib. 2), I suggest to add log information about authors/major coding histories in the codes especially for the mgbf lib part. Such a mgbf lib with enough information about its' authors and major developing progresses could also be used for JEDI's mgbf component. I hope the major mgbf developers at EMC like @MiodragRancic-NOAA @JimPurser-NOAA and @ManuelPondeca-NOAA could help with this.

[shoyokota]

@TingLei-NOAA Thank you for the suggestions.
(1) I moved src/gsi/mgbf to src/mgbf and re-organized it as a separate lib. This mgbf lib is linked in compiling gsi as like the other libraries.
(2) I added GSI-style headers in src/mgbf/*f90. @JimPurser-NOAA and @MiodragRancic-NOAA, these headers are based on the comments in your MGBF code. If you find wrong information in the headers or want to revise them, please let me know.

[ShunLiu-NOAA]

@ManuelPondeca-NOAA, @guoqing-noaa and @TingLei-NOAA Could you please review this PR? MGBF might be implemented in RTMA in operation. Given Sho's return to JMA at the end of the next month, it is better that we can address the isusses before Sho departs EMC.

[MiodragRancic-NOAA]

Parameters for localization and background error will be need to be held separately. Application to background error will use six grid generations (probably) because of very large correlation lengths of stream function. In addition, aspect tensors and scale weights will all be depended of model level and variable. Also, the filter will be applied to 2D variable, not only to 3D variables as in the case of filtering.

[MiodragRancic-NOAA]

It would be nice to order subroutines as they are ordered in the case select statement.

[shoyokota]

@MiodragRancic-NOAA Thank you for reviewing the code.

Parameters for localization and background error will be need to be held separately. Application to background error will use six grid generations (probably) because of very large correlation lengths of stream function. In addition, aspect tensors and scale weights will all be depended of model level and variable. Also, the filter will be applied to 2D variable, not only to 3D variables as in the case of filtering.

Thank you for clarifying what is additionally required for MGBF-based static background error. Yes, what you are suggesting should be implemented in another PR to apply MGBF-based static background error.

With the MGBF library added in this PR, we can input two independent parameter files for localization and static background error by separately calling mg_initialize even without adding another MGBF library. So, I expect that MGBF-based static background error also will be implemented based on this MGBF library with some modifications.

It would be nice to order subroutines as they are ordered in the case select statement.

I modified the order in "select case" in mg_filtering.f90.

[MiodragRancic-NOAA]

It would be also good to add comments in mg_paramater and the namelist from which flag mg_filt gets values that clarify its meaning

[MiodragRancic-NOAA]

Ting, Are you referring to application of vertical beta filter? Thanks Misha Miodrag Rancic Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct, Rm 2786 College Park, MD 20740 301-683-3732

…

On Fri, Mar 1, 2024 at 10:58 AM TingLei-NOAA ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In src/gsi/hybrid_ensemble_isotropic.F90 <#700 (comment)>: > real(r_kind),allocatable,dimension(:):: a_en_work ipnt=1 ! Apply vertical smoother on each ensemble member + if(regional.and.l_mgbf_loc.and.ig==1) then + allocate(hwork_mgbf(obj_mgbf(1)%km_a_all,obj_mgbf(1)%nm,obj_mgbf(1)%mm)) @shoyokota <https://github.com/shoyokota> is the size of hwork_mgbf the same as (lon2, lat2, :) or (lon1,lat1,:)? Second, to avoid confusion, I suggest to use work_mgbf /work_tmp rather than hwork_mgbf/hwork_tmp, because, in gsi, hwork is used for arrays of (nlat, nlon, :) (namely,pieces on the whole the horizontal domain). — Reply to this email directly, view it on GitHub <#700 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJHP7FVHMI3BECCI6JJWMADYWCQRXAVCNFSM6AAAAABDMRRGTSVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSMJRGQ3TSMRVGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[TingLei-NOAA]

Misha, Here, I meant the whole mgbf process. I am wondering how halo points are treated, by codes in GSI or delegated to mgbf functions. Regards, Ting

…

______________________________ Ting Lei Physical Scientist, Contractor with Lynker in support of EMC/NCEP/NWS/NOAA 5830 University Research Ct., Cubicle 2765 College Park, MD 20740 ***@***.*** 301-683-3624 On Fri, Mar 1, 2024 at 11:39 AM MiodragRancic-NOAA ***@***.***> wrote:

Ting, Are you referring to application of vertical beta filter? Thanks Misha Miodrag Rancic Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct, Rm 2786 College Park, MD 20740 301-683-3732 On Fri, Mar 1, 2024 at 10:58 AM TingLei-NOAA ***@***.***> wrote: > ***@***.**** commented on this pull request. > ------------------------------ > > In src/gsi/hybrid_ensemble_isotropic.F90 > <#700 (comment)>: > > > real(r_kind),allocatable,dimension(:):: a_en_work > > ipnt=1 > > ! Apply vertical smoother on each ensemble member > + if(regional.and.l_mgbf_loc.and.ig==1) then > + allocate(hwork_mgbf(obj_mgbf(1)%km_a_all,obj_mgbf(1)%nm,obj_mgbf(1)%mm)) > > @shoyokota <https://github.com/shoyokota> is the size of hwork_mgbf the > same as (lon2, lat2, :) or (lon1,lat1,:)? > Second, to avoid confusion, I suggest to use work_mgbf /work_tmp rather > than hwork_mgbf/hwork_tmp, because, in gsi, > hwork is used for arrays of (nlat, nlon, :) (namely,pieces on the whole > the horizontal domain). > > — > Reply to this email directly, view it on GitHub > <#700 (review)>, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AJHP7FVHMI3BECCI6JJWMADYWCQRXAVCNFSM6AAAAABDMRRGTSVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSMJRGQ3TSMRVGE> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub <#700 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APEFS7GADSI2DPSKULUAHDTYWCVLPAVCNFSM6AAAAABDMRRGTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZTGUYTEOBUHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[MiodragRancic-NOAA]

Ting, What you mean by ‘halo’ points? I have developed all message passing code relevant to MGBF. Replacing that would be too risky, time consuming and unnecessary! Misha Miodrag Rancic Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct, Rm 2786 College Park, MD 20740 301-683-3732 On Fri, Mar 1, 2024 at 1:14 PM TingLei-NOAA ***@***.***> wrote:

…

Misha, Here, I meant the whole mgbf process. I am wondering how halo points are treated, by codes in GSI or delegated to mgbf functions. Regards, Ting ______________________________ Ting Lei Physical Scientist, Contractor with Lynker in support of EMC/NCEP/NWS/NOAA 5830 University Research Ct <https://www.google.com/maps/search/5830+University+Research+Ct?entry=gmail&source=g>., Cubicle 2765 College Park, MD 20740 ***@***.*** 301-683-3624 On Fri, Mar 1, 2024 at 11:39 AM MiodragRancic-NOAA ***@***.***> wrote: > Ting, > > Are you referring to application of vertical beta filter? > > Thanks > > Misha > > Miodrag Rancic > Lynker at NOAA/NWS/NCEP/EMC > 5830 University Research Ct, Rm 2786 <https://www.google.com/maps/search/5830+University+Research+Ct,+Rm+2786%0D%0A+%0D%0A+College+Park,+MD+20740?entry=gmail&source=g> <https://www.google.com/maps/search/5830+University+Research+Ct,+Rm+2786%0D%0A+%0D%0A+College+Park,+MD+20740?entry=gmail&source=g>> College Park, MD 20740 <https://www.google.com/maps/search/5830+University+Research+Ct,+Rm+2786%0D%0A+%0D%0A+College+Park,+MD+20740?entry=gmail&source=g> > 301-683-3732 > > > On Fri, Mar 1, 2024 at 10:58 AM TingLei-NOAA ***@***.***> > wrote: > > > ***@***.**** commented on this pull request. > > ------------------------------ > > > > In src/gsi/hybrid_ensemble_isotropic.F90 > > <#700 (comment)>: > > > > > real(r_kind),allocatable,dimension(:):: a_en_work > > > > ipnt=1 > > > > ! Apply vertical smoother on each ensemble member > > + if(regional.and.l_mgbf_loc.and.ig==1) then > > + > allocate(hwork_mgbf(obj_mgbf(1)%km_a_all,obj_mgbf(1)%nm,obj_mgbf(1)%mm)) > > > > @shoyokota <https://github.com/shoyokota> is the size of hwork_mgbf the > > same as (lon2, lat2, :) or (lon1,lat1,:)? > > Second, to avoid confusion, I suggest to use work_mgbf /work_tmp rather > > than hwork_mgbf/hwork_tmp, because, in gsi, > > hwork is used for arrays of (nlat, nlon, :) (namely,pieces on the whole > > the horizontal domain). > > > > — > > Reply to this email directly, view it on GitHub > > <#700 (review)>, > > > or unsubscribe > > < > https://github.com/notifications/unsubscribe-auth/AJHP7FVHMI3BECCI6JJWMADYWCQRXAVCNFSM6AAAAABDMRRGTSVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSMJRGQ3TSMRVGE> > > > . > > You are receiving this because you were mentioned.Message ID: > > ***@***.***> > > > > — > Reply to this email directly, view it on GitHub > <#700 (comment)>, or > unsubscribe > < https://github.com/notifications/unsubscribe-auth/APEFS7GADSI2DPSKULUAHDTYWCVLPAVCNFSM6AAAAABDMRRGTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZTGUYTEOBUHA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub <#700 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJHP7FX244OORTZBBQ54ZDDYWDAPLAVCNFSM6AAAAABDMRRGTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZTGY4DIMJSGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[TingLei-NOAA]

@shoyokota and @GangZhao-NOAA On wcoss2, when gsi was built with debug mode, for rrfs regression test , the first rrfs_3denvar_glbens_loproc_updat failed for

 nlat,nlon=nya,nxa=          117         199                   
forrtl: severe (71): integer divide by zero
Image              PC                Routine            Line        Source     
                   
gsi.x              0000000007DBB2BB  Unknown               Unknown  Unknown
libpthread-2.31.s  00001486B586D8C0  Unknown               Unknown  Unknown    gsi.x              000000000091627F  general_sub2grid_         447  general_sub
2grid_mod.f90      
gsi.x              0000000000C6E593  gridmod_mp_init_g         680  gridmod.F90

I am trying to see what the problem is.
Could you also try this regression tests using debug mode GSI ?
Thanks
An update: the cultprit is an empty anavinfo file. I will see how it will be fixed.

[GangZhao-NOAA]

@shoyokota and @GangZhao-NOAA On wcoss2, when gsi was built with debug mode, for rrfs regression test , the first rrfs_3denvar_glbens_loproc_updat failed for

 nlat,nlon=nya,nxa=          117         199                   
forrtl: severe (71): integer divide by zero
Image              PC                Routine            Line        Source     
                   
gsi.x              0000000007DBB2BB  Unknown               Unknown  Unknown
libpthread-2.31.s  00001486B586D8C0  Unknown               Unknown  Unknown    gsi.x              000000000091627F  general_sub2grid_         447  general_sub
2grid_mod.f90      
gsi.x              0000000000C6E593  gridmod_mp_init_g         680  gridmod.F90

I am trying to see what the problem is. Could you also try this regression tests using debug mode GSI ? Thanks An update: the cultprit is an empty anavinfo file. I will see how it will be fixed.

@TingLei-NOAA @shoyokota Hi Ting and Sho,
I haven't run the ctest in debug mode before. As you mentioned, to run the regression test in debug mode, I need to re-compile the GSI code , I mean both control code and updated code, right?

Then after the compilations, to run the regression test, is there any specific option for running? For example, in the command line, "ctest -R rrfs_3denvar_glbens", do I need to add any option to it, like "-debug"?

Sho mentioned the configuration file "regression_namelist_db.txt", is this the file used for debug mode. Then how should I let the ctest to use it, instead of regular "regression_namelist.txt"?

Thank you!

-Gang

[shoyokota]

@GangZhao-NOAA In my understanding, all we have to do to run ctest in the debug-mode is to re-compile GSI with BUILD_TYPE="Debug" both for control and updated codes. In running ctests, we don't need to add any options, and regression_namelist_db.txt is automatically applied.

@TingLei-NOAA Is my understanding correct? I also will run the debug-mode ctests on Orion.

[TingLei-NOAA]

@GangZhao-NOAA Yes. As @shoyokota , just re-compile gsi using the debug mode and run the ctests. Regression tests with debug mode GSI are not required. But I feel that will be very helpful to verify the codes.

[TingLei-NOAA]

@GangZhao-NOAA I had finished debug mode GSI 's regression tests on wcoss2. I will give an update below. I think you don't have to do them again on other machines. But if you could run a rtma operational case with this PR, that will be great while I would run a rrfs operational case with this PR.
Debug mode regression tests results: rtma, rrfs_3denvar_glbens, netcdf_fv3_regional passed. global_4denvar and hafs regression tests passed for the known issues to be fixed by PR #722 and #679. Hence, it is concluded debug mode GSI regression tests on wcoss2 haven't revealed any issues with this PR.

[GangZhao-NOAA]

@GangZhao-NOAA I had finished debug mode GSI 's regression tests on wcoss2. I will give an update below. I think you don't have to do them again on other machines. But if you could run a rtma operational case with this PR, that will be great while I would run a rrfs operational case with this PR. Debug mode regression tests results: rtma, rrfs_3denvar_glbens, netcdf_fv3_regional passed. global_4denvar and hafs regression tests passed for the known issues to be fixed by PR #722 and #679. Hence, it is concluded debug mode GSI regression tests on wcoss2 haven't revealed any issues with this PR.

@TingLei-NOAA @shoyokota Hi, Ting and Sho,
Thank you for the instructions and suggestions on the debug mode run.
I ran the regression tests in debug mode on Hera. Here are the brief reports:

(base) [Gang.Zhao@hfe08:build] (feature/PR_NOAA-EMC_EnVar-MGBF)$ ctest -j 7
Test project /scratch1/NCEPDEV/da/Gang.Zhao/ProdGSI_dev/gsi_mgbf_syokota/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_glbens
    Start 4: netcdf_fv3_regional
    Start 5: hafs_4denvar_glbens
    Start 6: hafs_3denvar_hybens
    Start 7: global_enkf
1/7 Test #6: hafs_3denvar_hybens ..............***Failed   61.55 sec
2/7 Test #5: hafs_4denvar_glbens ..............***Failed  182.56 sec
3/7 Test #4: netcdf_fv3_regional ..............   Passed  856.88 sec
4/7 Test #3: rrfs_3denvar_glbens ..............   Passed  980.15 sec
5/7 Test #2: rtma .............................***Failed  6153.79 sec

netcdf_fv3_regional and rrfs_3denvar_glbens passed, rtma almost passed (just one test took too much time which is beyond the limit and count towards failure); two hafs tests crashed very quickly; two global tests failed due to running time beyond the wall clock limits.

As you mentioned to run a rtma operational case with this PR, the GSI code of the current (2D)RTMA is somehow fairly different to the current EMC/GSI code. We tested it one month ago, it could run but with an issue related to analysis of the obs of total cloud amount.

Thank you!

[TingLei-NOAA]

This PR passed a test using a modified rrfs operational case (while miter=2) in which it produced identical results compared with the head of EMC GSI.
Approval is pending upon the suggested changes

[TingLei-NOAA]

@GangZhao-NOAA I had finished debug mode GSI 's regression tests on wcoss2. I will give an update below. I think you don't have to do them again on other machines. But if you could run a rtma operational case with this PR, that will be great while I would run a rrfs operational case with this PR. Debug mode regression tests results: rtma, rrfs_3denvar_glbens, netcdf_fv3_regional passed. global_4denvar and hafs regression tests passed for the known issues to be fixed by PR #722 and #679. Hence, it is concluded debug mode GSI regression tests on wcoss2 haven't revealed any issues with this PR.

@TingLei-NOAA @shoyokota Hi, Ting and Sho, Thank you for the instructions and suggestions on the debug mode run. I ran the regression tests in debug mode on Hera. Here are the brief reports:

(base) [Gang.Zhao@hfe08:build] (feature/PR_NOAA-EMC_EnVar-MGBF)$ ctest -j 7
Test project /scratch1/NCEPDEV/da/Gang.Zhao/ProdGSI_dev/gsi_mgbf_syokota/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_glbens
    Start 4: netcdf_fv3_regional
    Start 5: hafs_4denvar_glbens
    Start 6: hafs_3denvar_hybens
    Start 7: global_enkf
1/7 Test #6: hafs_3denvar_hybens ..............***Failed   61.55 sec
2/7 Test #5: hafs_4denvar_glbens ..............***Failed  182.56 sec
3/7 Test #4: netcdf_fv3_regional ..............   Passed  856.88 sec
4/7 Test #3: rrfs_3denvar_glbens ..............   Passed  980.15 sec
5/7 Test #2: rtma .............................***Failed  6153.79 sec

netcdf_fv3_regional and rrfs_3denvar_glbens passed, rtma almost passed (just one test took too much time which is beyond the limit and count towards failure); two hafs tests crashed very quickly; two global tests failed due to running time beyond the wall clock limits.

As you mentioned to run a rtma operational case with this PR, the GSI code of the current (2D)RTMA is somehow fairly different to the current EMC/GSI code. We tested it one month ago, it could run but with an issue related to analysis of the obs of total cloud amount.

Thank you!

@GangZhao-NOAA Thanks. I think your tests a alongside with the test using rrfs-like real case areported re enough, at least for this reviewing , to verify this PR!

[shoyokota]

I just merged the latest GSI develop branch in the last push.

[TingLei-NOAA]

@shoyokota Thanks for clarification!

[shoyokota]

I confirmed all regression tests with the latest code were passed on Orion. (For global_4denvar and global_enkf, the 2022110900 case was tested.)

Test project /home/syokota/da/gsi/ctest/PR_NOAA-EMC_EnVar-MGBF_09/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_glbens
    Start 4: netcdf_fv3_regional
    Start 5: hafs_4denvar_glbens
    Start 6: hafs_3denvar_hybens
    Start 7: global_enkf
1/7 Test #4: netcdf_fv3_regional ..............   Passed  1923.88 sec
2/7 Test #3: rrfs_3denvar_glbens ..............   Passed  2167.45 sec
3/7 Test #7: global_enkf ......................   Passed  2351.31 sec
4/7 Test #2: rtma .............................   Passed  2829.07 sec
5/7 Test #6: hafs_3denvar_hybens ..............   Passed  2899.09 sec
6/7 Test #5: hafs_4denvar_glbens ..............   Passed  2962.67 sec
7/7 Test #1: global_4denvar ...................   Passed  3405.76 sec

100% tests passed, 0 tests failed out of 7

Total Test time (real) = 3405.77 sec

For rrfs_3denvar_glbens, I confirmed that the debug-mode run was also passed. (The debug-mode failures except for rrfs_3denvar_glbens should not be related to this PR.)

[GangZhao-NOAA]

@shoyokota @ShunLiu-NOAA @TingLei-NOAA
With Sho's latest updated code (commit #b1b5a962) vs the control code (EMC-GSI develop branch #f7e93abf), I ran the regression tests on WCOSS2 (Cactus) and Hera (Rocky-8), all the 7 tasks passed successfully. Here is the brief reports:
on WCOSS2/Cactus:
regression results are under directory "/lfs/h2/emc/da/noscrub/gang.zhao/noscrub/regression_b1b5a962_mgbf_vs_f7e93abf_wcoss2_cactus"
regression tests were running under directory "/lfs/h2/emc/ptmp/gang.zhao/gsi_mgbf_syokota"

[gang.zhao@clogin07:build] (feature/PR_NOAA-EMC_EnVar-MGBF)$ ctest -j 7
Test project /lfs/h2/emc/da/save/gang.zhao/WorkDir/ProdGSI_Dev/gsi_mgbf_syokota/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_glbens
    Start 4: netcdf_fv3_regional
    Start 5: hafs_4denvar_glbens
    Start 6: hafs_3denvar_hybens
    Start 7: global_enkf
1/7 Test #4: netcdf_fv3_regional ..............   Passed  482.90 sec
2/7 Test #3: rrfs_3denvar_glbens ..............   Passed  545.67 sec
3/7 Test #7: global_enkf ......................   Passed  856.54 sec
4/7 Test #2: rtma .............................   Passed  975.33 sec
5/7 Test #6: hafs_3denvar_hybens ..............   Passed  1154.16 sec
6/7 Test #5: hafs_4denvar_glbens ..............   Passed  1214.35 sec
7/7 Test #1: global_4denvar ...................   Passed  1682.69 sec

100% tests passed, 0 tests failed out of 7

Total Test time (real) = 1682.70 sec

On Hera (Rocky-8):
regression results are under directory "/scratch1/NCEPDEV/da/Gang.Zhao/noscrub/regression_b1b5a962_mgbf_vs_f7e93abf_hera"
regression tests were running under directory "/scratch1/NCEPDEV/stmp2/Gang.Zhao/gsi_mgbf_syokota"

(base) [Gang.Zhao@hfe08:build] (feature/PR_NOAA-EMC_EnVar-MGBF)$ ctest -j 7
Test project /scratch1/NCEPDEV/da/Gang.Zhao/ProdGSI_dev/gsi_mgbf_syokota/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_glbens
    Start 4: netcdf_fv3_regional
    Start 5: hafs_4denvar_glbens
    Start 6: hafs_3denvar_hybens
    Start 7: global_enkf
1/7 Test #4: netcdf_fv3_regional ..............   Passed  493.18 sec
2/7 Test #3: rrfs_3denvar_glbens ..............   Passed  561.20 sec
3/7 Test #2: rtma .............................   Passed  975.66 sec
4/7 Test #7: global_enkf ......................   Passed  1252.31 sec
5/7 Test #6: hafs_3denvar_hybens ..............   Passed  1290.15 sec
6/7 Test #5: hafs_4denvar_glbens ..............   Passed  1408.73 sec
7/7 Test #1: global_4denvar ...................   Passed  1979.08 sec

100% tests passed, 0 tests failed out of 7

Total Test time (real) = 1979.27 sec

[ShunLiu-NOAA]

@GangZhao-NOAA thank you for regression test!