Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG/ISSUE] Time interpolation error for leaf area index data, dependent on # of cores #132

Closed
yantosca opened this issue Mar 4, 2022 · 10 comments · Fixed by #134
Closed
Assignees
Labels
category: Bug Something isn't working topic: Input Data Related to input/emissions data, or disk read/write operations topic: Performance Related to HEMCO performance, parallelization, or memory issues topic: Regridding or Interpolation Related to issues with time interpolation or horiziontal/vertical regridding
Milestone

Comments

@yantosca
Copy link
Contributor

yantosca commented Mar 4, 2022

While debugging geoschem/geos-chem#1103, I stumbled upon an indexing error that is selecting the wrong time slice for leaf area index variable XLAI42. I added the following debug input to HEMCO/src/Core/hcoio_read_std_mod.F90.

    ! ----------------------------------------------------------------
    ! Read data
    ! ----------------------------------------------------------------

    ! Verbose mode
    IF ( HCO_IsVerb(HcoState%Config%Err,2) ) THEN
       WRITE(MSG,*) 'Reading variable ', TRIM(Lct%Dct%Dta%ncPara)
       CALL HCO_MSG(HcoState%Config%Err,MSG)
    ENDIF

    CALL NC_READ_ARR( fID     = ncLun,              &
                      ncVar   = Lct%Dct%Dta%ncPara, &
                      lon1    = 1,                  &
                      lon2    = nlon,               &
                      lat1    = 1,                  &
                      lat2    = nlat,               &
                      lev1    = lev1,               &
                      lev2    = lev2,               &
                      time1   = tidx1,              &
                      time2   = tidx2,              &
                      ncArr   = ncArr,              &
                      varUnit = thisUnit,           &
                      wgt1    = wgt1,               &
                      wgt2    = wgt2,               &
                      MissVal = HCO_MISSVAL,        &
                      ArbIdx  = ArbIdx,             &
                      RC      = NCRC                 )

    IF ( NCRC /= 0 ) THEN
       CALL HCO_ERROR( 'NC_READ_ARRAY', RC )
       RETURN
    ENDIF
    !### debug
    IF ( TRIM( Lct%Dct%Cname ) == 'XLAI42' ) THEN
       print*, '@@@ xlai42----------------------------------------------'
       print*, '@@@ xlai42 : after NC_READ_ARR in hcoio_read_std_mod.F90'
       print*, '@@@ xlai42 : ncPara    : ', TRIM(Lct%Dct%Dta%ncPara)
       print*, '@@@ xlai42 : nlon      : ', nlon
       print*, '@@@ xlai42 : nlat      : ', nlat
       print*, '@@@ xlai42 : lev1      : ', lev1
       print*, '@@@ xlai42 : lev2      : ', lev2
       print*, '@@@ xlai42 : tidx1     : ', tidx1
       print*, '@@@ xlai42 : tidx2     : ', tidx2
       print*, '@@@ xlai42 : sum ncarr : ', sum(ncarr)
       print*, '@@@ xlai42 : varunit   : ', trim(thisunit)
       print*, '@@@ xlai42 : wgt1      : ', wgt1
       print*, '@@@ xlai42 : wgt2      : ', wgt2
       print*, '@@@ xlai42 : arbidx    : ', arbidx
       stop
    endif

With the exact same executable, I get this output when I run GC-Classic on 1 core:

 @@@ xlai42----------------------------------------------
 @@@ xlai42 : after NC_READ_ARR in hcoio_read_std_mod.F90
 @@@ xlai42 : ncPara    : XLAI42
 @@@ xlai42 : nlon      :         1440
 @@@ xlai42 : nlat      :          720
 @@@ xlai42 : lev1      :            0
 @@@ xlai42 : lev2      :            0
 @@@ xlai42 : tidx1     :           20
 @@@ xlai42 : tidx2     :           21
 @@@ xlai42 : sum ncarr :    4403.71533    
 @@@ xlai42 : varunit   : cm2 cm-2
 @@@ xlai42 : wgt1      :    0.00000000    
 @@@ xlai42 : wgt2      :    1.00000000    
 @@@ xlai42 : arbidx    :           -1

and this output with 5 cores:

 @@@ xlai42----------------------------------------------
 @@@ xlai42 : after NC_READ_ARR in hcoio_read_std_mod.F90
 @@@ xlai42 : ncPara    : XLAI42
 @@@ xlai42 : nlon      :         1440
 @@@ xlai42 : nlat      :          720
 @@@ xlai42 : lev1      :            0
 @@@ xlai42 : lev2      :            0
 @@@ xlai42 : tidx1     :           23
 @@@ xlai42 : tidx2     :           24
 @@@ xlai42 : sum ncarr :    5284.36621    
 @@@ xlai42 : varunit   : cm2 cm-2
 @@@ xlai42 : wgt1      :   0.500000000    
 @@@ xlai42 : wgt2      :   0.500000000    
 @@@ xlai42 : arbidx    :           -1

and this output with 8 cores:

 @@@ xlai42----------------------------------------------
 @@@ xlai42 : after NC_READ_ARR in hcoio_read_std_mod.F90
 @@@ xlai42 : ncPara    : XLAI42
 @@@ xlai42 : nlon      :         1440
 @@@ xlai42 : nlat      :          720
 @@@ xlai42 : lev1      :            0
 @@@ xlai42 : lev2      :            0
 @@@ xlai42 : tidx1     :           20
 @@@ xlai42 : tidx2     :           21
 @@@ xlai42 : sum ncarr :    4403.71533    
 @@@ xlai42 : varunit   : cm2 cm-2
 @@@ xlai42 : wgt1      :    0.00000000    
 @@@ xlai42 : wgt2      :    1.00000000    
 @@@ xlai42 : arbidx    :           -1

As you can see, the time slice indices tidx1 and tidx2 (which are the upper & lower time bounds of the data) are the same with 1 and 8 cores, but with 5 cores, HEMCO is pulling a different time slice.

Wondering if it is a parallelization issue upstream. Any ideas: @christophkeller @jimmielin @msulprizio @lizziel

@yantosca yantosca added category: Bug Something isn't working topic: Input Data Related to input/emissions data, or disk read/write operations labels Mar 4, 2022
@yantosca yantosca self-assigned this Mar 4, 2022
@yantosca yantosca changed the title [BUG/ISSUE] Time interpolation error for leaf area index data, [BUG/ISSUE] Time interpolation error for leaf area index data, dependent on # of cores Mar 4, 2022
@yantosca
Copy link
Contributor Author

yantosca commented Mar 4, 2022

The issue might lie within routine Get_TimeIdx. Will continue looking there. Would appreciate any insight!

@yantosca
Copy link
Contributor Author

yantosca commented Mar 7, 2022

I put some more debug in Get_TimeIdx. It looks like for some reason the run with 5 cores is flagging the LAI data as not discontinuous but the run with 8 cores is:

5 cores;

 @@@ xlai42 : origYMDhm :    201907010000.00000     
 @@@ xlai42 : prefYMDhm :    201907010000.00000     
 @@@ xlai42 : tidx1a after availYMDhm:           23
 @@@ xlai42 : isClosest 1, prefymdhm  :    201907010000.00000     
 @@@ xlai42 : isClosest 1, availymdhm :    201906270000.00000     
 @@@ xlai42 : IsClosest 1, ntime      :           46
 @@@ xlai42 : IsClosest 1, tidx1a     :           23
 @@@ xlai42 : after IsClosest 1, exitSearch:  F
 @@@ xlai42 : after 1st or last timestep, :           23
 @@@ xlai42 : after test for cont search, tidx1a:           23
 @@@ xlai42 : after test for cont search, exit  :  T
 @@@ xlai42 : data discontinuous                :  F

8 cores:

 @@@ xlai42 : origYMDhm :    201907010000.00000     
 @@@ xlai42 : prefYMDhm :    201907010000.00000     
 @@@ xlai42 : tidx1a after availYMDhm:           23
 @@@ xlai42 : isClosest 1, prefymdhm  :    201907010000.00000     
 @@@ xlai42 : isClosest 1, availymdhm :    201906270000.00000     
 @@@ xlai42 : IsClosest 1, ntime      :           46
 @@@ xlai42 : IsClosest 1, tidx1a     :           23
 @@@ xlai42 : after IsClosest 1, exitSearch:  F
 @@@ xlai42 : after 1st or last timestep, :           23
 @@@ xlai42 : after test for cont search, tidx1a:           23
 @@@ xlai42 : after test for cont search, exit  :  F
 @@@ xlai42 : data discontinuous                :  T
---
 @@@ xlai42 : after adjust, ntime      :           46
 @@@ xlai42 : after adjust, availYmdhm :    201912280000.00000     
 @@@ xlai42 : after adjust, prefymdhm  :    201907010000.00000     
 @@@ xlai42 : after adjust, cnt        :            1
 @@@ xlai42 : after adjust, tidx1a     :           23
 @@@ xlai42 : after check_availymdhm    201907010000.00000                1          23
 @@@ xlai42 : after is_closest 2    201907010000.00000               46          23          -1
 @@@ xlai42 : after adjust, ntime      :           46
 @@@ xlai42 : after adjust, availYmdhm :    201912280000.00000     
 @@@ xlai42 : after adjust, prefymdhm  :    201906010000.00000     
 @@@ xlai42 : after adjust, cnt        :            2
 @@@ xlai42 : after adjust, tidx1a     :           23
 @@@ xlai42 : after check_availymdhm    201906010000.00000                2          19
 @@@ xlai42 : after is_closest 2    201906010000.00000               46          19          -1
 @@@ xlai42 : after adjust, ntime      :           46
 @@@ xlai42 : after adjust, availYmdhm :    201912280000.00000     
 @@@ xlai42 : after adjust, prefymdhm  :    201906070000.00000     
 @@@ xlai42 : after adjust, cnt        :            3
 @@@ xlai42 : after adjust, tidx1a     :           19
 @@@ xlai42 : after check_availymdhm    201906070000.00000                3          20
 @@@ xlai42 : after is_closest 2    201906070000.00000               46          20          20

It may be that the run on 5 cores is correct because it is pulling the date 20190627 as the closest date, where the run on 8 cores keeps iterating back in time until it pulls 20190607 as the closest date (which is wrong).

@yantosca
Copy link
Contributor Author

yantosca commented Mar 7, 2022

The XLAI is tagged with "I" (Interpolated), which should force dta%discontinuous = .FALSE., since this this data has a regular 8-day time stride.

* XLAI42 $ROOT/Yuan_XLAI/v2021-06/Yuan_proc_MODIS_XLAI.025x025.$YYYY.nc XLAI42 2000-2020/1-12/1-31/0 I xy cm2/cm2 * - 1 1 *
* ```

@yantosca
Copy link
Contributor Author

yantosca commented Mar 7, 2022

Found the issue. In HEMCO/src/Core/hco_config_mod.F90, we have TWO if blocks where time cycle flags are interpreted. The first is at about line 960:

! Set time cycling behaviour. Possible values are:
! - "C" : cycling <-- DEFAULT
! - "CS" : cycling, skip if not exist
! - "CY" : cycling, always use simulation year
! - "CYS" : cycling, always use simulation yr, skip if not exist
! - "R" : range
! - "RA" : range, average outside
! - "RF" : range, forced (error if not in range)
! - "RFY" : range, forced, always use simulation year
! - "RFY3 : range, forced, always use simulation year, 3-hourly
! - "RY" : range, always use simulation year
! - "E" : exact, read/query once
! - "EF" : exact, forced (error if not exist), read/query once
! - "EFY" : exact, forced, always use sim year
! - "EFYO": exact, forced, always use sim year, read once
! - "EC" : exact, read/query continuously (e.g. for ESMF interface)
! - "ECF" : exact, forced, read/query continuously
! - "EY" : exact, always use simulation year, read/query once
! - "A" : average
! - "I" : interpolate
! - "ID" : interpolate, discontinuous dataset
Dta%MustFind = .FALSE.
Dta%UseSimYear= .FALSE.
Dta%Discontinuous = .FALSE.
IF ( TRIM(TmCycle) == "C" ) THEN
Dta%CycleFlag = HCO_CFLAG_CYCLE
Dta%MustFind = .TRUE.
ELSEIF ( TRIM(TmCycle) == "CS" ) THEN
Dta%CycleFlag = HCO_CFLAG_CYCLE
Dta%MustFind = .FALSE.
ELSEIF ( TRIM(TmCycle) == "CY" ) THEN
Dta%CycleFlag = HCO_CFLAG_CYCLE
Dta%MustFind = .TRUE.
Dta%UseSimYear= .TRUE.
ELSEIF ( TRIM(TmCycle) == "CYS" ) THEN
Dta%CycleFlag = HCO_CFLAG_CYCLE
Dta%MustFind = .FALSE.
Dta%UseSimYear= .TRUE.
ELSEIF ( TRIM(TmCycle) == "R" ) THEN
Dta%CycleFlag = HCO_CFLAG_RANGE
ELSEIF ( TRIM(TmCycle) == "RA" ) THEN
Dta%CycleFlag = HCO_CFLAG_RANGEAVG
ELSEIF ( TRIM(TmCycle) == "RF" ) THEN
Dta%CycleFlag = HCO_CFLAG_RANGE
Dta%MustFind = .TRUE.
ELSEIF ( TRIM(TmCycle) == "RFY" ) THEN
Dta%CycleFlag = HCO_CFLAG_RANGE
Dta%MustFind = .TRUE.
Dta%UseSimYear= .TRUE.
ELSEIF ( TRIM(TmCycle) == "RFY3" ) THEN
Dta%CycleFlag = HCO_CFLAG_RANGE
Dta%MustFind = .TRUE.
Dta%UseSimYear= .TRUE.
Dta%UpdtFlag = HCO_UFLAG_3HR
ELSEIF ( TRIM(TmCycle) == "RY" ) THEN
Dta%CycleFlag = HCO_CFLAG_RANGE
Dta%UseSimYear= .TRUE.
ELSEIF ( TRIM(TmCycle) == "E" ) THEN
Dta%CycleFlag = HCO_CFLAG_EXACT
Dta%UpdtFlag = HCO_UFLAG_ONCE
ELSEIF ( TRIM(TmCycle) == "EF" ) THEN
Dta%CycleFlag = HCO_CFLAG_EXACT
Dta%UpdtFlag = HCO_UFLAG_ONCE
Dta%MustFind = .TRUE.
ELSEIF ( TRIM(TmCycle) == "EFY" ) THEN
Dta%CycleFlag = HCO_CFLAG_EXACT
Dta%MustFind = .TRUE.
Dta%UseSimYear= .TRUE.
ELSEIF ( TRIM(TmCycle) == "EFYO" ) THEN
Dta%CycleFlag = HCO_CFLAG_EXACT
Dta%UpdtFlag = HCO_UFLAG_ONCE
Dta%MustFind = .TRUE.
Dta%UseSimYear= .TRUE.
ELSEIF ( TRIM(TmCycle) == "EC" ) THEN
Dta%CycleFlag = HCO_CFLAG_EXACT
ELSEIF ( TRIM(TmCycle) == "ECF" ) THEN
Dta%CycleFlag = HCO_CFLAG_EXACT
Dta%MustFind = .TRUE.
ELSEIF ( TRIM(TmCycle) == "EY" ) THEN
Dta%CycleFlag = HCO_CFLAG_EXACT
Dta%UpdtFlag = HCO_UFLAG_ONCE
Dta%UseSimYear= .TRUE.
ELSEIF ( TRIM(TmCycle) == "A" ) THEN
Dta%CycleFlag = HCO_CFLAG_AVERG
ELSEIF ( TRIM(TmCycle) == "I" ) THEN
Dta%CycleFlag = HCO_CFLAG_INTER
ELSEIF ( TRIM(TmCycle) == "ID" ) THEN
Dta%CycleFlag = HCO_CFLAG_INTER
Dta%Discontinuous = .TRUE.
ELSEIF ( TRIM(TmCycle) == "-" ) THEN
Dta%CycleFlag = HCO_CFLAG_CYCLE
ELSE
MSG = 'Invalid time cycling attribute: ' // &
TRIM(TmCycle) // ' - in ' // TRIM(tagcName)
CALL HCO_ERROR ( MSG, RC, THISLOC=LOC )
RETURN
ENDIF

and the second is at:

! Set time cycling behaviour. Possible values are:
! - "C" : cycling <-- DEFAULT
! - "CS" : cycling, skip if not exist
! - "CY" : cycling, always use simulation year
! - "CYS" : cycling, always use simulation yr, skip if not exist
! - "R" : range
! - "RA" : range, average outside
! - "RF" : range, forced (error if not in range)
! - "RFY" : range, forced, always use simulation year
! - "RFY3": range, forced, always use simulation year, 3-hourly
! - "RY" : range, always use simulation year
! - "E" : exact, read/query once
! - "EF" : exact, forced (error if not exist), read/query once
! - "EFY" : exact, forced, always use sim year
! - "EFYO": exact, forced, always use sim year, read once
! - "EC" : exact, read/query continuousl (e.g. for ESMF interface)
! - "ECF" : exact, forced, read/query continuously
! - "EY" : exact, always use simulation year, read/query once
! - "A" : average
! - "I" : interpolate
! - "ID" : interpolate, discontinuous dataset
Dta%MustFind = .FALSE.
Dta%UseSimYear= .FALSE.
IF ( TRIM(TmCycle) == "C" ) THEN
Dta%CycleFlag = HCO_CFLAG_CYCLE
Dta%MustFind = .TRUE.
ELSEIF ( TRIM(TmCycle) == "CS" ) THEN
Dta%CycleFlag = HCO_CFLAG_CYCLE
Dta%MustFind = .FALSE.
ELSEIF ( TRIM(TmCycle) == "CY" ) THEN
Dta%CycleFlag = HCO_CFLAG_CYCLE
Dta%MustFind = .TRUE.
Dta%UseSimYear= .TRUE.
ELSEIF ( TRIM(TmCycle) == "CYS" ) THEN
Dta%CycleFlag = HCO_CFLAG_CYCLE
Dta%MustFind = .FALSE.
Dta%UseSimYear= .TRUE.
ELSEIF ( TRIM(TmCycle) == "R" ) THEN
Dta%CycleFlag = HCO_CFLAG_RANGE
ELSEIF ( TRIM(TmCycle) == "RA" ) THEN
Dta%CycleFlag = HCO_CFLAG_RANGEAVG
ELSEIF ( TRIM(TmCycle) == "RF" ) THEN
Dta%CycleFlag = HCO_CFLAG_RANGE
Dta%MustFind = .TRUE.
ELSEIF ( TRIM(TmCycle) == "RFY" ) THEN
Dta%CycleFlag = HCO_CFLAG_RANGE
Dta%MustFind = .TRUE.
Dta%UseSimYear= .TRUE.
ELSEIF ( TRIM(TmCycle) == "RFY3" ) THEN
Dta%CycleFlag = HCO_CFLAG_RANGE
Dta%MustFind = .TRUE.
Dta%UseSimYear= .TRUE.
Dta%UpdtFlag = HCO_UFLAG_3HR
ELSEIF ( TRIM(TmCycle) == "RY" ) THEN
Dta%CycleFlag = HCO_CFLAG_RANGE
Dta%UseSimYear= .TRUE.
ELSEIF ( TRIM(TmCycle) == "E" ) THEN
Dta%CycleFlag = HCO_CFLAG_EXACT
Dta%UpdtFlag = HCO_UFLAG_ONCE
ELSEIF ( TRIM(TmCycle) == "EF" ) THEN
Dta%CycleFlag = HCO_CFLAG_EXACT
Dta%UpdtFlag = HCO_UFLAG_ONCE
Dta%MustFind = .TRUE.
ELSEIF ( TRIM(TmCycle) == "EFY" ) THEN
Dta%CycleFlag = HCO_CFLAG_EXACT
Dta%MustFind = .TRUE.
Dta%UseSimYear= .TRUE.
ELSEIF ( TRIM(TmCycle) == "EFYO" ) THEN
Dta%CycleFlag = HCO_CFLAG_EXACT
Dta%UpdtFlag = HCO_UFLAG_ONCE
Dta%MustFind = .TRUE.
Dta%UseSimYear= .TRUE.
ELSEIF ( TRIM(TmCycle) == "EC" ) THEN
Dta%CycleFlag = HCO_CFLAG_EXACT
ELSEIF ( TRIM(TmCycle) == "ECF" ) THEN
Dta%CycleFlag = HCO_CFLAG_EXACT
Dta%MustFind = .TRUE.
ELSEIF ( TRIM(TmCycle) == "EY" ) THEN
Dta%CycleFlag = HCO_CFLAG_EXACT
Dta%UpdtFlag = HCO_UFLAG_ONCE
Dta%UseSimYear= .TRUE.
ELSEIF ( TRIM(TmCycle) == "A" ) THEN
Dta%CycleFlag = HCO_CFLAG_AVERG
ELSEIF ( TRIM(TmCycle) == "I" ) THEN
Dta%CycleFlag = HCO_CFLAG_INTER
ELSEIF ( TRIM(TmCycle) == "ID" ) THEN
Dta%CycleFlag = HCO_CFLAG_INTER
Dta%Discontinuous = .TRUE.
ELSEIF ( TRIM(TmCycle) == "-" ) THEN
Dta%CycleFlag = HCO_CFLAG_CYCLE
ELSE
MSG = 'Invalid time cycling attribute: ' // &
TRIM(TmCycle) // ' - in ' // TRIM(tagcName)
CALL HCO_ERROR ( MSG, RC, THISLOC=LOC )
RETURN
ENDIF

Note that at the top of the IF block starting at line 960 we initialize some fields of the data container object:

                Dta%MustFind  = .FALSE.
                Dta%UseSimYear= .FALSE.
                Dta%Discontinuous = .FALSE.

but at the top of the IF block starting at line 1298, we have:

             Dta%MustFind  = .FALSE.
             Dta%UseSimYear= .FALSE.

so for the 2nd IF block, Dta%Discontinuous is not set to a default value. Because of this, there is the possibility that Dta%Discontinuous can get unexpectedly set to TRUE due to it being populated with junk values. This seems precisely to have been causing the differences in the LAI fields.

By adding

            Dta%Discontinuous = .FALSE.

atop the IF block starting at around line 1298, the issue is now fixed. I will prepare a corresponding PR.

yantosca added a commit that referenced this issue Mar 7, 2022
This commit fixes the issue reported in geoschem/HEMCO #132.  We had
observed a difference in output in runs using different numbers of
computational cores.  We had thought this was a parallelization issue
but in fact, the interpolation of leaf area indices was being done
incorrectly.

The root cause of the issue was that the Dta%Discontinuous field was
being used before it was initialized.  In some instances, this was
inadvertently assigned a TRUE value when it should have been FALSE.
Adding a Dta%Discontinuous = .FALSE. statement above the IF block
where it is used fixes the problem.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
@lizziel
Copy link
Contributor

lizziel commented Mar 7, 2022

Nice find! That code looks identical in both, which is pretty dangerous, as this bug shows. Could it be abstracted somehow so we don't need to duplicate?

@jimmielin
Copy link
Collaborator

Thanks Bob, this is incredible detective work. I was also wondering whether this code repetition can somehow be avoided, especially in an often-updated place like the interpretation of time cycling flags because new ones may be added pretty frequently. This can probably be abstracted into a subroutine or into a separate file that is #included?

@yantosca
Copy link
Contributor Author

yantosca commented Mar 7, 2022

I can try to abstract that into a subroutine. I've also added some extra initialization of variables in other routines so I'll create a PR with the updated HEMCO. Will test against the prior version, we should just get numerical noise diffs if anything.

@yantosca
Copy link
Contributor Author

yantosca commented Mar 9, 2022

Also see HEMCO issue #133

@yantosca yantosca added the topic: Performance Related to HEMCO performance, parallelization, or memory issues label Mar 9, 2022
@yantosca yantosca linked a pull request Mar 22, 2022 that will close this issue
@yantosca
Copy link
Contributor Author

yantosca commented Mar 22, 2022

We can close this issue because PR #134, which resolves this issue, has been merged into the HEMCO 3.4.0 / GEOS-Chem 13.4.0 development stream.

@yantosca
Copy link
Contributor Author

Remaining parallelization issues in GEOS-Chem Classic simulations have now been solved in PR geoschem/geos-chem#1190.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: Bug Something isn't working topic: Input Data Related to input/emissions data, or disk read/write operations topic: Performance Related to HEMCO performance, parallelization, or memory issues topic: Regridding or Interpolation Related to issues with time interpolation or horiziontal/vertical regridding
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants