Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

20220414 BoCui sync and merge UPP/develop into post_2d_decomp #7

Merged
merged 23 commits into from
Apr 22, 2022

Conversation

BoCui-NOAA
Copy link
Collaborator

No description provided.

WenMeng-NOAA and others added 21 commits October 26, 2019 21:50
* Doxygen for CALDRG.f

* This is part of Issue NOAA-EMC#392
Fixes the doxygen warnings in CALGUST.f

Please review this subroutine. Thank you.
* Modify CALUPDHEL.f for restricting undefined grids in computation.

* Update VERSION to 10.0.12.
Co-authored-by: Tracy <tracy.hertneky@noaa.gov>
* Further refinement to the tables and logs.

* Further refinement to the tables and logs. (NOAA-EMC#455)
Fixes the doxygen warnings in CALHEL.f CALHEL2.f CALHEL3.f

Please review this subroutine. Thank you.
* Unify the interfaces for reading FV3 outputs in netcdf.

* Remove interface INITPOST_GFS_NETCDF_PARA.

* Remove INITPOST_GFS_NETCDF.f.

* Remove the capability of serial netcdf reading FV3 outputs.

* Correct reading rswinc.

* Remove duplication in CLDRAD.f.

* Remove duplicated avgalbedo reading

* Add changes for reading pwat from model.

* Clean up commented out code

* Clean up duplicated lines
…e modulefiles for gnu and intel (NOAA-EMC#468)

* Add cheyenne modulefile

* Intel 19 --> 2021

* Add ability to specify compiler; move all existing modulefiles to ${name}_intel since they are all for intel compilers; add "cheyenne_gnu" modulefile

* Revert move of intel modulefiles; now the implicit default for a modulefile is intel, only gnu and other compilers will have the compiler name appended to the modulefile

* Allow for lua modulefiles with ".lua" appended
Fixes the doxygen warnings in CALLCL.f CALMCVG.f CALMICT.f
Fixes the doxygen warnings in CALPBL.f CALPBLREGIME.f CALPOT.f CALPW.f.
…E.f CALUPDHEL.f. (NOAA-EMC#467)

* This is part of Issue NOAA-EMC#392.
Fixes the doxygen warnings in CALRAD_WCLOUD_newcrtm.f CALRCH.f
CALSTRM.f CALTAU.f CALTHTE.f CALUPDHEL.f.

* Fixed typo.

* Update to PR NOAA-EMC#458.
….f, DEALLOCATE.f, and DEALLOCATE.f (NOAA-EMC#469)

* This is part of Issue NOAA-EMC#392.

Fixes the doxygen warnings in CALVOR.f, CALWXT_BOURG.f, CLDRAD.f, COLLECT.f, COLLECT_LOC.f, DEALLOCATE.f, and DEALLOCATE.f.

* Fixed minor bug  in CALVOR.f.

* Update to PR#453.

* More updates.
Merge remote-tracking branch 'upstream/develop' into develop
@BoCui-NOAA
Copy link
Collaborator Author

@WenMeng-NOAA @JesseMeng-NOAA
1.
The RT tests use rt_for_developer.sh from /u/Wen.Meng/noscrubd/ncep_post/post_regression_test_new/scripts
To test numx for fv3r, new job scripts with the "numx" option are saved at:

/gpfs/dell2/emc/retros/noscrub/Bo.Cui/post_regression_test_new/jobs-dev/run_post_fv3r_WCOSS_DELL_P3.sh
/gpfs/dell2/emc/retros/noscrub/Bo.Cui/post_regression_test_new/jobs-dev/run_post_fv3r_pe_test_WCOSS_DELL_P3.sh

2..
The RT tests worked fine for numx=1 but failed for the fv3r when choosing numx=2. The differences come from variable ABSV Absolute Vorticity.

  1. File CALVOR.f. comes from UPP/develop and useless for this version because file UPP_PHYSICS.f in branch post_2d_decomp has all subroutines in CALVOR.f.

Jesse, would you please take a look at below files and see if there is any extra discrepancy in addition to the 2d decomposition part? Thanks!

https://github.com/NOAA-EMC/UPP/blob/develop/sorc/ncep_post.fd/CALVOR.f
https://github.com/BoCui-NOAA/EMC_post/blob/upp_2d/sorc/ncep_post.fd/UPP_PHYSICS.f

@JesseMeng-NOAA
Copy link
Collaborator

@WenMeng-NOAA @BoCui-NOAA
Wen, my understanding is currently we are unifying INIPOST for gfs and fv3r to support 2d_decomp.
We have numx check in WRFPOST.f as
IF(TRIM(IOFORM) /= 'netcdfpara' .AND. TRIM(IOFORM) /= 'netcdf' ) THEN
numx=1
if(me == 0) print*,'2D decomposition only supports netcdfpara IO.'
if(me == 0) print*,'Reset numx= ',numx
ENDIF

Do we want to support 2d_decomp for hrrr, rap, rtma? Those are also netcdf.
If not, I suggest add the following after the previous IF block,
IF(MODELNAME /= 'FV3R' .AND. MODELNAME /= 'GFS') THEN
numx=1
if(me == 0) print*,'2D decomposition only supports GFS and FV3R.'
if(me == 0) print*,'Reset numx= ',numx
ENDIF

@JesseMeng-NOAA 1. The RT tests use rt_for_developer.sh from /u/Wen.Meng/noscrubd/ncep_post/post_regression_test_new/scripts To test numx for fv3r, new job scripts with the "numx" option are saved at:

/gpfs/dell2/emc/retros/noscrub/Bo.Cui/post_regression_test_new/jobs-dev/run_post_fv3r_WCOSS_DELL_P3.sh /gpfs/dell2/emc/retros/noscrub/Bo.Cui/post_regression_test_new/jobs-dev/run_post_fv3r_pe_test_WCOSS_DELL_P3.sh

2.. The RT tests worked fine for numx=1 but failed for the fv3r when choosing numx=2. The differences come from variable ABSV Absolute Vorticity.

  1. File CALVOR.f. comes from UPP/develop and useless for this version because file UPP_PHYSICS.f in branch post_2d_decomp has all subroutines in CALVOR.f.

Jesse, would you please take a look at below files and see if there is any extra discrepancy in addition to the 2d decomposition part? Thanks!

https://github.com/NOAA-EMC/UPP/blob/develop/sorc/ncep_post.fd/CALVOR.f https://github.com/BoCui-NOAA/EMC_post/blob/upp_2d/sorc/ncep_post.fd/UPP_PHYSICS.f

@JesseMeng-NOAA
Copy link
Collaborator

@BoCui-NOAA
UPP_PHYSICS.f has all the function in CALVOR.f, but CALVOR.f has the updated doxygen doc blocks. Please update those in UPP_PHYSICS.f before deleting CALVOR.f

  1. File CALVOR.f. comes from UPP/develop and useless for this version because file UPP_PHYSICS.f in branch post_2d_decomp has all subroutines in CALVOR.f.

Jesse, would you please take a look at below files and see if there is any extra discrepancy in addition to the 2d decomposition part? Thanks!

https://github.com/NOAA-EMC/UPP/blob/develop/sorc/ncep_post.fd/CALVOR.f https://github.com/BoCui-NOAA/EMC_post/blob/upp_2d/sorc/ncep_post.fd/UPP_PHYSICS.f

@WenMeng-NOAA
Copy link
Owner

Do we want to support 2d_decomp for hrrr, rap, rtma? Those are also netcdf.
If not, I suggest add the following after the previous IF block,
IF(MODELNAME /= 'FV3R' .AND. MODELNAME /= 'GFS') THEN
numx=1
if(me == 0) print*,'2D decomposition only supports GFS and FV3R.'
if(me == 0) print*,'Reset numx= ',numx
ENDIF

I agree with @JesseMeng-NOAA. The plan is 2d decomposition feature only supports FV3 based model outputs.

@BoCui-NOAA
Copy link
Collaborator Author

Do we want to support 2d_decomp for hrrr, rap, rtma? Those are also netcdf.
If not, I suggest add the following after the previous IF block,
IF(MODELNAME /= 'FV3R' .AND. MODELNAME /= 'GFS') THEN
numx=1
if(me == 0) print*,'2D decomposition only supports GFS and FV3R.'
if(me == 0) print*,'Reset numx= ',numx
ENDIF

I agree with @JesseMeng-NOAA. The plan is 2d decomposition feature only supports FV3 based model outputs.

@JesseMeng-NOAA @WenMeng-NOAA How about fv3gefs test? It also has "modelname=GFS". Its current RT running still choose the default numx=1 in WRFPOST.f because the "numx" input hasn't been added in the NAMPGB.

@WenMeng-NOAA
Copy link
Owner

@BoCui-NOAA The 2d decomposition testing will not used in fv3gefs test. It would be fine as long as no numx input in itag for fv3gefs.

@WenMeng-NOAA
Copy link
Owner

@BoCui-NOAA The nemsio interface will be removed after gfsv16/gefsv13 parallel starts.

@BoCui-NOAA
Copy link
Collaborator Author

@BoCui-NOAA The nemsio interface will be removed after gfsv16/gefsv13 parallel starts.

@WenMeng-NOAA Thanks for the clarification.

@BoCui-NOAA
Copy link
Collaborator Author

@JesseMeng-NOAA @WenMeng-NOAA Please check today's update: Doxygen in UPP_PHYSICS.f, and add restriction to run 2D decomposition only for GFS and FV3R. Thanks!

@JesseMeng-NOAA
Copy link
Collaborator

Looks good.

@JesseMeng-NOAA @WenMeng-NOAA Please check today's update: Doxygen in UPP_PHYSICS.f, and add restriction to run 2D decomposition only for GFS and FV3R. Thanks!

Copy link
Owner

@WenMeng-NOAA WenMeng-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look good to me.

@WenMeng-NOAA
Copy link
Owner

@JesseMeng-NOAA Do you have more comments on this PR? Thanks!

@JesseMeng-NOAA
Copy link
Collaborator

Do you have more comments on this PR? Thanks!
Looks good. No more comments.

@WenMeng-NOAA
Copy link
Owner

@BoCui-NOAA and @JesseMeng-NOAA Thanks for working on this PR. I will merge it soon.

@WenMeng-NOAA WenMeng-NOAA merged commit 5186eb2 into WenMeng-NOAA:post_2d_decomp Apr 22, 2022
@BoCui-NOAA
Copy link
Collaborator Author

Thanks Wen and Jesse. @WenMeng-NOAA @JesseMeng-NOAA

WenMeng-NOAA added a commit that referenced this pull request Jun 10, 2022
* Changes to support 2D Decomposition shape changes.  Full I dimension remains in all arrays bu
t SURFCE is altered to use isx:iex bounds which are equivalent to IM which is equivalent to 1:IM

* test of commit only  added one comment to PROCESS.f

* comment test 2

* Added more routines to 2D decomposition

* Changed isx and iex to ISTA and IEND in listed routines.   ISTA had prior use in MDL2P.f and that needed changing.
IEND had prior use in PARA_CONFIG, changed to IENDJ in PARA_CONFIG2 where needed PARA_CONFIG did not
need changing

* test commit of new base into clone

* repair of a bunch of changed routines somehow corrupted by a merge 5/3/2021

* fixed 5/25 problem

* added ctlblk change to support ista and iend partial I dimensons

* added para_range2 to PARA_RANGE.f to support 2D decomposition

* Modified MPI_FIRST.f to support a 2D decomposition but the actual numbers used
remain the 1D special case and the changes just add two more indices for start and end I domains
and logic to convert scatter counts and displacements to the product of the I and J subdomain sizes rather than I full domain x J subdomain.

* Added support for halo settings of 2D boundaries i.e ista_2l to match the 1D analogs long present in the source
6/1/2021

* Added definitions for ista_2u, and ista_2u in MPI_FIRST.f

* Reshaped arrays  in MDL2FLD.f to support 2D decomposition.  Modified TWO (only) loops to try thie

* Remove TIMEF.f.

* 20210702 JesseMeng modify MPI_FIRST.f MDLFLD.f for 2D decomposition

* 20210706 BoCui modify INITPOST_GFS_NETCDF_PARA.f

* 20210707 JesseMeng modified grib2_module.f for 2d decomposition

* test version INITPOST_GFS_NETCDF_PARA.f

* test version INITPOST_GFS_NETCDF_PARA.f and MPI_FIRST.f

* 20210713 BoCui test INITPOST_GFS_NETCDF_PARA.f, MPI_FIRST.f and ALLOCATE_ALL.f

* 20210713 BoCui test INITPOST_GFS_NETCDF_PARA.f, MPI_FIRST.f and ALLOCATE_ALL.f

* 20210719 BoCui Modified CLDRAD.f for 2d decomposition

* 20210816 Jesse Meng commit George's EXCH update for 1 layer 2D halos transforming

* 20210816 Jesse Meng remove ifcore in EXCH

* 20210903 Bo Cui update ALLOCATE_ALL.f after new merge from 'upstream/develop'

* 20210903 Bo Cui Added new routines to 2D decomposition

* 20210903 Jesse Meng fixed the ieql allocation bug in ALLOCATE_ALL.f

* 20210904 Bo Cui fixed CLDRAD.f

* 20210913 Jesse Meng commit progress of 2D decomposition

* 20210917 Bo Cui add new routines to 2D decomposition

* 20210917 Jesse Meng remove 4 legacy files

* 20210917 Jesse Meng remove legacy code SLP_NMM EXCH2

* 20210917 Jesse Meng remove legacy code INITPOST_GFS_SIGIO.f INITPOST_GFS_NEMS.f

* 20210917 Jesse Meng add INITPOST_GFS_NEMS.f back to avoid compiling error.

* 20210930 Jesse Meng add George's itag/numx entry and progress on 2D decompose

* 20211015 Jesse Meng progress on 2D DECOMPOSITION

* 20211017 Bo Cui add new subroutines to UPP 2D decomposition

* 20211026 Jesse Meng commit progress in 2D DECOMPOSITION

* 20211026 Jesse Meng fix INITPOST_GFS_NETCDF_PARA.f for 2D DECOMPOSITION

* 20211026 Jesse Meng update SURFCE.f for 2D DECOMPOSITION

* 20211103 Jesse Meng commit progress of 2D DECOMPOSITION

* 20211106 Bo Cui commit progress of 2D decomposition

* 20211109 Jesse Meng updates for 2D DECOMPOSITION

* 20211110 Jesse Meng updates in 2D DECOMPOSITION

* 20211112 Jesse Meng updates for 2D DECOMPOSITION

* 20211115 Jesse Meng updates for 2D DECOMPOSITION

* 20211119 Jesse Meng updates for 2D DECOMPOSITION

* 20211201 Jesse Meng move CALVOR to UPP_PHYSICS module; implement fullpole in MPI_FIRST

* 20211202 Jesse Meng minor update of MPI_FIRST using mpi_allgatherv

* 20211203 Jesse Meng implement fullpole in MDL2THANDPV and CALDIV

* 20211207 Jesse Meng update MISCLN.f SURFCE.f

* 20211208 Jesse Meng add parm/hafs_nosat files

* 20211213 Bo Cui updates for 2D decomposition

* 20211213 Jesse Meng update the merged 'upstream/develop' and 'post_2d_decomp' branch to 2d_decomp style

* 20211215 Jesse Meng use i=0:im+1 for GFS in MPI_FIRST and EXCH

* 20211216 Jesse Meng minor update UPP_PHYSICS for i=0:im+1 expansion

* 20220104 Jesse Meng add George's minor fix in EXHC.f

* 20220118 Jesse Meng commit George's cleaned up code.

* 20220203 Jesse Meng - update MAPSSLP.f to be consistent with develop branch.

* 20220222 Jesse Meng add checkcoords flag in EXCH.f to move around George's debug code.

* 20220302 Jesse Meng Restrict computation from undefined grids

* 20220304 Bo Cui Add a reset of numx=1 if remainder of num_procs/numx is not 0

* 20220310 Jesse Meng Add a reset of numx=1 if(numx>num_procs/2)

* 20220322 Jesse Meng mpi_allgatherv change communicator to MPI_COMM_COMP

* 20220328 Jesse Meng bug fix for passing 2d subarrays between subroutines

* 20220328 Jesse Meng minor fix for CLDRAD call BOUND variable array size

* 20220330 Jesse Meng fix cloud cover variabes full field collection in 2d decomp

* 20220401 Jesse Meng minor fix for CALMCVG.f and update CALUPDHEL.f with develop

* 20220414 BoCui  sync and merge UPP/develop into post_2d_decomp (#7)

* Added lambert conformal projection for FV3SAR and handled composite reflectivity correctly (NOAA-EMC#14) (#2)

* exec is selectable for executable directory (NOAA-EMC#448)

* Doxygen for CALGUST (NOAA-EMC#451)

* Doxygen for CALDRG.f

* This is part of Issue NOAA-EMC#392
Fixes the doxygen warnings in CALGUST.f

Please review this subroutine. Thank you.

* The fix in  CALUPDHEL.f (NOAA-EMC#458)

* Modify CALUPDHEL.f for restricting undefined grids in computation.

* Update VERSION to 10.0.12.

* Bug fix for SLLEVEL bound issue when not RUC LSM (NOAA-EMC#463)

Co-authored-by: Tracy <tracy.hertneky@noaa.gov>

* Doxygen caldrg caldwcp calgust refinement (NOAA-EMC#464)

* Further refinement to the tables and logs.

* Further refinement to the tables and logs. (NOAA-EMC#455)

* This is part of Issue NOAA-EMC#392 (NOAA-EMC#460)

Fixes the doxygen warnings in CALHEL.f CALHEL2.f CALHEL3.f

Please review this subroutine. Thank you.

* Unify global and regional FV3 read interfaces (NOAA-EMC#453)

* Unify the interfaces for reading FV3 outputs in netcdf.

* Remove interface INITPOST_GFS_NETCDF_PARA.

* Remove INITPOST_GFS_NETCDF.f.

* Remove the capability of serial netcdf reading FV3 outputs.

* Correct reading rswinc.

* Remove duplication in CLDRAD.f.

* Remove duplicated avgalbedo reading

* Add changes for reading pwat from model.

* Clean up commented out code

* Clean up duplicated lines

* Add ability to compile script to use non-intel compilers; add Cheyenne modulefiles for gnu and intel (NOAA-EMC#468)

* Add cheyenne modulefile

* Intel 19 --> 2021

* Add ability to specify compiler; move all existing modulefiles to ${name}_intel since they are all for intel compilers; add "cheyenne_gnu" modulefile

* Revert move of intel modulefiles; now the implicit default for a modulefile is intel, only gnu and other compilers will have the compiler name appended to the modulefile

* Allow for lua modulefiles with ".lua" appended

* This is part of Issue NOAA-EMC#392 (NOAA-EMC#465)

Fixes the doxygen warnings in CALLCL.f CALMCVG.f CALMICT.f

* This is part of Issue NOAA-EMC#392 (NOAA-EMC#466)

Fixes the doxygen warnings in CALPBL.f CALPBLREGIME.f CALPOT.f CALPW.f.

* Doxygen in CALRAD_WCLOUD_newcrtm.f CALRCH.f CALSTRM.f CALTAU.f CALTHTE.f CALUPDHEL.f. (NOAA-EMC#467)

* This is part of Issue NOAA-EMC#392.
Fixes the doxygen warnings in CALRAD_WCLOUD_newcrtm.f CALRCH.f
CALSTRM.f CALTAU.f CALTHTE.f CALUPDHEL.f.

* Fixed typo.

* Update to PR NOAA-EMC#458.

* Doxygen in CALVOR.f, CALWXT_BOURG.f, CLDRAD.f, COLLECT.f, COLLECT_LOC.f, DEALLOCATE.f, and DEALLOCATE.f (NOAA-EMC#469)

* This is part of Issue NOAA-EMC#392.

Fixes the doxygen warnings in CALVOR.f, CALWXT_BOURG.f, CLDRAD.f, COLLECT.f, COLLECT_LOC.f, DEALLOCATE.f, and DEALLOCATE.f.

* Fixed minor bug  in CALVOR.f.

* Update to PR#453.

* More updates.

* 20220411 Bo Cui 2D decompositio CALVOR.f and INITPOST_NETCDF.f

* 20220411 Bo Cui update doxgen and global and regional FV3 read interface

* 20220414 Bo Cui: remove read_netcdf_2d_scatter and read_netcdf_3d_scatter from INITPOST_NETCDF.f

* 20220415 Bo Cui delete CALVOR.f, add exch of gdlon in INITPOST_NETCDF.f

* 20220421 Bo Cui Doxygen in UPP_PHYSICS.f,add restriction run 2D decomp only for GFS/FV3R in WRFPOST.f

Co-authored-by: WenMeng-NOAA <48260754+WenMeng-NOAA@users.noreply.github.com>
Co-authored-by: Wen Meng <wen.meng@noaa.gov>
Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com>
Co-authored-by: kayeekayee <kayee.wong@noaa.gov>
Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com>
Co-authored-by: Tracy <tracy.hertneky@noaa.gov>
Co-authored-by: Michael Kavulich <kavulich@ucar.edu>

* 20220502 Bo Cui code cleanup

* 20220512 Jesse Meng minor fix for INITPOST_GFS_NEMS_MPIIO.f calling EXCH

* 20220525 Jesse Meng minor fix to 2d_decomp syntax

* Update VERSION to 11.0.0

* 20220607 Jesse Meng add variable declaration block in PARA_RANGE.f

* 20220608 Jesse Meng remove blank spaces in sorc/ncep_post.fd/AllGETHERV_GSD.f

* Add 2D decomp overview documentation (#6)

* Add 2D decomp overview documentation

* Update 2D overview

Co-authored-by: George Vandenberghe <George.Vandenberghe@noaa.gov>
Co-authored-by: wx15gv <George.Vandenberghe@v71a1.ncep.noaa.gov>
Co-authored-by: wx22mj <Jesse.Meng@m71a1.ncep.noaa.gov>
Co-authored-by: Bo Cui <Bo.Cui@noaa.gov>
Co-authored-by: wx22mj <Jesse.Meng@m71a2.ncep.noaa.gov>
Co-authored-by: wx22mj <Jesse.Meng@m72a1.ncep.noaa.gov>
Co-authored-by: wx22mj <Jesse.Meng@m71a3.ncep.noaa.gov>
Co-authored-by: wx22mj <Jesse.Meng@v71a1.ncep.noaa.gov>
Co-authored-by: wx22mj <Jesse.Meng@v72a1.ncep.noaa.gov>
Co-authored-by: wx22mj <Jesse.Meng@v71a3.ncep.noaa.gov>
Co-authored-by: wx22mj <Jesse.Meng@v71a2.ncep.noaa.gov>
Co-authored-by: Jesse.Meng <Jesse.Meng@noaa.gov>
Co-authored-by: BoCui-NOAA <53531984+BoCui-NOAA@users.noreply.github.com>
Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com>
Co-authored-by: kayeekayee <kayee.wong@noaa.gov>
Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com>
Co-authored-by: Tracy <tracy.hertneky@noaa.gov>
Co-authored-by: Michael Kavulich <kavulich@ucar.edu>
Co-authored-by: Jesse Meng <jmeng@Orion-login-4.HPC.MsState.Edu>
Co-authored-by: Kate Fossell <fossell@ucar.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
7 participants