Parallelization of GSI IO for fv3-lam

[TingLei-NOAA]

The parallelization of GSI IO for fv3-lam (including reading of fv3-lam ensemble) has been considered at EMC and outside collaborating institutes. Especially, recent fv3-lam forecast model prefer to directly dump data on subdomains in selected layout to speed up the model IO. This needs corresponding treatment on GSI IO part.
This IO issue is complicated because it is in association with the needed conversion between the native model grids and GSI analysis regular grids, which is done in the full domain. To implement a more complicated conversion scheme, which can be done on subdomain, need the efforts, seemingly, beyond current consideration.
However, as GSI deal with the horizontal recursive filter, the parallelization along the vertical levels could cause significant speeding up.
For example, recently @guoqing-noaa realized the parallelization of analysis writing by parallelization in the vertical levels with coding "from scratch" , NOAA-GSL#4. His results show obvious improvement in the writing step.
This issue is demonstrate our efforts to make use of GSI existing interface (based on general_sub2grid_mod.f90) for the parallelization of the horizontal recursive filtering as the basic parallelization frame for FV-LAM IO to incorporate use of the "parallel netcdf" and use of parallel IO subroutine in fms (used by FV3 model IO). We expect such developed coding would provide well wrapped and easier expandable interface for GSI parallelized IO for FV3-LAM.
The design is maybe changed according to discussions/tests among developers.
This issue will demonstrate discussions/comments/suggestions and test results during this developing process.

[TingLei-NOAA]

An update:
An initial version of parallel IO for fv3-lam is implemented ( feature/gsi_fv3lam_parallel_io at https://github.com/TingLei-daprediction/GSI.git) . The attached is an email to Guoqin @guoqing-noaa and Minghu @hu5970 to seek for their comments/suggestions on the current version and request for their help on the verification using the sw-ne orientation grids.
^^^^^^^
I had finished some verifications of the implemented coding with the "old " gsi grid orientation, namely, NE-SW. The next option I need to verify is for l_use_dbz_directDA=.true.
The options which had been verified include fv3sar_bg_opt=0 or 1 , and l_reg_update_hydro_delz and use of the fv3-lam ensemble.
It is verified when the new results are identical to the "old" gsi compared using the standard output and the updated fv3_dynvar and fv3_tracer.
The verifications were done using 13km conus domain.
For one regular GSI using 20 member, the old wallclock time is 150s and the new one 120 seconds.
One tricky part is how to keep this part is flexible to different choices of the met variables used in the GSI while reduce its' vulnerability for errors.
I hope the current interface is easy to understand , at least, in the expert eyes like yours.
So, your comments and suggestions are welcome. and , at the same, that will be great if you can verify it using the sw-ne grids .
VVVVVVV

[TingLei-NOAA]

A rather complete verification and polishing (removing unused variables and duplicated variables for the same parameters) of the parallel IO for fv3 lam have been finished. In an 3km conus domain cold start test ,the parallel IO for fv3-cam demonstrates significant speeding up and much less maximum resident memory . For details, pls see the attachment.
A PR is in preparation .
emc-work-gsi-parallel IO performance.pptx

[TingLei-NOAA]

The tests includes the use of control variables with hydrometeors when l_use_direct_dbz is true, though no dbz observation is used. Thanks to J. Park @junpark217 OU providing the namelist block and anavinfo file for this test.