Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trajectory sampler to ingest multiple IODA files via one route handle #2353

Merged
merged 4 commits into from
Sep 19, 2023

Conversation

metdyn
Copy link
Contributor

@metdyn metdyn commented Sep 15, 2023

Description

The trajectory sampler code originally written by @bena-nasa to regrid to flight trajectory is extended to ingest and regrid to multiple IODA file locations. The implementation mainly have followed #2144. The listed items therein, except for the item-8: related to 3D height interpolation, have been finished in this PR.

The important features include:

  • Epoch time is tunable by user to determine output freqeuency to netCDF files
  • Read in multiple IODA files from one collection and regrid with the route handle therein
  • Loc stream is extended from root to distributed MPI processes based on background grids
  • Written out to NC files happens at Epoch time only, while each time step the regridded region is accumulated to an intermediate bundle
  • A derived type is allocated to handle each individual IODA type (filename, metadata, lon/lat/field)

The drawback is the original aircraft trajectory functionality is disabled, as the code is tuned for IODA files format.

Note:

  • I modified MAPL_HistoryGridComp.F90 to read in a table inside a collection

Input:

VERSION: 1

  COLLECTIONS:
  geovals
  ::

  GRID_LABELS:
  ::

  geovals.template:       '%y4%m2%d2_%h2%n2z.nc4',
  geovals.format:         'CFIO',
  geovals.obs_files:     # table start from next line
      DIR/aircraft.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/airs_aqua.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/amsr2_gcom-w1.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/amsua_aqua.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/amsua_metop-b.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/amsua_n15.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/amsua_n18.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/amsua_n19.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/atms_n20.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/atms_npp.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/avhrr3_metop-b.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/avhrr3_n18.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/avhrr3_n19.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/cris-fsr_n20.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/cris-fsr_npp.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/gmi_gpm.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/gps.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/iasi_metop-b.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/mhs_metop-b.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/mhs_n19.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/mls55_aura.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/omi_aura.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/ompsnm_npp.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/satwind.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/scatwind.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/sfc.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/sfcship.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/sondes.%y4%m2%d2T%h2%n2%S2Z.nc4
      DIR/ssmis_f17.%y4%m2%d2T%h2%n2%S2Z.nc4
::
  geovals.obs_file_interval:  '000000 060000'    #  fixed string format:  yymmdd hhmmss
  geovals.Epoch:          060000                 #  fixed integer format:  hhmmss
  geovals.nc_Index:       Location
  geovals.nc_Time:        MetaData/dateTime
  geovals.nc_Longitude:   MetaData/longitude
  geovals.nc_Latitude:    MetaData/latitude

  geovals.regrid_method:  'BILINEAR' ,
  geovals.fields:         'var2', 'Root'
                         'var3', 'Root'
::

Output: *.geovals.20190801_0300z.nc4 etc.

How Has This Been Tested?

Tested regridding the synthetic data from C720 to 29-IODA files spanning 12 hr time period (from 2019-07-31T21Z). Run finishes in 8 min with 6-cores. Previous experience shows when synthetic data runs can succeed through ExtDataDriver.x, the G5NR run can finish without error.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Trivial change (affects only documentation or cleanup)

Checklist:

  • I have tested this change with a run of GEOSgcm (if non-trivial)
  • I have added one of the required labels (0 diff, 0 diff trivial, 0 diff structural, non 0-diff)
  • I have updated the CHANGELOG.md accordingly following the style of Keep a Changelog

@metdyn metdyn added the 0 Diff The changes in this pull request have verified to be zero-diff with the target branch. label Sep 15, 2023
@metdyn metdyn requested a review from a team as a code owner September 15, 2023 21:36
CHANGELOG.md Outdated Show resolved Hide resolved
Copy link
Collaborator

@tclune tclune left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of items that need to be addressed immediately (see inline comments). A couple of others that can be addressed later.

@tclune
Copy link
Collaborator

tclune commented Sep 18, 2023

The initial description of this PR would probably make a better entry in the CHANGELOG than what is there now.

tclune
tclune previously approved these changes Sep 18, 2023
# Conflicts:
#	base/Plain_netCDF_Time.F90
#	gridcomps/History/MAPL_HistoryTrajectoryMod_smod.F90
@mathomp4
Copy link
Member

Aww nuts. I think my cleanup from #2346 touched a couple files that @metdyn did when I did an "update branch":

base/Plain_netCDF_Time.F90
gridcomps/History/MAPL_HistoryTrajectoryMod_smod.F90

So, I resolved the conflicts by assuming what @metdyn had is the correct code. This might mean some cleanup was lost, but that's safer.

@metdyn Can you pull your branch and test to make sure everything is still working.

@metdyn
Copy link
Contributor Author

metdyn commented Sep 19, 2023

@metdyn Can you pull your branch and test to make sure everything is still working.
Thank you, @mathomp4. Your merge works for the code, I have built and checked the output netCDF. They look good!

@mathomp4 mathomp4 self-requested a review September 19, 2023 16:50
Copy link
Member

@mathomp4 mathomp4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving and I have tested it and seems to be zero-diff (I'm guessing we don't use any of this in regular GCM work)

@mathomp4 mathomp4 merged commit 3a17ad0 into develop Sep 19, 2023
23 of 25 checks passed
@mathomp4 mathomp4 deleted the feature/PR_traj_sampler_Sep_15 branch September 19, 2023 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0 Diff The changes in this pull request have verified to be zero-diff with the target branch.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants