Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance the performance of point2grid #1421

Closed
4 of 19 tasks
hsoh-u opened this issue Jul 22, 2020 · 1 comment · Fixed by #1607
Closed
4 of 19 tasks

Enhance the performance of point2grid #1421

hsoh-u opened this issue Jul 22, 2020 · 1 comment · Fixed by #1607
Assignees
@hsoh-u
Labels
priority: medium Medium Priority requestor: NOAA/EMC NOAA Environmental Modeling Center type: enhancement Improve something that it is currently doing
Projects
MET-10.0.0-beta3 (1/27/21)
Milestone
MET 10.0.0

Comments

@hsoh-u
Copy link
Collaborator

hsoh-u commented Jul 22, 2020 •
edited by TaraJensen

Describe the Enhancement

Jemie Wolff reported point2grid took less than 1 minute with HRRR domain (1799 by 1059) but it took more than 40 minutes with GFS domain (3072 by 1536) at Cheyenne.
HRRR case, the data is mapped to 1.7M grid cells (out of 1.8 M cells). GFS case, mapped to 3.1 M grid cells (out of 4.5 M cells).

Commands to load:

module use /glade/p/ral/jntp/MET/MET_releases/modulefiles
module load met/9.0
setenv MET_TMP_DIR /glade/p/ral/jntp/AF/cloud_vx/goes_data/test_met

To HRRR grid:

point2grid /glade/p/ral/jntp/AF/cloud_vx/goes_data/level2/abi_tb_fd_2020070300_C13_G16.nc /glade/p/ral/jntp/AF/cloud_vx/model_data/HRRR/2020060800-05/hrrr.t00z.wrfprsf01.grib2 test_hrrr_chey.nc -field 'name="tb"; level="(,)";' -v 3

To GFS grid:

point2grid /glade/p/ral/jntp/AF/cloud_vx/goes_data/level2/abi_tb_fd_2020070300_C13_G16.nc /glade/p/ral/jntp/AF/cloud_vx/model_data/GFS/gfs.20200608/00/gfs.t00z.special.grb2f001 test_gfs_chey.nc -field 'name="tb"; level="(,)";' -v 3

Time Estimate

4 to 8 hours

Sub-Issues

Consider breaking the enhancement down into sub-issues.

  • Add a checkbox for each sub-issue here.

Relevant Deadlines

NONE.

Funding Source

2791541

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required (Howard)
  • Select scientist(s) or no scientist required (None)

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Review projects and select relevant Repository and Organization ones
  • Select milestone

Define Related Issue(s)

Consider the impact to the other METplus components.

Enhancement Checklist

See the METplus Workflow for details.

  • Complete the issue definition above.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@hsoh-u hsoh-u added type: enhancement Improve something that it is currently doing component: application code priority: medium Medium Priority labels Jul 22, 2020
@hsoh-u hsoh-u self-assigned this Jul 22, 2020
@hsoh-u hsoh-u changed the title Enhanve the performance of point2grid Enhance the performance of point2grid Jul 22, 2020
@hsoh-u hsoh-u mentioned this issue Oct 2, 2020
Merged
@JohnHalleyGotway JohnHalleyGotway added the alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle label Oct 6, 2020
@JohnHalleyGotway JohnHalleyGotway added this to the MET 10.0 milestone Oct 6, 2020
@hsoh-u hsoh-u added this to To do in MET-10.0.0-beta3 (1/27/21) via automation Oct 7, 2020
@hsoh-u hsoh-u removed the alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle label Oct 7, 2020
@JohnHalleyGotway JohnHalleyGotway added this to To do in MET-10.0.0-beta2 (12/7/20) via automation Oct 27, 2020
@TaraJensen TaraJensen added requestor: DTC/Ensembles requestor: NOAA/EMC NOAA Environmental Modeling Center labels Dec 1, 2020
@hsoh-u hsoh-u moved this from To do to In progress in MET-10.0.0-beta2 (12/7/20) Dec 1, 2020
hsoh-u added a commit that referenced this issue Dec 4, 2020
@hsoh-u
#1421 Added log message
6ebfd79
hsoh-u added a commit that referenced this issue Dec 4, 2020
@hsoh-u
#1421 Added log message
b40976e
hsoh-u added a commit that referenced this issue Dec 4, 2020
@hsoh-u
#1421 Performance enhencement for high resolution data. Do not use a …
0638c49 
…cell mapping file
@JohnHalleyGotway JohnHalleyGotway removed this from In progress in MET-10.0.0-beta2 (12/7/20) Dec 4, 2020
@JohnHalleyGotway JohnHalleyGotway added this to To do in MET-10.0.0-beta3 (1/27/21) via automation Dec 4, 2020
@hsoh-u
Copy link
Collaborator Author

hsoh-u commented Dec 4, 2020

Another test data is UKMet_tripolar at /d3/personal/hsoh/data/MET-1492/UKMet_tripolar/prodm_op_gl.dm3d.grid_T_20200907_00.036_temp.nc

hsoh-u added a commit that referenced this issue Dec 11, 2020
@hsoh-u
#1421 read all 2D data at once
14c3917
hsoh-u added a commit that referenced this issue Dec 11, 2020
@hsoh-u
#1421 Added log message
584bb98
hsoh-u added a commit that referenced this issue Dec 11, 2020
@hsoh-u
#1421 Avoid extra extending of IntArray
82cf2bd
@hsoh-u hsoh-u moved this from To do to Pull request review in MET-10.0.0-beta3 (1/27/21) Dec 11, 2020
@JohnHalleyGotway JohnHalleyGotway linked a pull request Dec 14, 2020 that will close this issue
Feature 1421 point2grid performance #1607
Merged
8 tasks
hsoh-u added a commit that referenced this issue Dec 15, 2020
@hsoh-u
#1421 Reset cell mapping for new variable
e1d47fb
JohnHalleyGotway pushed a commit that referenced this issue Dec 15, 2020
@hsoh-u
Feature 1421 point2grid performance (#1607)
0bfe5bf 
* #1421 Added log message

* #1421 Added log message

* #1421 Performance enhencement for high resolution data. Do not use a cell mapping file

* #1421 read all 2D data at once

* #1421 Added log message

* #1421 Avoid extra extending of IntArray

* #1421 Reset cell mapping for new variable

Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu>
hsoh-u added a commit that referenced this issue Dec 17, 2020
@hsoh-u
#1421 Use double instead of float at getData
f3ecc35
JohnHalleyGotway pushed a commit that referenced this issue Dec 17, 2020
@hsoh-u
Feature 1421 precision (#1610)
995ea5a 
* Use 0 instead of NULL to avoid compile warning

* #1421 Use double instead of float at getData

* Use 0 instead of NULL to avoid compile warning
@JohnHalleyGotway JohnHalleyGotway moved this from Pull request review to Done in MET-10.0.0-beta3 (1/27/21) Dec 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hsoh-u hsoh-u

Labels
priority: medium Medium Priority requestor: NOAA/EMC NOAA Environmental Modeling Center type: enhancement Improve something that it is currently doing
Projects
No open projects
MET-10.0.0-beta3 (1/27/21)
  
Done
Milestone
MET 10.0.0
Development

Successfully merging a pull request may close this issue.

Feature 1421 point2grid performance dtcenter/MET
3 participants
@hsoh-u @JohnHalleyGotway @TaraJensen