Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Supporting unsigned data type for the input NetCDF files #1492

Closed
7 of 21 tasks
hsoh-u opened this issue Sep 15, 2020 · 3 comments · Fixed by #1522
Closed
7 of 21 tasks

Supporting unsigned data type for the input NetCDF files #1492

hsoh-u opened this issue Sep 15, 2020 · 3 comments · Fixed by #1522
Assignees
@hsoh-u
Labels
MET: Library Code reporting: DTC NOAA R2O NOAA Research to Operations DTC Project requestor: NOAA/EMC NOAA Environmental Modeling Center type: enhancement Improve something that it is currently doing
Projects
MET-10.0.0-beta1 (10/22/20)
Milestone
MET 10.0.0

Comments

@hsoh-u
Copy link
Collaborator

hsoh-u commented Sep 15, 2020

MET does not support the unsigned data types for NetCDF file. MET only supported the unsigned values by applying "_Unsigned" attribute at the signed data type. Ho-Chun reported the data type changes at GOES16 AOD daata (https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96687). It's changed to the "unsigned short" and the "unsigned byte" type without "_Unsigned" attribute. The point2grid does not work with new AOD data file.

Describe the New Feature

MET library should support unsigned data types (unsigned byte/short/int/int64) and point2grid should check if the the data type is signed or unsigned before reading the NetCDF variables.

Acceptance Testing

point2grid should process the new GOES data file (dakota:/d3/personal/hsoh/data/RT-96687/OR_ABI-L2-AODC-M6_G16_s20202581441152_e20202581443525_c20202581445493.nc)

Time Estimate

2 or 3 days

Sub-Issues

Consider breaking the new feature down into sub-issues.

  • Add a checkbox for each sub-issue here.

Relevant Deadlines

NONE

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required (Howard)
  • Select scientist(s) or no scientist required (None)

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Review projects and select relevant Repository and Organization ones
  • Select milestone

Define Related Issue(s)

Consider the impact to the other METplus components.

New Feature Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s), Project(s), Milestone, and Linked issues
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@hsoh-u hsoh-u added type: enhancement Improve something that it is currently doing component: library code requestor: NOAA/EMC NOAA Environmental Modeling Center type: new feature Make it do something new alert: NEED ACCOUNT KEY Need to assign an account key to this issue labels Sep 15, 2020
@hsoh-u hsoh-u added this to the MET 10.0 milestone Sep 15, 2020
@hsoh-u hsoh-u added this to To do in MET-10.0.0-beta1 (10/22/20) via automation Sep 15, 2020
@hsoh-u hsoh-u self-assigned this Sep 15, 2020
@hsoh-u hsoh-u removed this from To do in MET-10.0.0-beta1 (10/22/20) Sep 15, 2020
@hsoh-u hsoh-u added alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle and removed type: new feature Make it do something new labels Sep 15, 2020
@JohnHalleyGotway JohnHalleyGotway changed the title Sopporting unsigned data type for the input NetCDF files Supporting unsigned data type for the input NetCDF files Sep 15, 2020
@JohnHalleyGotway JohnHalleyGotway added this to To do in MET-10.0.0-beta1 (10/22/20) via automation Sep 29, 2020
@JohnHalleyGotway JohnHalleyGotway moved this from To do to In progress in MET-10.0.0-beta1 (10/22/20) Sep 29, 2020
@hsoh-u
Copy link
Collaborator Author

hsoh-u commented Sep 30, 2020

How to handle a missing value for unsigned data type?
MET sets -9999 if the missing value is not given..

@JohnHalleyGotway
Copy link
Collaborator

How to handle a missing value for unsigned data type?
MET sets -9999 if the missing value is not given..

Howard, I assumed we were enhancing MET to READ unsigned data type inputs but STORE them into DataPlane objects as doubles. If that's the case, storing bad data as -9999 would not be a problem. But if there's a particular line of code or function that's causing a problem in the logic, please point me to it.

@hsoh-u hsoh-u mentioned this issue Oct 6, 2020
Closed
21 tasks
hsoh-u added a commit that referenced this issue Oct 13, 2020
@hsoh-u
#1492 Renamed get_global_att_double to get_global_att
c545e7d
hsoh-u added a commit that referenced this issue Oct 13, 2020
@hsoh-u
#1492 Changed qc_value as unsigned char
15292b9
hsoh-u added a commit that referenced this issue Oct 13, 2020
@hsoh-u
#1492 Renamed get_global_att_double to get_global_att. Passing refere…
d7d3d5d 
…nce for performance
hsoh-u added a commit that referenced this issue Oct 13, 2020
@hsoh-u
#1492 Implemented unsigned data type. Reduced duplicated code by usin…
679b105 
…g template
hsoh-u added a commit that referenced this issue Oct 13, 2020
@hsoh-u
#1492 Added a unit test, point2grid_GOES_16_AOD_TO_G212_unsigned
bfaa116
@hsoh-u hsoh-u linked a pull request Oct 13, 2020 that will close this issue
Feature 1492 nc_unsigned_data_type #1522
Merged
8 tasks
@hsoh-u hsoh-u moved this from In progress to Pull request review in MET-10.0.0-beta1 (10/22/20) Oct 13, 2020
hsoh-u added a commit that referenced this issue Oct 13, 2020
@hsoh-u
#1492 Refactoring: calls has_unsigned_attribute fir _Unsigned attribu…
22322a3 
…te handling
hsoh-u added a commit that referenced this issue Oct 13, 2020
@hsoh-u
#1492 Got the double attribute value from double and float data type
3f79c38
JohnHalleyGotway pushed a commit that referenced this issue Oct 14, 2020
@hsoh-u
Feature 1492 nc_unsigned_data_type (#1522)
cc35eef 
* #1492 Renamed get_global_att_double to get_global_att

* #1492 Changed qc_value as unsigned char

* #1492 Renamed get_global_att_double to get_global_att. Passing reference for performance

* #1492 Implemented unsigned data type. Reduced duplicated code by using template

* #1492 Added a unit test, point2grid_GOES_16_AOD_TO_G212_unsigned

* #1492 Refactoring: calls has_unsigned_attribute fir _Unsigned attribute handling

* #1492 Got the double attribute value from double and float data type
@JohnHalleyGotway
Copy link
Collaborator

On 10/15/2020, I recreated the develop-ref tag to get past the addition to the unit tests for this work.

But it was actually a little messier than that. On kiowa in NB20201015, the regression test for develop-ref actually failed because of bug #1508. So I manually ran comp_dir.sh to compare NB20201014/develop-ref (which succeeded) to NB20201015/develop. The only difference was the addition of one new output file:

ERROR: folder ../NB20201014/MET-develop-ref/test_output missing 1 files
point2grid/point2grid_GOES_16_AOD_TO_G212_unsigned.nc

MET-10.0.0-beta1 (10/22/20) automation moved this from Pull request review to Done Oct 15, 2020
@TaraJensen TaraJensen added reporting: DTC NOAA R2O NOAA Research to Operations DTC Project and removed alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle labels Dec 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hsoh-u hsoh-u

Labels
MET: Library Code reporting: DTC NOAA R2O NOAA Research to Operations DTC Project requestor: NOAA/EMC NOAA Environmental Modeling Center type: enhancement Improve something that it is currently doing
Projects
No open projects
MET-10.0.0-beta1 (10/22/20)
  
Done
Milestone
MET 10.0.0
Development

Successfully merging a pull request may close this issue.

Feature 1492 nc_unsigned_data_type dtcenter/MET
3 participants
@hsoh-u @JohnHalleyGotway @TaraJensen