Feature #2142 airnow_take2 #2373

JohnHalleyGotway · 2022-12-07T23:29:49Z

Still waiting for confirmation from @Ho-ChunHuang-NOAA that his testing on Hera using the feature_2142_airnow_take2 branch works as expected.
This PR includes the following changes:
- Updates documentation of the MET_AIRNOW_STATIONS environment variable.
- Updates file_handler.h/.cc to provide StringArray obs_units and write its contents to the NetCDF output file.
- Updates airnow_handler.h/.cc:
  - Improve error warning messages with more info about MET_AIRNOW_STATIONS
  - Fix bug in defining the varIndex for each observation.
  - Store the units string that was already being parsed.
  - Parse and write the "averaging period" as an integer number of seconds to the obs_lvl slot.
  - Write the obs_hgt as bad data since we don't know the height above ground level at which these obs are collected.

Expected Differences

Do these changes introduce new tools, command line arguments, or configuration file options? [No]

If yes, please describe:
Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]

If yes, please describe:

Pull Request Testing

Describe testing already performed for these changes:
Ran ascii2nc commands for AirNow inputs from unit_ascii2nc.xml many times.
Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
Please review logic of code changes.
GHA will produce diffs in the output due to bugfix, but make sure that none are unexpected.
Review the doc updates at:
- https://met.readthedocs.io/en/feature_2142_airnow_take2/Users_Guide/config_options.html#met-airnow-stations
Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]
Do these changes include sufficient testing updates? [Yes]
Will this PR result in changes to the test suite? [Yes]

If yes, describe the new output and/or changes to the existing output:
Updates the contents of the AirNow output files written by ascii2nc after varIndex bugfix.
Please complete this pull request review by [ASAP 12/9/22].

Pull Request Checklist

See the METplus Workflow for details.

Review the source issue metadata (required labels, projects, and milestone).
Complete the PR definition above.
Ensure the PR title matches the feature or bugfix branch name.
Define the PR metadata, as permissions allow.
Select: Reviewer(s)
Select: Organization level software support Project or Repository level development cycle Project
Select: Milestone as the version that will include these changes
After submitting the PR, select Development with the original issue number.
After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
Close the linked issue and delete your feature or bugfix branch from GitHub.

…variable names and report the correct index value. For daily v2 files, extract the time averaging number of hours and report it as the number of seconds in the observation level column to be consistent with the pb2nc handling of airnow observations.

…s 1-hour (3600 seconds). And report the height as bad data rather than using the elevation. That's more consistent with how we handle precip obs, and there's nothing in the input file to indicate the height above ground level at which these observations are collected.

…w_take2

JohnHalleyGotway · 2022-12-09T19:25:20Z

@davidalbo, FYI, here's more details about the testing I did for these changes. For each call to AirNow in unit_ascii2nc.xml, I ran that command with the updated version, used a python utility script to dump the output NetCDF file back to ascii, and then manually compared the input/output ascii files. I did this on my laptop, not seneca.

If you'd like like to the same on seneca, please LMK, and I can compile the feature branch there for your testing.

MET_BIN=/Volumes/d1/projects/MET/MET_development/MET-feature_2142_airnow_take2/bin
MET_TEST_INPUT=/Volumes/d1/projects/MET/MET_unit_test/MET_test_input
MET_TEST_OUTPUT=/Volumes/d1/projects/MET/MET_unit_test/MET_test_output

$MET_BIN/ascii2nc \
$MET_TEST_INPUT/obs_data/airnow/daily_data_v2.dat \
$MET_TEST_OUTPUT/ascii2nc/airnow/daily_data_v2_20220312.nc

python $MET_BIN/../scripts/utility/print_pointnc2ascii.py \
$MET_TEST_OUTPUT/ascii2nc/airnow/daily_data_v2_20220312.nc > daily_data_v2_20220312.txt

# Compare input daily_data_v2.dat to output daily_data_v2_20220312.txt

$MET_BIN/ascii2nc \
$MET_TEST_INPUT/obs_data/airnow/HourlyAQObs_2022031200.dat \
$MET_TEST_OUTPUT/airnow/HourlyAQObs_20220312.nc

python $MET_BIN/../scripts/utility/print_pointnc2ascii.py \
$MET_TEST_OUTPUT/ascii2nc/airnow/HourlyAQObs_20220312.nc > HourlyAQObs_20220312.txt

# Compare input HourlyAQObs_2022031200.dat to output HourlyAQObs_20220312.txt

$MET_BIN/ascii2nc \
$MET_TEST_INPUT/obs_data/airnow/HourlyData_2022031200.dat \
$MET_TEST_OUTPUT/ascii2nc/airnow/HourlyData_20220312.nc

python $MET_BIN/../scripts/utility/print_pointnc2ascii.py \
$MET_TEST_OUTPUT/ascii2nc/airnow/HourlyData_20220312.nc > HourlyData_20220312.txt

# Compare input HourlyData_2022031200.dat to output HourlyData_20220312.txt

davidalbo · 2022-12-09T20:10:49Z

@davidalbo, FYI, here's more details about the testing I did for these changes. For each call to AirNow in unit_ascii2nc.xml, I ran that command with the updated version, used a python utility script to dump the output NetCDF file back to ascii, and then manually compared the input/output ascii files. I did this on my laptop, not seneca.

If you'd like like to the same on seneca, please LMK, and I can compile the feature branch there for your testing.

MET_BIN=/Volumes/d1/projects/MET/MET_development/MET-feature_2142_airnow_take2/bin
MET_TEST_INPUT=/Volumes/d1/projects/MET/MET_unit_test/MET_test_input
MET_TEST_OUTPUT=/Volumes/d1/projects/MET/MET_unit_test/MET_test_output

$MET_BIN/ascii2nc \
$MET_TEST_INPUT/obs_data/airnow/daily_data_v2.dat \
$MET_TEST_OUTPUT/ascii2nc/airnow/daily_data_v2_20220312.nc

python $MET_BIN/../scripts/utility/print_pointnc2ascii.py \
$MET_TEST_OUTPUT/ascii2nc/airnow/daily_data_v2_20220312.nc > daily_data_v2_20220312.txt

# Compare input daily_data_v2.dat to output daily_data_v2_20220312.txt

$MET_BIN/ascii2nc \
$MET_TEST_INPUT/obs_data/airnow/HourlyAQObs_2022031200.dat \
$MET_TEST_OUTPUT/airnow/HourlyAQObs_20220312.nc

python $MET_BIN/../scripts/utility/print_pointnc2ascii.py \
$MET_TEST_OUTPUT/ascii2nc/airnow/HourlyAQObs_20220312.nc > HourlyAQObs_20220312.txt

# Compare input HourlyAQObs_2022031200.dat to output HourlyAQObs_20220312.txt

$MET_BIN/ascii2nc \
$MET_TEST_INPUT/obs_data/airnow/HourlyData_2022031200.dat \
$MET_TEST_OUTPUT/ascii2nc/airnow/HourlyData_20220312.nc

python $MET_BIN/../scripts/utility/print_pointnc2ascii.py \
$MET_TEST_OUTPUT/ascii2nc/airnow/HourlyData_20220312.nc > HourlyData_20220312.txt

# Compare input HourlyData_2022031200.dat to output HourlyData_20220312.txt

Sure, I'll try running your tests from seneca, so please do build there.

davidalbo

I approve this pull request.

I reviewed the documentation and it did add meaningful and useful content.
I reviewed the code changes and they all made sense and were clear improvements, some cosmetic, and others adding functionality such as correctly handling units.
The testing done showed that the changes made did give the expected changes in results.

JohnHalleyGotway · 2022-12-09T22:36:56Z

Thanks @davidalbo for the review. Ho-Chun has no objection to us proceeding with these changes. So I'll merge this PR and proceed with the release.

Ho-ChunHuang-NOAA · 2022-12-11T14:21:22Z

Sorry John: Please review the comments in feature #2142. I need advice to enable the ascii2nc airnow daily max/ave obs to point_stat. Ho-Chun Huang, Ph.D. Physical Scientist III, Contractor with Lynker in Support of NOAA/NWS/NCEP/EMC, U.S. Department of Commerce 5830 University Research Ct., Rm. 2792 College Park, MD 20740 ***@***.*** ***@***.***> 301-683-3958

…

On Fri, Dec 9, 2022 at 5:37 PM johnhg ***@***.***> wrote: Merged #2373 <#2373> into develop. — Reply to this email directly, view it on GitHub <#2373 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALPHE3BUVEVDI2UJBU4XMUTWMOYBVANCNFSM6AAAAAASXNW2RQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> Co-authored-by: Dave Albo <dave@seneca.rap.ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: Seth Linden <linden@seneca.rap.ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@seneca.rap.ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: davidalbo <dave@ucar.edu> Co-authored-by: Seth Linden <linden@ucar.edu> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu>

JohnHalleyGotway added 5 commits December 7, 2022 11:25

Per #2142, remove debugging code writing the header contents to cout.

b04eb84

Per #2142, update the ascii2nc airnow handler to write out units.

e25b142

Merge remote-tracking branch 'origin/develop' into feature_2142_airno…

19b7b66

…w_take2

JohnHalleyGotway added this to the MET 11.0.0 milestone Dec 7, 2022

JohnHalleyGotway requested a review from davidalbo December 7, 2022 23:29

JohnHalleyGotway linked an issue Dec 7, 2022 that may be closed by this pull request

Add support for EPA AirNow ASCII data in ASCII2NC #2142

Closed

16 tasks

JohnHalleyGotway added 6 commits December 9, 2022 10:08

Per #2142, add logic to print a warning if the units string changes.

a901f8c

Merge remote-tracking branch 'origin/develop' into feature_2142_airno…

fcd7cee

…w_take2

Per #2142, add more documentation about setting MET_AIRNOW_STATIONS.

f419607

Per #2142, consistent whitespace formatting.

4fa77e2

Per #2142, consistent whitespace formatting.

508fe4a

Per #2142, delete trailing whitespace.

d859b78

JohnHalleyGotway marked this pull request as ready for review December 9, 2022 18:48

davidalbo approved these changes Dec 9, 2022

View reviewed changes

JohnHalleyGotway merged commit 8a910a6 into develop Dec 9, 2022

JohnHalleyGotway deleted the feature_2142_airnow_take2 branch December 9, 2022 22:37

JohnHalleyGotway mentioned this pull request Dec 12, 2022

Update develop-ref after #2371, #2372, and #2373 #2376

Merged

DanielAdriaansen mentioned this pull request Apr 29, 2024

Add MET ASCII2NC supported datasets to verification datasets dtcenter/METplus#2568

Open

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature #2142 airnow_take2 #2373

Feature #2142 airnow_take2 #2373

JohnHalleyGotway commented Dec 7, 2022 •

edited

Loading

JohnHalleyGotway commented Dec 9, 2022

davidalbo commented Dec 9, 2022

davidalbo left a comment

JohnHalleyGotway commented Dec 9, 2022

Ho-ChunHuang-NOAA commented Dec 11, 2022 via email

Feature #2142 airnow_take2 #2373

Feature #2142 airnow_take2 #2373

Conversation

JohnHalleyGotway commented Dec 7, 2022 • edited Loading

Expected Differences

Pull Request Testing

Pull Request Checklist

JohnHalleyGotway commented Dec 9, 2022

davidalbo commented Dec 9, 2022

davidalbo left a comment

Choose a reason for hiding this comment

JohnHalleyGotway commented Dec 9, 2022

Ho-ChunHuang-NOAA commented Dec 11, 2022 via email

JohnHalleyGotway commented Dec 7, 2022 •

edited

Loading