Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance METdataio schema to handle changes to the ECNT, VL1L2, VAL1L2, and VCNT line types during the MET 12.0.0 beta4 development cycle #282

Closed
11 of 25 tasks
JohnHalleyGotway opened this issue Feb 14, 2024 · 2 comments
Closed
11 of 25 tasks

Enhance METdataio schema to handle changes to the ECNT, VL1L2, VAL1L2, and VCNT line types during the MET 12.0.0 beta4 development cycle #282

JohnHalleyGotway opened this issue Feb 14, 2024 · 2 comments
Assignees
@bikegeek @hankenstein2
Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue component: load Tasks related to the database loading priority: blocker Blocker requestor: NOAA/EMC NOAA Environmental Modeling Center required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone type: enhancement Improve something that it is currently doing
Milestone
METdataio-3.0.0

Comments

@JohnHalleyGotway
Copy link
Contributor

JohnHalleyGotway commented Feb 14, 2024 •
edited
Loading

Describe the Enhancement

Issue dtcenter/MET#2583 and pull request dtcenter/MET#2838 adds new columns to the end of the ECNT line type.

Issue dtcenter/MET#2395 and pull request dtcenter/MET#2820 add new columns to the existing .stat output lines for MET version 12.0.0, as described below:

  • Add 3 new VL1L2 columns:
    DIR_ME, DIR_MAE, and DIR_MSE
  • Add 3 new VAL1L2 columns:
    DIRA_ME, DIRA_MAE, DIRA_MSE
  • Add 12 new VCNT columns:
    DIR_ME, DIR_MAE, DIR_MSE, and DIR_RMSE plus _BCL and _BCU columns to store the bootstrap CI's.

This issue is to update the METdataio loader to parse data from these new columns.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.
None needed.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Select Repository and/or Organization level Project(s) or add alert: NEED CYCLE ASSIGNMENT label
  • Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

Enhancement Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Add any new Python packages to the METplus Components Python Requirements table.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Development issues

Select: Repository level development cycle Project for the next official release
Select: Milestone as the next official version

  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@JohnHalleyGotway JohnHalleyGotway added type: enhancement Improve something that it is currently doing alert: NEED ACCOUNT KEY Need to assign an account key to this issue component: load Tasks related to the database loading priority: blocker Blocker requestor: NOAA/EMC NOAA Environmental Modeling Center required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone labels Feb 14, 2024
@JohnHalleyGotway JohnHalleyGotway added this to the METdataio-3.0.0 milestone Feb 14, 2024
This was referenced Feb 14, 2024
Open
Closed
Closed
@JohnHalleyGotway JohnHalleyGotway changed the title Update schema to load new wind direction statistics in the VL1L2, VAL1L2, and VCNT line types Update schema to handle changes to the ECNT, VL1L2, VAL1L2, and VCNT line types during the MET 12.0.0 beta4 development cycle Mar 14, 2024
@JohnHalleyGotway JohnHalleyGotway changed the title Update schema to handle changes to the ECNT, VL1L2, VAL1L2, and VCNT line types during the MET 12.0.0 beta4 development cycle Enhance METdataio schema to handle changes to the ECNT, VL1L2, VAL1L2, and VCNT line types during the MET 12.0.0 beta4 development cycle Mar 14, 2024
@bikegeek
Copy link
Collaborator

bikegeek commented Apr 1, 2024

update the data for the METreformat tests. These changes may impact the reformatter for ECNT, VL1L2, and VCNT

@bikegeek
Copy link
Collaborator

bikegeek commented Apr 1, 2024 •
edited
Loading

Outline of work to do:

Add the new columns

METdbLoad/ush/constants.py

  • LINE_DATA_FIELDS[VCNT]
  • LINE_DATA_FIELDS[VAL1L2]
  • LINE_DATA_FIELDS[VL1L2]
  • LINE_DATA_FIELDS[ECNT]
  • LC_ECNT_SPECIFIC
  • LC_VCNT_SPECIFIC
  • LC_VCNT_STATISTICS_HEADERS
  • NUM_STAT_VCNT_COLS (update value)
    from 87 to 99 (addition of 12 new columns total)
  • NUM_STAT_VL1L2_COLS (update value)
    from 35 to 38
  • NUM_STAT_ECNT_COLS
    from 50 to 52
  • LC_VCNT_BOOTSTRAP_HEADERS
  • LC_VL1L2_SPECIFIC
  • VCNT_BCL_HEADERS
  • VCNT_BCU_HEADERS

METdbLoad/updates

  • create a new update_for_5_1_beta4.sql file

  • add syntax like this to add new columns:

ALTER TABLE line_data_val1l2
ADD COLUMN dira_ma DOUBLE |

  • line_data_vcnt
  • line_data_vl1l2
  • line_data_val2l2
  • line_data_ecnt

in METdbLoad/sql/mv_mysql.sql:

  • update the line_data_vcnt table with the new columns and their data types (~ line 1632)
  • update the line_data_ecnt table with the new columns and their data types (~ line 421)
  • update the line_data_vl1l2 table with the new columns and their data types (~ line 889)
  • update the line_data_val1l2 table with the new columns and their data types (~ line 927)

Test

  • update the Github workflow compare_db.yml
  • with new data (that contains the latest column additions)
    use data on 'seneca': d1/projects/MET/MET_regression/develop/NB20240401/MET-develop/test_output/point_stat
  • verify that new data is loading in the database
  • skip the tests for VL1L2, ECNT, and VCNT in METdataio/METreformat/test/test_reformatting.py for now
    see Update test data for VCNT, ECNT, and VL1L2 with newly added columns to the linetype #288

Documentation:

  • add more information in the Contributor's Guide in the Update the Database Schema section with sample syntax to
    add a new column to an existing table.

bikegeek added a commit that referenced this issue Apr 2, 2024
@bikegeek
Issue #282 update the schema for additions to the ECNT, VCNT, VL1L2, …
4e47ba8 
…and VAL1L2 linetypes
@bikegeek bikegeek mentioned this issue Apr 2, 2024
Open
22 tasks
bikegeek added a commit that referenced this issue Apr 2, 2024
@bikegeek
Issue #282 updates for the additional columns in ECNT, VL1L2, VAL1L2,…
1e28012 
… and VCNT linetypes
bikegeek added a commit that referenced this issue Apr 2, 2024
@bikegeek
Issue #282 skip the tests for VCNT, VL1L2, and ECNT. These no longer …
20c77bd 
…work with the updates to constants.py as a result of additional columns in these linetypes.
bikegeek added a commit that referenced this issue Apr 3, 2024
@bikegeek
Issue #282 Update the schema with the latest column additions to VCNT…
78527cf 
…, ECNT, VL1L2, and VAL1L2 linetype tables
bikegeek added a commit that referenced this issue Apr 5, 2024
@bikegeek
issue #282 tests for checking the updates to the schema
2200ba2
bikegeek added a commit that referenced this issue Apr 5, 2024
@bikegeek
issue #282 added the RMSE columns for VCNT
8c79e45
bikegeek added a commit that referenced this issue Apr 8, 2024
@bikegeek
Feature 282 enhance schema (#292)
d0bf16c 
* Issue #282 update the schema for additions to the ECNT, VCNT, VL1L2, and VAL1L2 linetypes

* Add sample syntax for adding a column to a linetype table.

* Corrected the incorrect table from line_data_val1l1 to line_data_val2.

* Fix warnings due to errors in rst syntax

* Issue #282 updates for the additional columns in ECNT, VL1L2, VAL1L2, and VCNT linetypes

* Issue #282 skip the tests for VCNT, VL1L2, and ECNT.  These no longer work with the updates to constants.py as a result of additional columns in these linetypes.

* Issue #282 Update the schema with the latest column additions to VCNT, ECNT, VL1L2, and VAL1L2 linetype tables

* issue #282 tests for checking the updates to the schema

* Tarballs with the latest data for the ECNT, VCNT, VL1L2, and VAL1L2 linetypes

* issue #282 added the RMSE columns for VCNT

* Added test for loaded ECNT data

* Additional information on testing

* Added step to clean up/remove test database after testing is finished.

* modify instructions to just remove the test database.

* some of the line_data_vl1l2 commands were mislabelled as line_data_val1l2

* replace version from 5_1 to 6

* try disabling this test until the updated schema changes have been merged into the develop branch and baseold has been updated

* remove the code that was used to disable workflow-doesn't work
@bikegeek bikegeek closed this as completed Apr 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bikegeek bikegeek

@hankenstein2 hankenstein2

Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue component: load Tasks related to the database loading priority: blocker Blocker requestor: NOAA/EMC NOAA Environmental Modeling Center required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone type: enhancement Improve something that it is currently doing
Projects
METplus-Analysis-6.0.0 Development
Status: ✅ Done
Milestone
METdataio-3.0.0
Development

No branches or pull requests
3 participants
@bikegeek @JohnHalleyGotway @hankenstein2