Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create aggregation support for MET .stat output #325

Closed
4 of 25 tasks
bikegeek opened this issue Aug 16, 2023 · 0 comments
Closed
4 of 25 tasks

Create aggregation support for MET .stat output #325

bikegeek opened this issue Aug 16, 2023 · 0 comments
Assignees
Labels
priority: high High Priority reporting: DTC NOAA R2O NOAA Research to Operations DTC Project requestor: DTC/T&E General DTC Testing and Evaluation work required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone type: enhancement Improve something that it is currently doing

Comments

@bikegeek
Copy link
Collaborator

bikegeek commented Aug 16, 2023

Describe the Enhancement

Time Estimate

~1-3 days for each sub-task

Sub-Issues

Consider breaking the enhancement down into sub-issues.
This work should be done under metcalcpy/pre-processing

  • Identify the settings that will go into the YAML config file to support the aggregation features found in the MET stat analysis tool
  • Using a pandas dataframe as input (expected output from reading, filtering, and reformatting the MET .stat file), invoke the necessary aggregation methods in METcalcpy aggregation modules (e.g. agg_stat.py, etc). Provide output in the form of a pandas dataframe (which is required of the METreformat code).
  • Generate tests
  • Generate documentation

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Select Repository and/or Organization level Project(s) or add alert: NEED CYCLE ASSIGNMENT label
  • Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

Enhancement Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Add any new Python packages to the METplus Components Python Requirements table.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Development issues

Select: Repository level development cycle Project for the next official release
Select: Milestone as the next official version

  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@bikegeek bikegeek added type: enhancement Improve something that it is currently doing alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED MORE DEFINITION Not yet actionable, additional definition required priority: high High Priority alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle reporting: DTC NOAA R2O NOAA Research to Operations DTC Project requestor: DTC/T&E General DTC Testing and Evaluation work required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone labels Aug 16, 2023
@bikegeek bikegeek transferred this issue from dtcenter/METdataio Aug 30, 2023
bikegeek added a commit that referenced this issue Jan 29, 2024
bikegeek added a commit that referenced this issue Jan 29, 2024
bikegeek added a commit that referenced this issue Jan 29, 2024
bikegeek added a commit that referenced this issue Jan 29, 2024
…(included the path to the agg_stat.py module)
bikegeek added a commit that referenced this issue Jan 31, 2024
bikegeek added a commit that referenced this issue Jan 31, 2024
bikegeek added a commit that referenced this issue Feb 1, 2024
…to fail.

Remove pandas chaining such as:
  df['column_name'][index] = var_name

with:
 df.loc[index, 'column_name'] = var_name
bikegeek added a commit that referenced this issue Feb 1, 2024
…to fail.

Remove pandas chaining such as:
  df['column_name'][index] = var_name

with:
 df.loc[index, 'column_name'] = var_name
bikegeek added a commit that referenced this issue Feb 1, 2024
bikegeek added a commit that referenced this issue Feb 1, 2024
bikegeek added a commit that referenced this issue Feb 2, 2024
fisherhucar pushed a commit that referenced this issue Feb 2, 2024
* Update README.md (#321)

update required Python version to 3.10+

* Added aggregation features

* Test

* removed folders

* Added aggregation features

* Updates settings and improved folder search algorithm; Added README

* Corrected FBIAS stat fields

* Create Aggregation.rst

creating file from:https://github.com/dtcenter/METcalcpy/blob/feature_325_aggregation_support/metcalcpy/pre_processing/aggregation/README.md
Copied Vertical Interpolation as a template

* adding aggregation

* Rename Aggregation.rst to aggregation.rst

* first pass at cleaning up warnings

* changing to 3rd person

* issue #325 CTS data from RRFS to test aggregation

* Issue #325 added background on agg_stat.py

* issue #325 added instructions for bash and csh, added links to external references

* issue #325 fix syntax for subsection

* Issue #325 fix grammar, add instructions for importing and invoked by another script

* issue #325 more fixes to grammar for import instructions

* issue #325 added corrected instructions for running via command-line (included the path to the agg_stat.py module)

* Issue #325 modify config file to specify valid paths for input and output files.

* Issue #325 modified for User's Guide instructions

* issue #325 added reformatted data for ECNT and compatible for METcalcpy agg_stat input

* Delete test/data/rrfs_cts_reformatted.data

not used for testing.  Using the ECNT data instead.

* issue #325 pytest on ECNT data reformatted with METdataio METreformat and aggregation statistics calculated

* issue #325 added latest test for ECNT aggregation

* Issue #325 address pandas future warning that causes current pytests to fail.
Remove pandas chaining such as:
  df['column_name'][index] = var_name

with:
 df.loc[index, 'column_name'] = var_name

* Issue #325 address pandas future warning that causes current pytests to fail.
Remove pandas chaining such as:
  df['column_name'][index] = var_name

with:
 df.loc[index, 'column_name'] = var_name

* Issue #325 updated input data to ECNT data, corrected the explanation of expected input format for agg_stat.

* Issue #325 modify config file to use RRFS ECNT .stat data reformatted by METdataio

* issue #325 point to actual config file via literalinclude

* issue #325 replace reference to the CTS output file with ECNT

* replace pandas append with concat

* Update unit_tests.yml

added test_reformatted_for_agg.py

* fixed syntax error with list

* issue #325 update test data with correctly reformatted ECNT line data

* issue #325 removed some unneccessary text

---------

Co-authored-by: VanderleiVargas-NOAA <vanderlei.vargas@noaa.gov>
Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com>
@bikegeek bikegeek closed this as completed Feb 6, 2024
@bikegeek bikegeek removed alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED MORE DEFINITION Not yet actionable, additional definition required alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle labels Feb 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: high High Priority reporting: DTC NOAA R2O NOAA Research to Operations DTC Project requestor: DTC/T&E General DTC Testing and Evaluation work required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone type: enhancement Improve something that it is currently doing
Projects
Status: ✅ Done
Development

No branches or pull requests

3 participants