METdataio, in METreformat address PerformanceWarning #219
Labels
alert: NEED ACCOUNT KEY
Need to assign an account key to this issue
component: METdataio
priority: high
High Priority
reporting: DTC NOAA R2O
NOAA Research to Operations DTC Project
required: FOR OFFICIAL RELEASE
Required to be completed in the official release for the assigned milestone
type: bug
Fix something that is not working
Milestone
PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling
frame.insert
many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, usenewframe = frame.copy()
cnt_df.insert(loc=0, column='Idx', value=idx)
Remove the requirement for the XML specification file used by METdbLoad. Encapsulate that information in the YAML configuration file and pass that into the METdbLoad module read_data_files.
Describe the Problem
When using the METreformatter on large data sets, the PerformanceWarning is observed.
Expected Behavior
Provide a clear and concise description of what you expected to happen here.
Environment
Describe your runtime environment:
1. Machine: (e.g. HPC name, Linux Workstation, Mac Laptop)
2. OS: (e.g. RedHat Linux, MacOS)
3. Software version number(s)
anything that uses pandas 2.0.3 or above
(previously could NOT reproduce when using pandas 1.5.2)
To Reproduce
Describe the steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
Post relevant sample data following these instructions:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-help-desk#ftp
Relevant Deadlines
List relevant project deadlines here or state NONE.
Funding Source
Define the source of funding and account keys here or state NONE.
Define the Metadata
Assignee
Labels
Projects and Milestone
Define Related Issue(s)
Consider the impact to the other METplus components.
Bugfix Checklist
See the METplus Workflow for details.
Branch name:
bugfix_<Issue Number>_main_<Version>_<Description>
Pull request:
bugfix <Issue Number> main_<Version> <Description>
Select: Reviewer(s) and Development issues
Select: Organization level software support Project for the current coordinated release
Select: Milestone as the next bugfix version
Branch name:
bugfix_<Issue Number>_develop_<Description>
Pull request:
bugfix <Issue Number> develop <Description>
Select: Reviewer(s) and Development issues
Select: Repository level development cycle Project for the next official release
Select: Milestone as the next official version
The text was updated successfully, but these errors were encountered: