Address Warnings from pandas in METcalcpy code

[bikegeek]

Describe the Problem

A user discovered a DeprecationWarning in the METcalcpy code (utils module) when using METviewer:
dtcenter/METplus#1988

Identify all METcalcpy code that is using pd.append and replace with pd.concat.

When running the metviewer batch job from WCOSS2, receiving warning message as

Timeseries
==== Start Python error ====
/opt/metviewer_python/METcalcpy/metcalcpy/util/utils.py:707: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
output_ee_data = output_ee_data.append(series_data_after_ee)
/opt/metviewer_python/METcalcpy//metcalcpy/sum_stat.py:238: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
aggregated_values = aggregated_values.append(point_data)
:::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::;;
/opt/metviewer_python/METcalcpy//metcalcpy/sum_stat.py:238: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
aggregated_values = aggregated_values.append(point_data)
==== End Python error ====

Taylor Diagram
==== Start Python error ====
/opt/metviewer_python/METcalcpy/metcalcpy/util/utils.py:707: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
output_ee_data = output_ee_data.append(series_data_after_ee)
:::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::;;
/opt/metviewer_python/METcalcpy/metcalcpy/util/utils.py:707: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
output_ee_data = output_ee_data.append(series_data_after_ee)
==== End Python error ====
Performance plot seems to be free of similar message.

Expected Behavior

Create a test in METcalcpy/test/test_utils.py to reproduce the warning.

Environment

Describe your runtime environment:
1. Machine: (e.g. HPC name, Linux Workstation, Mac Laptop)
2. OS: (e.g. RedHat Linux, MacOS)
3. Software version number(s)

To Reproduce

Describe the steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
Post relevant sample data following these instructions:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-help-desk#ftp

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2792542, 2773542

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Organization level Project for support of the current coordinated release
• Select Repository level Project for development toward the next official release or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next bugfix version

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Bugfix Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of main_<Version>.
  Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
• Fix the bug and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Update requirements.txt to be consistent with NCO requirements
• Push local changes to GitHub.
• Submit a pull request to merge into main_<Version>.
  Pull request: bugfix <Issue Number> main_<Version> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issues
  Select: Organization level software support Project for the current coordinated release
  Select: Milestone as the next bugfix version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Complete the steps above to fix the bug on the develop branch.
  Branch name: bugfix_<Issue Number>_develop_<Description>
  Pull request: bugfix <Issue Number> develop <Description>
  Select: Reviewer(s) and Development issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Close this issue.

[bikegeek]

cannot reproduce using pandas 1.2.3. Updating to pandas 1.5 doesn't reproduce the warnings either. This is because the code that generates the warning isn't getting tested. Create tests for the util.utils.sum_stat aggregate_special_fields() code.

[bikegeek]

Possible candidates for replacing pd.append with pd.concat:

1. metcalcpy.sum_stat.aggregate_special_fields() line 238
   util.utils.equalize_axis_data() line 707

2. metcalcpy.agg_stat_event_equalize line 96 in calculate_value function

3. metcalcpy.agg_stat_event_equalize line 148 in
   run_ee_on_axis function

4. metcalcpy.util.pstd_statistics :

   a. line 271 in calculate_pstd_roc_auc():
   final_roc = final_roc.append(roc, ignore_index=True)

   b. line 272 in calculate_pstd_roc_auc():
   final_roc = final_roc.append(
   pd.DataFrame(
   {'thresh': 0, 'n11': 0, 'n10': 0, 'n01': 0, 'n00': 0, 'pody': 0,
   'pofd': 0},
   index=[0]),
   ignore_index=True)

   c. Verified not necessary in line 350 in
   _calc_common_stats(columns_names, input_data):
   if "oy_i" in column:
   sum_val = sum_column_data_by_name(input_data,
   columns_names, column)
   pct_perm['oy_i'].append(sum_val)

Reason:
the pct_perm['oy_i'] and pct_perm['on_i'] and pct_perm['thresh_i'] in this code block are list types, not data frames.

5. metcalcpy.util.utils

   a. line 510:

   # add it to the result
   input_data_frame =
   input_data_frame.append(rows_for_agg.iloc[:1])

   b. metcalcpy.util.utils line 537:

   # add it to the result
   input_data_frame =
   input_data_frame.append(rows_for_agg.iloc[:1])

   c.metcalcpy.util.utils line 707:

   output_ee_data =
   output_ee_data.append(series_data_after_ee)

6.metcalcpy.agg_stat_eqz
a. line 139 in perform_ee_on_axis():
output_ee_data =
output_ee_data.append(series_data_after_ee)

 b. line 90 in calculate_values():
        if self.params['series_val_2']:
            **output_ee_data =           output_ee_data.append(self.perform_ee_on_axis(prev_cases, '2'))**

Task:

1. generate tests for these methods/functions to reproduce warnings
2. modify affected code
3. re-run tests as they are written to ensure they are all failing (i.e. no more warnings)

[bikegeek]

Code changes made for items 1-4

[bikegeek]

code changes made to item 5 and 6.

[bikegeek]

still need to address issue 1
metcalcpy.sum_stat.aggregate_special_fields() line 238

[bikegeek]

Remember to make updates to the requirements.txt to be consistent with the nco_requirements