Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect calculation of PSTD_ROC_AUC statistic #306

Closed
22 tasks
TatianaBurek opened this issue May 12, 2023 · 1 comment · Fixed by #307
Closed
22 tasks

Incorrect calculation of PSTD_ROC_AUC statistic #306

TatianaBurek opened this issue May 12, 2023 · 1 comment · Fixed by #307
Assignees
Labels

Comments

@TatianaBurek
Copy link
Collaborator

Describe the Problem

Aggregation statistic doesn't calculate correct values for PSTD_ROC_AUC statistic.
plot_20230512_203500.agg_stat.info.txt
plot_20230512_203500.data.agg_stat.txt

Expected Behavior

values should be like this:
RRFSE_CONUS_ICperts_nostoch.rrfs_conuscompact_3km_prob 160000 PSTD_ROC_AUC 0.954596669974263
RRFSE_CONUS_ICperts_nostoch.rrfs_conuscompact_3km_prob 170000 PSTD_ROC_AUC 0.951804745793181
RRFSE_CONUS_ICperts_nostoch.rrfs_conuscompact_3km_prob 180000 PSTD_ROC_AUC 0.952286333211654
RRFSE_CONUS_ICperts_nostoch.rrfs_conuscompact_3km_prob 190000 PSTD_ROC_AUC 0.950166505972019
RRFSE_CONUS_ICperts_stoch.rrfs_conuscompact_3km_prob 160000 PSTD_ROC_AUC 0.959631793570794
RRFSE_CONUS_ICperts_stoch.rrfs_conuscompact_3km_prob 170000 PSTD_ROC_AUC 0.959012367604244
RRFSE_CONUS_ICperts_stoch.rrfs_conuscompact_3km_prob 180000 PSTD_ROC_AUC 0.95513305325736
RRFSE_CONUS_ICperts_stoch.rrfs_conuscompact_3km_prob 190000 PSTD_ROC_AUC 0.950740286744192

Environment

Describe your runtime environment:
1. Machine: (e.g. HPC name, Linux Workstation, Mac Laptop)
2. OS: (e.g. RedHat Linux, MacOS)
3. Software version number(s)

To Reproduce

Describe the steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
Post relevant sample data following these instructions:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-help-desk#ftp

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2771021

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Select Organization level Project for support of the current coordinated release
  • Select Repository level Project for development toward the next official release or add alert: NEED CYCLE ASSIGNMENT label
  • Select Milestone as the next bugfix version

Define Related Issue(s)

Consider the impact to the other METplus components.

Bugfix Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of main_<Version>.
    Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
  • Fix the bug and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into main_<Version>.
    Pull request: bugfix <Issue Number> main_<Version> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Development issues
    Select: Organization level software support Project for the current coordinated release
    Select: Milestone as the next bugfix version
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Complete the steps above to fix the bug on the develop branch.
    Branch name: bugfix_<Issue Number>_develop_<Description>
    Pull request: bugfix <Issue Number> develop <Description>
    Select: Reviewer(s) and Development issues
    Select: Repository level development cycle Project for the next official release
    Select: Milestone as the next official version
  • Close this issue.
@TatianaBurek TatianaBurek added this to the METcalcpy 2.1.0 milestone May 12, 2023
@TatianaBurek TatianaBurek self-assigned this May 12, 2023
@TatianaBurek
Copy link
Collaborator Author

The problem was in incorrect indexes in the final DataFrame. After concatenation the indexes were not reset and the roc_auc calculation was performed using wrong indexes.
After adding 'reset_index' roc_auc was calculated correctly. I also added 'reset_index' to other statistic calculations

TatianaBurek added a commit that referenced this issue May 15, 2023
TatianaBurek added a commit that referenced this issue May 15, 2023
@TatianaBurek TatianaBurek linked a pull request May 15, 2023 that will close this issue
13 tasks
bikegeek pushed a commit that referenced this issue May 23, 2023
* make sure that the parameters file is closed #306

* perform dataframe reindexing before  statistic calculation #306

* add new test to test roc_auc #306

* initial version #306

* replace deprecated argument 'interpolation' to 'method' in percentile method #306
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants