Skip to content

Add basic plot_pred_obs_pointinterval() functionality#146

Merged
dylanhmorris merged 17 commits into
mainfrom
dhm-predicted-actual-pointintervals
Jun 11, 2025
Merged

Add basic plot_pred_obs_pointinterval() functionality#146
dylanhmorris merged 17 commits into
mainfrom
dhm-predicted-actual-pointintervals

Conversation

@dylanhmorris

@dylanhmorris dylanhmorris commented May 22, 2025

Copy link
Copy Markdown
Collaborator

Closes #97

EDIT: expand to see previous request for feedback, now addressed

Would like feedback on how automagically/not to handle the question of horizons and forecast dates (aka reference dates). At present, the plot is faceted. Facet row groups are given the by horizon data column. Facet column groups are given by the interaction of every data column that isn't one of the following: target end date, predicted, observed, quantile level, or horizon. This means that if any columns are redundant with horizon given target end date (e.g. a forecast date = target end date - horizon), each facet will contain at most one prediction and observation.

See the following reprex that demonstrates the current behavior:

library(forecasttools)
library(scoringutils)
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union

# one facet per forecast
scoringutils::example_quantile |>
    filter(target_type == "Cases", location == "DE") |>
    plot_pred_obs_pointintervals()
#> ℹ Some rows containing NA values may be removed. This is fine if not
#>   unexpected.
#> Warning in ggplot2::scale_y_continuous(transform = y_transform): log-10
#> transformation introduced infinite values.

## removing the forecast_date column, which is redundant
## with horizon, gives more expected behavior
scoringutils::example_quantile |>
    filter(target_type == "Cases", location == "DE") |>
    select(-"forecast_date") |>
    plot_pred_obs_pointintervals()
#> ℹ Some rows containing NA values may be removed. This is fine if not
#>   unexpected.
#> Warning in ggplot2::scale_y_continuous(transform = y_transform): log-10
#> transformation introduced infinite values.

Created on 2025-05-22 with reprex v2.1.1

We could

  • Allow the user to designate a forecast_date/reference_date column and do something with it, e.g.
    • Require exactly one of a horizon column and a forecast_date column, error if both are present, compute horizon from forecast date + target date if needed
    • Silently drop the forecast_date when horizon is also present, use it as a fallback when horizon is absent
    • etc
  • Warn or error if there is one forecast per facet
  • Leave as is
  • Something else?

@codecov

codecov Bot commented May 22, 2025

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 66.92%. Comparing base (6a19d93) to head (246bc06).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #146      +/-   ##
==========================================
+ Coverage   64.29%   66.92%   +2.63%     
==========================================
  Files          28       28              
  Lines        1574     1657      +83     
==========================================
+ Hits         1012     1109      +97     
+ Misses        562      548      -14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@damonbayer

damonbayer commented May 22, 2025

Copy link
Copy Markdown
Collaborator
  • Silently drop the forecast_date when horizon is also present, use it as a fallback when horizon is absent

This is my preferred option. If horizon is not present, we can either

  1. fallback to automatically compute horizons from horizons_from_target_end_dates or
  2. Error and tell the user that they may want to use horizons_from_target_end_dates to create the horizon column.

@dylanhmorris

dylanhmorris commented Jun 5, 2025

Copy link
Copy Markdown
Collaborator Author

Error and tell the user that they may want to use horizons_from_target_end_dates to create the horizon column.

Went with this in f8d53ad. Decided it was worth avoiding making the user specify the horizon timescale

@dylanhmorris
dylanhmorris marked this pull request as ready for review June 5, 2025 21:15
@dylanhmorris
dylanhmorris requested a review from damonbayer as a code owner June 5, 2025 21:15
@damonbayer
damonbayer requested a review from Copilot June 5, 2025 22:24

This comment was marked as outdated.

Comment thread R/plot_scorable_table.R Outdated
Comment thread R/plot_scorable_table.R Outdated
Comment thread R/plot_scorable_table.R Outdated
Comment thread R/plot_scorable_table.R Outdated
Comment thread R/plot_scorable_table.R Outdated
Co-authored-by: Damon Bayer <xum8@cdc.gov>
@damonbayer
damonbayer self-requested a review June 5, 2025 22:32

@damonbayer damonbayer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks nice. Just a few small changes requested.

dylanhmorris and others added 4 commits June 9, 2025 15:02
Co-authored-by: Damon Bayer <xum8@cdc.gov>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@dylanhmorris
dylanhmorris requested a review from damonbayer June 9, 2025 19:06
@dylanhmorris

Copy link
Copy Markdown
Collaborator Author

Ready for re-review @damonbayer

Comment thread R/plot_scorable_table.R Outdated
Comment thread R/plot_scorable_table.R Outdated
@damonbayer
damonbayer requested a review from Copilot June 11, 2025 16:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a basic implementation of plot_pred_obs_pointintervals() to visualize median predictions and prediction intervals against observed data. Key changes include implementation of the new plotting function in R/plot_scorable_table.R, accompanying documentation in man/plot_pred_obs_pointintervals.Rd, updated tests in tests/testthat, and exporting the function via NAMESPACE.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
tests/testthat/test_plot_predicted_observed.R Added tests checking behavior when both horizon and reference_date are provided and the error when horizon is missing.
man/plot_pred_obs_pointintervals.Rd New documentation for the introduced plotting function.
R/plot_scorable_table.R New function implementation with handling for redundant forecast_date columns, facetting logic, and renaming.
NAMESPACE Export of the new plotting function.
Comments suppressed due to low confidence (2)

tests/testthat/test_plot_predicted_observed.R:102

  • [nitpick] Comparing entire ggplot objects using expect_equal may be brittle. Consider comparing key components (such as the underlying data or layer mappings) to ensure the test focuses on relevant changes.
expect_equal(result, result_manual)

R/plot_scorable_table.R:370

  • [nitpick] The variable name 'qi' is a bit ambiguous. Consider renaming it to a more descriptive name like 'quantile_intervals'.
qi <- quantile_table_to_median_qi(

Comment thread R/plot_scorable_table.R
@dylanhmorris
dylanhmorris merged commit 773ce63 into main Jun 11, 2025
12 checks passed
@dylanhmorris
dylanhmorris deleted the dhm-predicted-actual-pointintervals branch June 11, 2025 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add generic version of predicted actual by horizon plot from pyrenew-hew

3 participants