Skip to content

Create an empty calibration set instead of erroring - #582

Merged
hfrick merged 17 commits into
mainfrom
mock-inner-split
Jul 25, 2025
Merged

Create an empty calibration set instead of erroring#582
hfrick merged 17 commits into
mainfrom
mock-inner-split

Conversation

@hfrick

@hfrick hfrick commented Jul 23, 2025

Copy link
Copy Markdown
Member

closes #570

This PR adds a mock calibration split which returns the (outer) analysis set as the (inner) analysis set and an empty, i.e., 0-row calibration set. This is used when the calibration split isn't possible, e.g., if there is only one row in the analysis set.

Below is an example with the Chicago data. A lookback = 0 means we only have one period in the analysis set and thus can't split further using the principles we settled on. This is the slightly less obvious case corresponding to "only one row available to split".

library(rsample)
data(Chicago, package = "modeldata")

chicago_r_set <- sliding_period(
  Chicago,
  index = "date",
  period = "month",
  lookback = 0,
  assess_stop = 1,
  step = 2
)

chicago_r_split <- get_rsplit(chicago_r_set, 1)

i_split <- inner_split(chicago_r_split, .get_split_args(chicago_r_set))
#> Warning: Cannot create calibration split; creating an empty calibration set.

# empty calibration set
assessment(i_split)
#> # A tibble: 0 × 50
#> # ℹ 50 variables: ridership <dbl>, Austin <dbl>, Quincy_Wells <dbl>,
#> #   Belmont <dbl>, Archer_35th <dbl>, Oak_Park <dbl>, Western <dbl>,
#> #   Clark_Lake <dbl>, Clinton <dbl>, Merchandise_Mart <dbl>, Irving_Park <dbl>,
#> #   Washington_Wells <dbl>, Harlem <dbl>, Monroe <dbl>, Polk <dbl>,
#> #   Ashland <dbl>, Kedzie <dbl>, Addison <dbl>, Jefferson_Park <dbl>,
#> #   Montrose <dbl>, California <dbl>, temp_min <dbl>, temp <dbl>,
#> #   temp_max <dbl>, temp_change <dbl>, dew <dbl>, humidity <dbl>, …

Created on 2025-07-23 with reprex v2.1.1

@hfrick
hfrick requested a review from topepo July 23, 2025 15:59
@hfrick
hfrick requested a review from Copilot July 24, 2025 11:01

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements a fallback mechanism for calibration splits that creates empty calibration sets instead of throwing errors when splits are not possible. This addresses cases where there are insufficient observations to create both an analysis set and a calibration set, particularly in time series and small dataset scenarios.

Key changes include:

  • Wrapping split creation logic in try/catch blocks to handle failures gracefully
  • Adding a new mock_internal_calibration_split() function to create empty calibration sets
  • Converting error conditions to warnings with informative messages
  • Comprehensive test coverage for all split types with mock split scenarios

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
R/inner_split.R Core implementation of fallback logic across all split methods and new mock split function
tests/testthat/test-inner_split.R Extensive test coverage for mock split creation across all split types
tests/testthat/_snaps/inner_split.md Updated test snapshots showing warnings instead of errors
R/misc.R Added class parameter to make_splits.data.frame method
man/make_splits.Rd Updated documentation for new class parameter
inst/calibration-sets.Rmd Documentation updates reflecting new behavior
NEWS.md Added PR reference to changelog
DESCRIPTION Version bump
Comments suppressed due to low confidence (1)

tests/testthat/test-inner_split.R:1205

  • [nitpick] The test name is too generic. Consider a more descriptive name like 'mock_internal_calibration_split creates proper mock split structure' to clearly indicate what functionality is being tested.
test_that("can create a mock split", {

Comment thread tests/testthat/test-inner_split.R
Comment thread R/inner_split.R Outdated
Comment thread R/inner_split.R Outdated
Comment thread R/inner_split.R Outdated
@hfrick
hfrick merged commit 661c122 into main Jul 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

What to do when we can't make a calibration split

3 participants