Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

catch and log extract issues #575

Merged
merged 8 commits into from
Nov 11, 2022
Merged

catch and log extract issues #575

merged 8 commits into from
Nov 11, 2022

Conversation

simonpcouch
Copy link
Contributor

@simonpcouch simonpcouch commented Nov 4, 2022

With CRAN tune, the package hides errors that occur with mis-specified extract functions:

library(tidymodels)

wf <- workflow(mpg ~ ., linear_reg())
set.seed(1)
boots <- bootstraps(mtcars, 3)

# warn at extract --------------------------------------------------------------
raise_warning <- function(x) {warning("AHHH"); TRUE}

res_extract_warn <-
  fit_resamples(
    wf, boots, control = control_resamples(extract = raise_warning)
  )
#> ! Bootstrap1: internal: AHHH
#> ! Bootstrap2: internal: AHHH
#> ! Bootstrap3: internal: AHHH

res_extract_warn
#> # Resampling results
#> # Bootstrap sampling 
#> # A tibble: 3 × 5
#>   splits          id         .metrics         .notes           .extracts       
#>   <list>          <chr>      <list>           <list>           <list>          
#> 1 <split [32/13]> Bootstrap1 <tibble [2 × 4]> <tibble [1 × 3]> <tibble [1 × 2]>
#> 2 <split [32/17]> Bootstrap2 <tibble [2 × 4]> <tibble [1 × 3]> <tibble [1 × 2]>
#> 3 <split [32/11]> Bootstrap3 <tibble [2 × 4]> <tibble [1 × 3]> <tibble [1 × 2]>
#> 
#> There were issues with some computations:
#> 
#>   - Warning(s) x3: AHHH
#> 
#> Run `show_notes(.Last.tune.result)` for more information.

# error at extract -------------------------------------------------------------
raise_error <- function(x) {stop("AHHH"); TRUE}

res_extract_error <-
  fit_resamples(
    wf, boots, control = control_resamples(extract = raise_error)
  )

res_extract_error
#> # Resampling results
#> # Bootstrap sampling 
#> # A tibble: 3 × 5
#>   splits          id         .metrics         .notes           .extracts       
#>   <list>          <chr>      <list>           <list>           <list>          
#> 1 <split [32/13]> Bootstrap1 <tibble [2 × 4]> <tibble [0 × 3]> <tibble [1 × 2]>
#> 2 <split [32/17]> Bootstrap2 <tibble [2 × 4]> <tibble [0 × 3]> <tibble [1 × 2]>
#> 3 <split [32/11]> Bootstrap3 <tibble [2 × 4]> <tibble [0 × 3]> <tibble [1 × 2]>

res_extract_error$.extracts[[1]]$.extracts
#> [[1]]
#> [1] "Error in extractor(object) : AHHH\n"
#> attr(,"class")
#> [1] "try-error"
#> attr(,"condition")
#> <simpleError in extractor(object): AHHH>

Created on 2022-11-04 with reprex v2.0.2

With this PR, the package surfaces extraction errors in the same way that it does errors that occur elsewhere:

library(tidymodels)

wf <- workflow(mpg ~ ., linear_reg())
set.seed(1)
boots <- bootstraps(mtcars, 3)

# warn at extract --------------------------------------------------------------
raise_warning <- function(x) {warning("AHHH"); TRUE}

res_extract_warn <-
  fit_resamples(
    wf, boots, control = control_resamples(extract = raise_warning)
  )
#> ! Bootstrap1: preprocessor 1/1, model 1/1 (extracts): AHHH
#> ! Bootstrap2: preprocessor 1/1, model 1/1 (extracts): AHHH
#> ! Bootstrap3: preprocessor 1/1, model 1/1 (extracts): AHHH

res_extract_warn
#> # Resampling results
#> # Bootstrap sampling 
#> # A tibble: 3 × 5
#>   splits          id         .metrics         .notes           .extracts       
#>   <list>          <chr>      <list>           <list>           <list>          
#> 1 <split [32/13]> Bootstrap1 <tibble [2 × 4]> <tibble [1 × 3]> <tibble [1 × 2]>
#> 2 <split [32/17]> Bootstrap2 <tibble [2 × 4]> <tibble [1 × 3]> <tibble [1 × 2]>
#> 3 <split [32/11]> Bootstrap3 <tibble [2 × 4]> <tibble [1 × 3]> <tibble [1 × 2]>
#> 
#> There were issues with some computations:
#> 
#>   - Warning(s) x3: AHHH
#> 
#> Run `show_notes(.Last.tune.result)` for more information.

# error at extract -------------------------------------------------------------
raise_error <- function(x) {stop("AHHH"); TRUE}

res_extract_error <-
  fit_resamples(
    wf, boots, control = control_resamples(extract = raise_error)
  )
#> x Bootstrap1: preprocessor 1/1, model 1/1 (extracts): Error in extractor(object): AHHH
#> x Bootstrap2: preprocessor 1/1, model 1/1 (extracts): Error in extractor(object): AHHH
#> x Bootstrap3: preprocessor 1/1, model 1/1 (extracts): Error in extractor(object): AHHH

res_extract_error
#> # Resampling results
#> # Bootstrap sampling 
#> # A tibble: 3 × 5
#>   splits          id         .metrics         .notes           .extracts
#>   <list>          <chr>      <list>           <list>           <list>   
#> 1 <split [32/13]> Bootstrap1 <tibble [2 × 4]> <tibble [1 × 3]> <tibble [1 × 2]>
#> 2 <split [32/17]> Bootstrap2 <tibble [2 × 4]> <tibble [1 × 3]> <tibble [1 × 2]>   
#> 3 <split [32/11]> Bootstrap3 <tibble [2 × 4]> <tibble [1 × 3]> <tibble [1 × 2]>   
#> 
#> There were issues with some computations:
#> 
#>   - Error(s) x3: Error in extractor(object): AHHH
#> 
#> Run `show_notes(.Last.tune.result)` for more information.

# both warn and error at extract
raise_both <- function(x) {warning("AH"); stop("AHHH"); TRUE}

res_extract_both <-
  fit_resamples(
    wf, boots, control = control_resamples(extract = raise_both)
  )
#> ! Bootstrap1: preprocessor 1/1, model 1/1 (extracts): AH
#> x Bootstrap1: preprocessor 1/1, model 1/1 (extracts): Error in extractor(object): AHHH
#> ! Bootstrap2: preprocessor 1/1, model 1/1 (extracts): AH
#> x Bootstrap2: preprocessor 1/1, model 1/1 (extracts): Error in extractor(object): AHHH
#> ! Bootstrap3: preprocessor 1/1, model 1/1 (extracts): AH
#> x Bootstrap3: preprocessor 1/1, model 1/1 (extracts): Error in extractor(object): AHHH

res_extract_both
#> # Resampling results
#> # Bootstrap sampling 
#> # A tibble: 3 × 5
#>   splits          id         .metrics         .notes           .extracts
#>   <list>          <chr>      <list>           <list>           <list>   
#> 1 <split [32/13]> Bootstrap1 <tibble [2 × 4]> <tibble [2 × 3]> <tibble [1 × 2]>   
#> 2 <split [32/17]> Bootstrap2 <tibble [2 × 4]> <tibble [2 × 3]> <tibble [1 × 2]>   
#> 3 <split [32/11]> Bootstrap3 <tibble [2 × 4]> <tibble [2 × 3]> <tibble [1 × 2]>   
#> 
#> There were issues with some computations:
#> 
#>   - Error(s) x3: Error in extractor(object): AHHH
#>   - Warning(s) x3: AH
#> 
#> Run `show_notes(.Last.tune.result)` for more information.

Created on 2022-11-04 with reprex v2.0.2

Fixes #565, to the extent that we will. :)

@@ -66,6 +66,7 @@ summarize_notes <- function(x) {
dplyr::group_nest(type) %>%
dplyr::mutate(data = purrr::map(data, ~ dplyr::count(.x, note))) %>%
tidyr::unnest(data) %>%
dplyr::rowwise() %>%
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file fix a bug in summarize_notes() where the condition messages for warnings and errors were pasted if there were multiple distinct notes. I indulged myself and went ahead and made this fix so that we could snapshot test that case well, but committed those changes separately so that we could document what that test output looks like without those changes.

Message
x Bootstrap1: preprocessor 1/1, model 1/1 (extracts): Error in extractor(object): AHHH
x Bootstrap2: preprocessor 1/1, model 1/1 (extracts): Error in extractor(object): AHHH
x Bootstrap3: preprocessor 1/1, model 1/1 (extracts): Error in extractor(object): AHHH
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to make these Error in extractor(object) bits of the error messages more informative, but that would require changes to more general machinery (and many snapshot updates) that would be better off living in a separate PR.

R/pull.R Outdated Show resolved Hide resolved
R/pull.R Outdated
)
)
extracts <- dplyr::bind_cols(grid, labels(split))
extracts$.extracts <- list(extract_details(workflow, ctrl$extract))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mutate() wraps the raised errors with information that, when collapsed by tune's machinery, obscures the condition messages raised from the extractor functions themselves.

DavisVaughan and others added 4 commits November 10, 2022 12:39
* delete unneeded `workflow` argument
* restores former `pulley` behavior
* correct spelling of `control` argument
Copy link
Member

@DavisVaughan DavisVaughan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

NEWS.md Outdated Show resolved Hide resolved
Co-authored-by: Davis Vaughan <davis@rstudio.com>
@simonpcouch simonpcouch merged commit 40e3328 into main Nov 11, 2022
@simonpcouch simonpcouch deleted the log-extract2 branch November 11, 2022 00:35
@github-actions
Copy link

This pull request has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.

@github-actions github-actions bot locked and limited conversation to collaborators Nov 25, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

balancing verbosity for errors and warnings
2 participants