Add `pretty()` method for `validation_set` class

[simonpcouch]

Warning (test-bayes.R:507:3): missing performance values
`validation_split()` was deprecated in rsample 1.2.0.
i Please use `initial_validation_split()` instead.
Backtrace:
    ▆
 1. └─rsample::validation_split(ames, prop = 0.9) at test-bayes.R:507:2

[simonpcouch]

I attempted to transition to the new interface in this test but saw new warnings from [.tbl_df:

library(tidymodels)

data(ames, package = "modeldata")

mod <- parsnip::decision_tree(cost_complexity = tune()) %>%
  parsnip::set_mode("regression")

set.seed(1)
three_way_split <- rsample::initial_validation_split(ames, prop = c(.8, .1))
folds <- rsample::validation_set(three_way_split)

set.seed(1)
res <-
  mod %>%
  tune_bayes(
    Sale_Price ~ Neighborhood + Gr_Liv_Area + Year_Built + Bldg_Type +
      Latitude + Longitude,
    resamples = folds,
    initial = 3,
    metrics = yardstick::metric_set(rsq),
    param_info = parameters(dials::cost_complexity(c(-2, 0)))
  )
#> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by
#> coercion
#> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by
#> coercion
#> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by
#> coercion
#> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by
#> coercion
#> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by
#> coercion
#> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by
#> coercion
#> → A | warning: A correlation computation is required, but `estimate` is constant and has 0 standard deviation, resulting in a divide by 0 error. `NA` will be returned.
#> ! For the rsq estimates, 1 missing value was found and removed before fitting
#>   the Gaussian process model.
#> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by
#> coercion
#> ! For the rsq estimates, 2 missing values were found and removed before
#>   fitting the Gaussian process model.
#> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by
#> coercion
#> ! For the rsq estimates, 3 missing values were found and removed before
#>   fitting the Gaussian process model.
#> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by
#> coercion
#> ! For the rsq estimates, 4 missing values were found and removed before
#>   fitting the Gaussian process model.
#> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by
#> coercion
#> ! For the rsq estimates, 5 missing values were found and removed before
#>   fitting the Gaussian process model.
#> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by
#> coercion
#> ! For the rsq estimates, 6 missing values were found and removed before
#>   fitting the Gaussian process model.
#> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by
#> coercion
#> There were issues with some computations   A: x7
#> 

^{Created on 2023-09-27 with reprex v2.0.2}

Reprex with old interface

library(tidymodels)

data(ames, package = "modeldata")

mod <- parsnip::decision_tree(cost_complexity = tune()) %>%
  parsnip::set_mode("regression")

set.seed(1)
folds <- rsample::validation_split(ames, .9)
#> Warning: `validation_split()` was deprecated in rsample 1.2.0.
#> ℹ Please use `initial_validation_split()` instead.
#> This warning is displayed once every 8 hours.
#> Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
#> generated.

set.seed(1)
res <-
  mod %>%
  tune_bayes(
    Sale_Price ~ Neighborhood + Gr_Liv_Area + Year_Built + Bldg_Type +
      Latitude + Longitude,
    resamples = folds,
    initial = 3,
    metrics = yardstick::metric_set(rsq),
    param_info = parameters(dials::cost_complexity(c(-2, 0)))
  )
#> → A | warning: A correlation computation is required, but `estimate` is constant and has 0 standard deviation, resulting in a divide by 0 error. `NA` will be returned.
#> ! For the rsq estimates, 1 missing value was found and removed before fitting
#>   the Gaussian process model.
#> ! For the rsq estimates, 2 missing values were found and removed before
#>   fitting the Gaussian process model.
#> ! For the rsq estimates, 3 missing values were found and removed before
#>   fitting the Gaussian process model.
#> ! For the rsq estimates, 4 missing values were found and removed before
#>   fitting the Gaussian process model.
#> ! For the rsq estimates, 5 missing values were found and removed before
#>   fitting the Gaussian process model.
#> ! For the rsq estimates, 6 missing values were found and removed before
#>   fitting the Gaussian process model.
#> ! For the rsq estimates, 7 missing values were found and removed before
#>   fitting the Gaussian process model.
#> There were issues with some computations   A: x8
#> 

^{Created on 2023-09-27 with reprex v2.0.2}

Is this behavior expected? cc @hfrick—I think I may have missed that these were new in tidymodels/tune#697.

[hfrick]

hm no, this is not expected. I also don't see that warning with my versions installed. which versions are you using?

(I also don't get that warning with the dev version of tune.)

─ Session info ───────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.3.1 (2023-06-16)
 os       macOS Ventura 13.5.2
 system   aarch64, darwin20
 ui       X11
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       Europe/London
 date     2023-09-28
 pandoc   NA

─ Packages ───────────────────────────────────────────────────────────────────
 package      * version    date (UTC) lib source
 backports      1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
 broom        * 1.0.5      2023-06-09 [1] CRAN (R 4.3.0)
 cachem         1.0.8      2023-05-01 [1] CRAN (R 4.3.1)
 callr          3.7.3      2022-11-02 [1] CRAN (R 4.3.0)
 class          7.3-22     2023-05-03 [2] CRAN (R 4.3.1)
 cli            3.6.1.9000 2023-09-26 [1] Github (r-lib/cli@641fe8c)
 codetools      0.2-19     2023-02-01 [2] CRAN (R 4.3.1)
 colorspace     2.1-0      2023-01-23 [1] CRAN (R 4.3.0)
 crayon         1.5.2      2022-09-29 [1] CRAN (R 4.3.0)
 data.table     1.14.8     2023-02-17 [1] CRAN (R 4.3.1)
 devtools     * 2.4.4      2022-07-20 [1] CRAN (R 4.3.0)
 dials        * 1.2.0      2023-04-03 [1] CRAN (R 4.3.1)
 DiceDesign     1.9        2021-02-13 [1] CRAN (R 4.3.0)
 digest         0.6.33     2023-07-07 [1] CRAN (R 4.3.1)
 dplyr        * 1.1.2      2023-04-20 [1] CRAN (R 4.3.0)
 ellipsis       0.3.2      2021-04-29 [1] CRAN (R 4.3.0)
 fansi          1.0.4      2023-01-22 [1] CRAN (R 4.3.0)
 fastmap        1.1.1      2023-02-24 [1] CRAN (R 4.3.1)
 foreach        1.5.2      2022-02-02 [1] CRAN (R 4.3.0)
 fs             1.6.3      2023-07-20 [1] CRAN (R 4.3.1)
 furrr          0.3.1      2022-08-15 [1] CRAN (R 4.3.1)
 future         1.33.0     2023-07-01 [1] CRAN (R 4.3.1)
 future.apply   1.11.0     2023-05-21 [1] CRAN (R 4.3.1)
 generics       0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
 ggplot2      * 3.4.3      2023-08-14 [1] CRAN (R 4.3.0)
 globals        0.16.2     2022-11-21 [1] CRAN (R 4.3.0)
 glue           1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
 gower          1.0.1      2022-12-22 [1] CRAN (R 4.3.1)
 GPfit          1.0-8      2019-02-08 [1] CRAN (R 4.3.0)
 gtable         0.3.3      2023-03-21 [1] CRAN (R 4.3.1)
 hardhat        1.3.0      2023-03-30 [1] CRAN (R 4.3.0)
 htmltools      0.5.6      2023-08-10 [1] CRAN (R 4.3.0)
 htmlwidgets    1.6.2      2023-03-17 [1] CRAN (R 4.3.0)
 httpuv         1.6.11     2023-05-11 [1] CRAN (R 4.3.0)
 infer        * 1.0.4      2022-12-02 [1] CRAN (R 4.3.1)
 ipred          0.9-14     2023-03-09 [1] CRAN (R 4.3.1)
 iterators      1.0.14     2022-02-05 [1] CRAN (R 4.3.0)
 later          1.3.1      2023-05-02 [1] CRAN (R 4.3.0)
 lattice        0.21-8     2023-04-05 [2] CRAN (R 4.3.1)
 lava           1.7.2.1    2023-02-27 [1] CRAN (R 4.3.1)
 lhs            1.1.6      2022-12-17 [1] CRAN (R 4.3.1)
 lifecycle      1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
 listenv        0.9.0      2022-12-16 [1] CRAN (R 4.3.1)
 lubridate      1.9.2      2023-02-10 [1] CRAN (R 4.3.1)
 magrittr       2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
 MASS           7.3-60     2023-05-04 [2] CRAN (R 4.3.1)
 Matrix         1.6-1      2023-08-14 [1] CRAN (R 4.3.0)
 memoise        2.0.1      2021-11-26 [1] CRAN (R 4.3.0)
 mime           0.12       2021-09-28 [1] CRAN (R 4.3.0)
 miniUI         0.1.1.1    2018-05-18 [1] CRAN (R 4.3.0)
 modeldata    * 1.2.0      2023-08-09 [1] CRAN (R 4.3.0)
 modelenv       0.1.1      2023-03-08 [1] CRAN (R 4.3.1)
 munsell        0.5.0      2018-06-12 [1] CRAN (R 4.3.0)
 nnet           7.3-19     2023-05-03 [2] CRAN (R 4.3.1)
 parallelly     1.36.0     2023-05-26 [1] CRAN (R 4.3.1)
 parsnip      * 1.1.1      2023-08-17 [1] CRAN (R 4.3.1)
 pillar         1.9.0      2023-03-22 [1] CRAN (R 4.3.0)
 pkgbuild       1.3.1      2021-12-20 [1] CRAN (R 4.3.0)
 pkgconfig      2.0.3      2019-09-22 [1] CRAN (R 4.3.0)
 pkgload        1.3.2.1    2023-07-08 [1] CRAN (R 4.3.1)
 prettyunits    1.1.1      2020-01-24 [1] CRAN (R 4.3.0)
 processx       3.8.2      2023-06-30 [1] CRAN (R 4.3.1)
 prodlim        2023.03.31 2023-04-02 [1] CRAN (R 4.3.1)
 profvis        0.3.7      2020-11-02 [1] CRAN (R 4.3.0)
 promises       1.2.1      2023-08-10 [1] CRAN (R 4.3.0)
 prompt         1.0.1      2021-03-12 [1] CRAN (R 4.3.0)
 ps             1.7.5      2023-04-18 [1] CRAN (R 4.3.1)
 purrr        * 1.0.2      2023-08-10 [1] CRAN (R 4.3.0)
 R6             2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
 Rcpp           1.0.11     2023-07-06 [1] CRAN (R 4.3.1)
 recipes      * 1.0.7      2023-08-10 [1] CRAN (R 4.3.0)
 remotes        2.4.2      2021-11-30 [1] CRAN (R 4.3.0)
 rlang          1.1.1.9000 2023-09-26 [1] Github (r-lib/rlang@c55f602)
 rpart        * 4.1.19     2022-10-21 [2] CRAN (R 4.3.1)
 rsample      * 1.2.0      2023-08-23 [1] CRAN (R 4.3.1)
 rstudioapi     0.15.0     2023-07-07 [1] CRAN (R 4.3.1)
 scales       * 1.2.1      2022-08-20 [1] CRAN (R 4.3.0)
 sessioninfo    1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
 shiny          1.7.5      2023-08-12 [1] CRAN (R 4.3.0)
 stringi        1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
 stringr        1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
 survival       3.5-7      2023-08-14 [1] CRAN (R 4.3.0)
 tibble       * 3.2.1      2023-03-20 [1] CRAN (R 4.3.0)
 tidymodels   * 1.1.1      2023-08-24 [1] CRAN (R 4.3.1)
 tidyr        * 1.3.0      2023-01-24 [1] CRAN (R 4.3.0)
 tidyselect     1.2.0      2022-10-10 [1] CRAN (R 4.3.0)
 timechange     0.2.0      2023-01-11 [1] CRAN (R 4.3.1)
 timeDate       4022.108   2023-01-07 [1] CRAN (R 4.3.1)
 tune         * 1.1.2      2023-08-18 [1] Custom
 urlchecker     1.0.1      2021-11-30 [1] CRAN (R 4.3.0)
 usethis      * 2.2.2.9000 2023-07-16 [1] Github (r-lib/usethis@467ff57)
 utf8           1.2.3      2023-01-31 [1] CRAN (R 4.3.0)
 vctrs          0.6.3      2023-06-14 [1] CRAN (R 4.3.0)
 withr          2.5.0      2022-03-03 [1] CRAN (R 4.3.0)
 workflows    * 1.1.3      2023-02-22 [1] CRAN (R 4.3.1)
 workflowsets * 1.0.1      2023-04-06 [1] CRAN (R 4.3.1)
 xtable         1.8-4      2019-04-21 [1] CRAN (R 4.3.0)
 yardstick    * 1.2.0      2023-04-21 [1] CRAN (R 4.3.0)

[simonpcouch]

Hmm, they look very similar. My dplyr is one version off from yours but I was able to downgrade and still see those warnings.

library(tidymodels)

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.1 (2023-06-16)
#>  os       macOS Ventura 13.5.2
#>  system   aarch64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       America/Chicago
#>  date     2023-09-28
#>  pandoc   3.1.1 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package      * version    date (UTC) lib source
#>  backports      1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
#>  broom        * 1.0.5      2023-06-09 [1] CRAN (R 4.3.0)
#>  class          7.3-22     2023-05-03 [2] CRAN (R 4.3.1)
#>  cli            3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
#>  codetools      0.2-19     2023-02-01 [2] CRAN (R 4.3.1)
#>  colorspace     2.1-0      2023-01-23 [1] CRAN (R 4.3.0)
#>  data.table     1.14.8     2023-02-17 [1] CRAN (R 4.3.0)
#>  dials        * 1.2.0      2023-04-03 [1] CRAN (R 4.3.0)
#>  DiceDesign     1.9        2021-02-13 [1] CRAN (R 4.3.0)
#>  digest         0.6.33     2023-07-07 [1] CRAN (R 4.3.0)
#>  dplyr        * 1.1.3      2023-09-03 [1] CRAN (R 4.3.0)
#>  evaluate       0.21       2023-05-05 [1] CRAN (R 4.3.0)
#>  fansi          1.0.4      2023-01-22 [1] CRAN (R 4.3.0)
#>  fastmap        1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
#>  foreach        1.5.2      2022-02-02 [1] CRAN (R 4.3.0)
#>  fs             1.6.3      2023-07-20 [1] CRAN (R 4.3.0)
#>  furrr          0.3.1      2022-08-15 [1] CRAN (R 4.3.0)
#>  future         1.33.0     2023-07-01 [1] CRAN (R 4.3.0)
#>  future.apply   1.11.0     2023-05-21 [1] CRAN (R 4.3.0)
#>  generics       0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
#>  ggplot2      * 3.4.3      2023-08-14 [1] CRAN (R 4.3.0)
#>  globals        0.16.2     2022-11-21 [1] CRAN (R 4.3.0)
#>  glue           1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
#>  gower          1.0.1      2022-12-22 [1] CRAN (R 4.3.0)
#>  GPfit          1.0-8      2019-02-08 [1] CRAN (R 4.3.0)
#>  gtable         0.3.4      2023-08-21 [1] CRAN (R 4.3.0)
#>  hardhat        1.3.0      2023-03-30 [1] CRAN (R 4.3.0)
#>  htmltools      0.5.6      2023-08-10 [1] CRAN (R 4.3.0)
#>  infer        * 1.0.5      2023-09-06 [1] CRAN (R 4.3.0)
#>  ipred          0.9-14     2023-03-09 [1] CRAN (R 4.3.0)
#>  iterators      1.0.14     2022-02-05 [1] CRAN (R 4.3.0)
#>  knitr          1.44       2023-09-11 [1] CRAN (R 4.3.0)
#>  lattice        0.21-8     2023-04-05 [2] CRAN (R 4.3.1)
#>  lava           1.7.2.1    2023-02-27 [1] CRAN (R 4.3.0)
#>  lhs            1.1.6      2022-12-17 [1] CRAN (R 4.3.0)
#>  lifecycle      1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
#>  listenv        0.9.0      2022-12-16 [1] CRAN (R 4.3.0)
#>  lubridate      1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
#>  magrittr       2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
#>  MASS           7.3-60     2023-05-04 [2] CRAN (R 4.3.1)
#>  Matrix         1.6-1.1    2023-09-18 [1] CRAN (R 4.3.1)
#>  modeldata    * 1.2.0      2023-08-09 [1] CRAN (R 4.3.0)
#>  munsell        0.5.0      2018-06-12 [1] CRAN (R 4.3.0)
#>  nnet           7.3-19     2023-05-03 [2] CRAN (R 4.3.1)
#>  parallelly     1.36.0     2023-05-26 [1] CRAN (R 4.3.0)
#>  parsnip      * 1.1.1      2023-08-17 [1] CRAN (R 4.3.0)
#>  pillar         1.9.0      2023-03-22 [1] CRAN (R 4.3.0)
#>  pkgconfig      2.0.3      2019-09-22 [1] CRAN (R 4.3.0)
#>  prodlim        2023.08.28 2023-08-28 [1] CRAN (R 4.3.0)
#>  purrr        * 1.0.2      2023-08-10 [1] CRAN (R 4.3.0)
#>  R6             2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
#>  Rcpp           1.0.11     2023-07-06 [1] CRAN (R 4.3.0)
#>  recipes      * 1.0.8      2023-08-25 [1] CRAN (R 4.3.0)
#>  reprex         2.0.2      2022-08-17 [1] CRAN (R 4.3.0)
#>  rlang          1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
#>  rmarkdown      2.25       2023-09-18 [1] CRAN (R 4.3.1)
#>  rpart          4.1.19     2022-10-21 [2] CRAN (R 4.3.1)
#>  rsample      * 1.2.0      2023-08-23 [1] CRAN (R 4.3.0)
#>  rstudioapi     0.15.0     2023-07-07 [1] CRAN (R 4.3.0)
#>  scales       * 1.2.1      2022-08-20 [1] CRAN (R 4.3.0)
#>  sessioninfo    1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
#>  survival       3.5-7      2023-08-14 [1] CRAN (R 4.3.0)
#>  tibble       * 3.2.1      2023-03-20 [1] CRAN (R 4.3.0)
#>  tidymodels   * 1.1.1      2023-08-24 [1] CRAN (R 4.3.0)
#>  tidyr        * 1.3.0      2023-01-24 [1] CRAN (R 4.3.0)
#>  tidyselect     1.2.0      2022-10-10 [1] CRAN (R 4.3.0)
#>  timechange     0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
#>  timeDate       4022.108   2023-01-07 [1] CRAN (R 4.3.0)
#>  tune         * 1.1.2      2023-08-23 [1] CRAN (R 4.3.0)
#>  utf8           1.2.3      2023-01-31 [1] CRAN (R 4.3.0)
#>  vctrs          0.6.3      2023-06-14 [1] CRAN (R 4.3.0)
#>  withr          2.5.0      2022-03-03 [1] CRAN (R 4.3.0)
#>  workflows    * 1.1.3      2023-02-22 [1] CRAN (R 4.3.0)
#>  workflowsets * 1.0.1      2023-04-06 [1] CRAN (R 4.3.0)
#>  xfun           0.40       2023-08-09 [1] CRAN (R 4.3.0)
#>  yaml           2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
#>  yardstick    * 1.2.0      2023-04-21 [1] CRAN (R 4.3.0)
#> 
#>  [1] /Users/simoncouch/Library/R/arm64/4.3/library
#>  [2] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

^{Created on 2023-09-28 with reprex v2.0.2}

[simonpcouch]

Looks like they were present in the original issue tidymodels/tune#697 which gives us a slightly more minimal reprex:

library(tidymodels) 

val_split <- initial_validation_split(mtcars) 
val_set <- validation_set(val_split) 

# fit_resamples() and tune_*() work with rsets
res <- fit_resamples(linear_reg(), mpg ~ ., resamples = val_set)
#> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by
#> coercion

^{Created on 2023-09-28 with reprex v2.0.2}

[simonpcouch]

Ah, okay. tune::pull_rset_attributes() calls pretty() on the inputted resamples, giving:

library(tidymodels) 

val_split <- initial_validation_split(mtcars) 
val_set <- validation_set(val_split) 

pretty(val_set)
#> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by
#> coercion
#> # A tibble: 0 × 0

# previously...
val_set_old <- validation_split(mtcars)
#> Warning: `validation_split()` was deprecated in rsample 1.2.0.
#> ℹ Please use `initial_validation_split()` instead.
#> This warning is displayed once every 8 hours.
#> Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
#> generated.
pretty(val_set_old)
#> [1] "Validation Set Split (0.75/0.25) "

^{Created on 2023-09-28 with reprex v2.0.2}

[hfrick]

Thanks for finding this! So validation_set (as opposed to validation_split) needs its own pretty() method. That should go into rsample so I'm moving the issue there.

[github-actions]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.