Reversing analysis and assessment splits

[mattwarkentin]

Hi,

I use the group_vfold_cv() function a lot, but I have recently been needing to reverse the analysis and assessment splits. For example, lets say you have 3 groups, group_vfold_cv() would create three folds, whereby within each fold 2 groups are used for training, and one is held aside for testing. I would like to do the opposite, where one group is used for training, and the remaining groups are used for testing.

I don't think this is possible using make_splits(), unless I'm just doing it incorrectly. Would you ever consider supporting this resampling scheme? If not, is it possible for me to create these kinds of splits using the existing rsample machinery?

[mattwarkentin]

Figured out how to do it manually, thought it might be worth sharing:

library(tidyverse)
library(rsample)
library(palmerpenguins)

sets <- list(
  list(
    analysis = filter(penguins, species == 'Adelie'),
    assessment = filter(penguins, species != 'Adelie')
  ),
  list(
    analysis = filter(penguins, species == 'Chinstrap'),
    assessment = filter(penguins, species != 'Chinstrap')
  ),
  list(
    analysis = filter(penguins, species == 'Gentoo'),
    assessment = filter(penguins, species != 'Gentoo')
  )
)

map(sets, ~ make_splits(.x[['analysis']], .x[['assessment']])) %>% 
  manual_rset(ids = c('Adelie', 'Chinstrap', 'Gentoo'))
#> # Manual resampling 
#> # A tibble: 3 × 2
#>   splits            id       
#>   <list>            <chr>    
#> 1 <split [152/192]> Adelie   
#> 2 <split [68/276]>  Chinstrap
#> 3 <split [124/220]> Gentoo

[juliasilge]

Thanks for sharing your more manual solution! 🙌 I'm going to leave this issue open to understand how common this need (reverse the splits) is.

[mattwarkentin]

@mikemahoney218 @juliasilge Now that group sampling is on the verge of being supported for many resampling methods via #313, #315, and #316, I am wondering if it might be a good time to revisit this feature/issue.

I had a thought today that perhaps something like reverse_splits() could be an S3 generic that can accept rset or rsplit objects and reverse the analysis and assessment splits? Various S3 methods could be defined for the relevant resampling methods to which this applies. Thoughts?

Something like:

data |>
  group_vfold_cv(group = var) |>
  reverse_splits()

Maybe there isn't enough of a need for this, but it's definitely something I've had to do on several occasions.

[juliasilge]

This makes me think of #79 and the idea of taking an existing rset and doing something to it (in that case, reshuffling it). It could be interesting to think through what kinds of operations folks might need to take on an existing object.

[mattwarkentin]

Agreed! Originally I thought this was something that should be implemented as an argument to the resampling method, but maybe it makes sense to think of various post-hoc procedures that could be done (e.g. reversing, reshuffling, etc.).

[mattwarkentin]

Note to self/someone, for reversing, I think you would only need to develop/maintain methods for rsplit objects and then for rset objects the method would just loop over the relevant split method. Thinking out loud...

While still thinking out loud, this would actually be fairly straightforward to implement by using rsample::populate() to get the out_id and simply replace in_id.

I should also say that I'm very happy to contribute a PR, if this is of interest.

[github-actions]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.