Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validation time split NOT working #376

Closed
pgg1309 opened this issue Nov 17, 2022 · 2 comments · Fixed by #387
Closed

Validation time split NOT working #376

pgg1309 opened this issue Nov 17, 2022 · 2 comments · Fixed by #387
Labels
bug an unexpected problem or unintended behavior

Comments

@pgg1309
Copy link

pgg1309 commented Nov 17, 2022

Hi,
The validation_time_split() is not working as expected when lag is used.

The calculation seems to be correct but in the end the source code removes the out_id using the function rm_out(split). As a result one gets always the same results for assessment regardless of the lag used.

It seems to be a simple fix, but I do not know how to make a pull request. Would appreciate if you could fix that. Thanks.

@hfrick
Copy link
Member

hfrick commented Nov 22, 2022

thanks for the report @pgg1309 !

library(rsample)
library(dplyr)

toy_data <- data.frame(id = 1:100)
(val_rset <- validation_time_split(toy_data, prob = 75, lag = 2))
#> # Validation Set Split (0.75/0.25)  
#> # A tibble: 1 × 2
#>   splits          id        
#>   <list>          <chr>     
#> 1 <split [75/25]> validation

# internally, this splits it into
# analysis: 1:n_train -> here 1:75
# assessment:  (n_train + 1 - lag):nrow(data) -> here 74:100

# but that's not what remains:
analysis(val_rset$splits[[1]]) %>% pull(id)
#>  [1]  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
#> [26] 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
#> [51] 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75
assessment(val_rset$splits[[1]]) %>% pull(id)
#>  [1]  76  77  78  79  80  81  82  83  84  85  86  87  88  89  90  91  92  93  94
#> [20]  95  96  97  98  99 100

Created on 2022-11-22 with reprex v2.0.2

@hfrick hfrick added the bug an unexpected problem or unintended behavior label Nov 22, 2022
@github-actions
Copy link

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.

@github-actions github-actions bot locked and limited conversation to collaborators Dec 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug an unexpected problem or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants