Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vignette: Use LOO CV with validate_search = FALSE instead of K-fold CV #305

Merged
merged 5 commits into from
May 5, 2022

Conversation

fweber144
Copy link
Collaborator

This makes the vignette use cv_varsel() with LOO CV and validate_search = FALSE instead of K-fold CV. This provides a speed improvement when building the vignette and should reflect the typical projpred workflow better (running cv_varsel() with validate_search = FALSE first for rough results and only later running cv_varsel() with validate_search = TRUE). Of course, this causes the vignette to show an unvalidated search, but this is emphasized in the text, so I think that should be ok.

@avehtari: Do you want to take a look at this? You don't need to, but you contributed some helpful comments on the last vignette PR (#237). I added you as reviewer, but could remove you again.

`validate_search = FALSE` instead of K-fold CV. This should also better reflect
the projpred workflow (running `cv_varsel()` with `validate_search = FALSE`
first for rough results and only later running `cv_varsel()` with
`validate_search = TRUE`).
@fweber144 fweber144 requested a review from avehtari April 23, 2022 13:27
@avehtari
Copy link
Collaborator

avehtari commented May 5, 2022

After running cv_varsel with validate_search = FALSE I would add something like

we directly plot with deltas = TRUE here (note that as the validate_search = FALSE this result is slightly optimistic, and the plot looks different when validate_search = TRUE is used):

close enough to the reference model's performance (this is also affected by validate_search = FALSE)

close enough to the reference model's performance (this is also affected by validate_search = FALSE)

same final submodel size (6) as by our manual decision (suggest_size() is also affected by validate_search = FALSE)

@avehtari
Copy link
Collaborator

avehtari commented May 5, 2022

Otherwise looks good

@fweber144 fweber144 merged commit cbba485 into stan-dev:master May 5, 2022
@fweber144 fweber144 deleted the vignette_loo branch May 5, 2022 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants