Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add validation period to synthetic control and interrupted time series #364

Open
drbenvincent opened this issue Jun 20, 2024 · 0 comments · May be fixed by #367
Open

Add validation period to synthetic control and interrupted time series #364

drbenvincent opened this issue Jun 20, 2024 · 0 comments · May be fixed by #367
Assignees
Labels
enhancement New feature or request geo project Related to geo-testing

Comments

@drbenvincent
Copy link
Collaborator

drbenvincent commented Jun 20, 2024

At the moment the model parameters are estimated on the whole pre-intervention period. What we could do instead is to do parameter estimation up to a certain validation window, and produce synthetic control predictions for the validation period as well as the post-intervention period.

I believe that a motivation for this from a Frequentist perspective is possible overfitting of the training period, but this may be less of a concern when doing Bayesian parameter estimation because there can be a degree of regularisation from the priors. Nevertheless, it could be a useful feature to have.

We might want to automatically compute some goodness-of-fit metrics to the validation period.

Could either have a new notebook which covers this functionality or build it in to the exiting synthetic control notebooks.

Screenshot 2024-06-20 at 19 39 54

Inspired by https://netflixtechblog.com/round-2-a-survey-of-causal-inference-applications-at-netflix-fd78328ee0bb?gi=7d795057528e

The changes would be implemented in cp.pymc_experiments.SyntheticControl so that they are specific to this experiment but not the model. It could either be done by adding a kwarg with information about the start of the validation period (for example) or by creating a new class, something like cp.pymc_experiments.SyntheticControlWithValidation. The key point is that these changes related to the experiment, not the model. For example, right now vanilla synthetic control as a model is implemented by cp.pymc_models.WeightedSumFitter. But we would want to be able to have the new validation functionality even if we swap out vanilla synthetic control for augmented or penalised SC.

@drbenvincent drbenvincent added enhancement New feature or request geo project Related to geo-testing labels Jun 20, 2024
@drbenvincent drbenvincent changed the title Add validation period to synthetic control Add validation period to synthetic control and interrupted time series Jun 21, 2024
@drbenvincent drbenvincent self-assigned this Jun 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request geo project Related to geo-testing
Projects
None yet
1 participant