Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Historical model variants #128

Open
halfak opened this issue Mar 9, 2016 · 0 comments
Open

Historical model variants #128

halfak opened this issue Mar 9, 2016 · 0 comments

Comments

@halfak
Copy link
Member

halfak commented Mar 9, 2016

I turns out that different modeling strategies produce different ranges of scoring probabilities and other differences in scorer model behavior. In order to not surprise users with such changes, we should allow users to choose to continue to use an old model even after a new one is deployed.

For example, the following URL gets a score for the "primary" model:

/scores/enwiki/damaging/123456789

This could be done explicitly with a variant param.

/scores/enwiki/damaging/123456789?variant=gradient_boosting

We'd need to change the output for when model info is requested so that there can be multiple variants reported.

/scores/enwiki/damaging/ returns:

{
  "linear_svc_balanced": { ...model_info..., "primary": false},
  "gradient_boosting": { ...model_info..., "primary": true}
}

This would also change the way we think about caching scores.  Right now, we a score is stored and retrieved based on a key "<context>:<model>:<version>:<rev_id>".  We'd need to add "variant" to that.  "<context>:<model>:<variant>:<version>:<rev_id>".  This then begs the question -- when we say "model", do we really mean *model*?  We're now generalizing the concept of a "model" to a "modeling problem" -- e.g. "predict when an edit is damaging". 

Under this scheme, we could still make updates to the models by adding new sources of signal and making backwards incompatible changes to `revscoring`, but the overall behavior of each variant should stay relatively consistent. 
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants
@halfak and others