Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Resetting an experiment doesn't reset existing participants #2

Closed
andrew opened this issue Jun 7, 2011 · 4 comments
Closed

Resetting an experiment doesn't reset existing participants #2

andrew opened this issue Jun 7, 2011 · 4 comments

Comments

@andrew
Copy link
Member

andrew commented Jun 7, 2011

If you reset an experiment it rolls the alternative counters back to zero but it doesn't empty the session for users who already have the alternative set in their cookie then the can finish the, now reset, experiment, without their participation being measured.

This could, in the extreme case, cause an infinite conversion rate if 0 users participated but 1 finished!

I suggest we put the version in the cookie and increment a version when resetting the experiment. The more extreme alternative would be to store the users session in redis but that add more overhead.
@ghost ghost assigned semanticart Jun 7, 2011
@matthewford
Copy link
Contributor

This also has knock-on effects with trying to delete an experiment. #3

+1 for keeping the session in redis.

@matthewford
Copy link
Contributor

Changed my mind, ids would be better. Would be also good to add ids to alternatives. When you reset an experiment just inc and use a new id.

Ids just make it easier to work with compared to strings for the keys.

@ghost ghost assigned andrew and semanticart Jun 20, 2011
@andrew
Copy link
Member Author

andrew commented Jun 20, 2011

I've implemented versioning on a branch: https://github.com/andrew/split/tree/versions

Please try it out and let me know if I've missed anything.

@andrew
Copy link
Member Author

andrew commented Jun 26, 2011

I've merged this back into master and it will be in the 0.2.3 release.

@andrew andrew closed this as completed Jun 26, 2011
@josedonizetti josedonizetti mentioned this issue Mar 23, 2012
Closed
stevenou referenced this issue in WhitehawkVentures/split Sep 22, 2015
@stevenou
override only works if user doesn't already have an alternative selec…
dbe2f2b 
…ted, acts as if user was selected into alternative by algorithm

expected behavior:
1. if user is new, override will always be their alternative
2. if user already has alternative, override will do nothing
3. if user selected alternative by override, next time around they will behave like #2 and override will do nothing
@stevenou stevenou mentioned this issue Sep 22, 2015
Closed
@jasonm jasonm mentioned this issue Oct 3, 2011
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@matthewford @andrew @semanticart