Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/pkgsite: experiment rollout imprecise and unstable #40314

Closed
abursavich opened this issue Jul 20, 2020 · 1 comment
Closed

x/pkgsite: experiment rollout imprecise and unstable #40314

abursavich opened this issue Jul 20, 2020 · 1 comment

Comments

@abursavich
Copy link

@abursavich abursavich commented Jul 20, 2020

What is the URL of the page with the issue?

N/A

What is your user agent?

N/A

Screenshot

N/A

What did you do?

Read the experiments code.

What did you expect to see?

Experiments that rollout with precision and per user stability.

What did you see instead?

The chosen arithmetic uint(h.Sum32())%(100/e.Rollout) == 0 has two problems.

  • The integer division of 100/e.Rollout greatly reduces the precision available (consider that e.Rollout = 51 actually results in 100% rollout).
  • Varying the modulus divisor with the rollout threshold shuffles users in and out of an experiment rather than making a user enter and remain in the experiment once at a given threshold.
@gopherbot gopherbot added this to the Unreleased milestone Jul 20, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 20, 2020

Change https://golang.org/cl/243659 mentions this issue: internal/middleware: fix experiment assignment precision and stability

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants