Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix how we check SRM in case of multiple variants #16

Closed
lukasvermeer opened this issue Jan 14, 2020 · 2 comments
Closed

Fix how we check SRM in case of multiple variants #16

lukasvermeer opened this issue Jan 14, 2020 · 2 comments
Assignees
@heinrich333
Labels
enhancement New feature or request

Comments

@lukasvermeer
Copy link
Owner

As discussed with @geoprofi we should:

  • check for SRM on the whole contingency table in one test, then
  • flag SRM for the whole experiment, then
  • let the user figure out manually which variant, if any single one, is the culprit
@lukasvermeer lukasvermeer added the enhancement New feature or request label Jan 14, 2020
@lukasvermeer lukasvermeer mentioned this issue Jan 14, 2020
Open
@lukasvermeer
Copy link
Owner Author

Whoops. I forgot that mentioning "fix #blah" in a commit will mark the issue as closed.

This is not fixed.

@lukasvermeer lukasvermeer reopened this Jan 20, 2020
@heinrich333
Copy link
Collaborator

check for SRM on the whole contingency table in one test, then
flag SRM for the whole experiment

Added in #20 -- feel (very) invited to double check the approach :)

lukasvermeer referenced this issue Jan 26, 2020
@heinrich333 @lukasvermeer
Adding chi-square functionality (#20)
37f1e5a 
* Moving from ztest(all vs. all using jStat) to chi²(table using statistics-distributions.js)

Taken from: https://github.com/lukasvermeer/significance/blob/master/statistics-distributions.js
Source: https://code.google.com/archive/p/statistics-distributions-js/

Initiated based on great feedback we got from @geoprofi -- This change also fixed the (almost certain) SRM alerts we were getting when variants with weights < 1% were present.

Also found a bug preventing the plugin to grab custom weights when the experiment was configured to split evenly and fixed it. (Certain SRM alert when 3 variants were present by defaulting to rounded weights.)

Co-authored-by: Lukas Vermeer <lukas@lukasvermeer.nl>
@lukasvermeer lukasvermeer mentioned this issue Oct 8, 2021
Closed
@lukasvermeer lukasvermeer mentioned this issue Nov 5, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@heinrich333 heinrich333

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lukasvermeer @heinrich333