For discussion: fake 'A/B' test with identical variations (a=b) returns B outperforming A by +28%, and with 99% confidence !

[feliperaul]

I created fake a/b that has literally no code inside it. It just runs on my landing page to trigger Split tracking, and I finish the experiment when someone signs up.

The code is literally just this:

<% ab_test "Pricing_page_conversion", "a", "b" do |style| %>
<% end %>

So, A=B.

Imagine my surprise when I saw this today on my split dashboard:

I'm in love with Split and I think it's one of the most important gems we use, but I'm posting this so that we can discuss a little bit about the math of the test so that we can understand how a 99% confidence can emerge, with such low numbers, when A=B.

I decided to take this numbers to a couple of A/B test calculators easily found on google (first results).

On this site, http://www.usereffect.com/split-test-calculator, it tells me I haven't even reached 90% significance, and I need 131 more visitors to do so.

Sorry, you have no clear winner
We estimate that you'll need 131 more visitors*

Group A:  6.04% conversion
Group B:  2.76% conversion

     Visitors:   Goals:
 Group A:       
 Group B:       

*Based on the current trend of your results, your test would require 131 additional visitors to reach significance at the 90% confidence level. Please note that this is at best a loose estimate, designed to help you decide whether or not to continue the test.

On this other site (http://www.abtester.com/calculator/), it tells me I have only 91.61% confidence with these numbers.

Even if I multiply them by two, I don't reach 99% confidence yet:

Last, I ran it on Visual Website Optimezer calculator, and here are the results (no significance as well):

I'd love to hear your thoughts.

[rceee]

I'm interested in this as well.

[andrew]

You're right, that does seem a bit off, the z-score calculation hasn't been looked at since we hacked it out at railsconf a couple years ago so I suspect it's time for a revamp, do you have any suggestions for a better algo to use?

[feliperaul]

Hi Andrew,

I wish I could help you more on this. I have very limited knowledge in programming since my work is entirely front-end related :(

What I did is start looking on the internet about really good articles about the topic at hand.

I found Evan Miller, she wrote these pieces:

http://www.evanmiller.org/how-not-to-run-an-ab-test.html

http://www.evanmiller.org/lazy-assignment-and-ab-testing.html

I also shoot them an e-mail regarding the 'Split' gem.

[rceee]

Two articles on z-index etc i found helpful:

http://visualwebsiteoptimizer.com/split-testing-blog/what-you-really-need-to-know-about-mathematics-of-ab-split-testing/

http://blog.rivalhost.com/how-bad-spiders-and-bots-inflate-google-analytics/ (more about bots affecting analytics/split tests etc)

[feliperaul]

Evan miller replied my email.

You guys probably want a chi-squared test, see here:

http://www.evanmiller.org/statistical-formulas-for-programmers.html#prop_chi

For a 2x2 test like you have, the z-score is the square root of the X^2 statistic.

Hope that helps,

Evan

[hakunin]

This is serious, if you're developing for a few months with A/B test due to this issue, it will leave you in a whole another place if you just go by the A/B result.

I'd really like to see this fixed..

[andrew]

I'd like to get this fixed as well, I'm just snowed under with work at the moment, if someone wants to give it a try and send a pull request that would be a massive help.

[nfm]

I'd like to take a look at this too but am a bit pressed for time at the moment.

Here's some code from another gem that might be helpful: https://github.com/livingsocial/abanalyzer/blob/master/lib/abanalyzer/abtest.rb

[feliperaul]

Guys, it seems caser's Pull request #221 was merged into the main branch already, and this could be closed.

[feliperaul]

Decided to give it another try.

This is the code

<% ab_test('statistical_validity','A','B') do |block| %> <% end %>

So A == B

This is the result. 95% confidence that one is better then the other =/

Split version v1.2.0

[nfm]

@feliperaul I punched those numbers into Thumbtack's split test calculator and got the same result: https://www.thumbtack.com/labs/abba/#Baseline=11204%2C88818&Variation+1=11568%2C89193&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true

It calculates the p-value as 0.025, which is > 95% confidence. I haven't checked this against any other calculators, but I assume they would return similar results.

At 95% confidence, 5% of your tests will see a false positive result - I think this test probably just falls into that category.

I think your other comments about how Split handles small sample sizes might have been addressed by #221, but it's possible there are still issues to tackle there.