Rails AB testing with “split” gem: Negative numbers on non-finished…?

[boazadato]

Usage is quite simple:

some_signin_view_file.erb:

<% signin_mode = ab_test( 'log in style','LogIn_ATest','LogIn_BTest' )%>
.
.
Do something according to signin_mode...
and
some_post_signin_controller_file.rb:

finished("log in style", :reset=> FALSE)
Did one simple test with no problems. However, my second test yield negative numbers on the non-finished columns, and only on the first experiment (marked as control).

How can it be negative...? Am i missing something?

[EDIT: remove a non working workaround that was here...]

see also http://stackoverflow.com/questions/9964357/rails-ab-testing-with-split-gem-negative-numbers-on-non-finished/9972546#9972546

[arcanoid]

What happened to me in this case was that when a user clicked refresh on the page it finalized twice, giving it negative numbers.

As I understand the non-finished is calculated as "finished - participants".

[andrew]

When you use the :reset => false option there is nothing to stop the user from completing the same conversion again which currently counts towards the totals.

I suspect the desired behaviour when using :reset => false is that the user should not be able to increment the completion counter any more?

[arcanoid]

For me yes! That would make sense.

[boazadato]

+1

[peppyheppy]

+1

@andrew, I think there should be a new config that specifies if a participant should be counted for multiple conversions. In some cases you would want to be able to see total real conversions, but in many other (funnels for example) I just want to know how many participants got to the goal. Does that make sense?

I might get around to adding this feature if its useful enough.

[andrew]

@peppyheppy I don't think we need another config option for this, if you are resetting the user then they will be counted for multiple conversions, otherwise they should always receive the same alternative and not increment any counters.

[swards]

Has there been progress on this issue?

[swards]

Looking a little further - even with :reset => true, the following line of code is not executing as expected. Line 31 in Split::Helper.rb

session[:split].delete(experiment_name) if options[:reset]

The session experiment is still there after calling finished('test'). One can work around this by resetting the session manually and the completed counts will no longer increment.

[apsoto]

Clarififcation: This comment is about the previous comment where reset => true not reseting things
I've looked at the code and I think the root cause is that it's not being consistent when accessing the values in the session and not removing (reseting) the experiment from the session because it's using the wrong key.

In most places, the code access the session using

ab_user[experiment.key]

But in the finished method is using

session['split'].delete[experiment_name]

Therefore easy to miss, but it should at a minimum do

session['split'].delete(experiment.key)

but even better if it was consistent and did

ab_user.delete(experiment.key)

I'll try to put a patch together in the next few hours

[buddhamagnet]

No further response on this, closing for now.