A/B test is not tracking alternatives

[akahn]

Hi,

I'm having an issue where Vanity is successfully tracking a metric, but not giving me numbers for the alternatives. This screenshot shows the problem. Out of 1,245 participants, 0.0% for each alternative. But I know that people have signed up. 16, in fact. Looking at my data in Redis, vanity:1:signup_text_size:alts:0:conversions is 10 and vanity:1:signup_text_size:alts:1:conversions is 6.

Here's my a/b test:
ab_test "Signup text size" do
description "Does increasing the size of the signup text increase signups"
alternatives "big", "normal"
metrics :signup
end

Here's the metric:
metric "Signup" do
description "Measures how many people signed up"
end

Here's UsersController#create:
def create
@user = User.new(params[:user])
if @user.save
track! :signup
flash[:message] = "Thank you for signing up! You are now logged in."
redirect_back_or_default account_url
else
render :action => :new
end
end

And then in the layout template I have:
%ul{:class => ab_test(:signup_text_size)}

Any idea what I'm doing wrong here? I can verify that folks are seeing each alternative. Vanity is clearly tracking the metric. Why is there no percentage for each alternative?

Thanks,
Alex

[mrmemes-eth]

I'm having the same issue on an app we're integrating Vanity into, with avery similar setup. The X-Factor I think might play into things is Authlogic
and it's session maintenance. The vanity_id cookie token appears to remain
unchanged even after Authlogic has gone through it's session maintenance
during user creation. However, it still fails to correlate an experiment
participant with it's subsequent conversion, so something (presumably related to
the session's modification) must have been affected.

In the interim, I've found an apparent workaround (using akahn's
UsersController#create as an example):

def create
  @user = User.new(params[:user])
  if @user.save_without_session_maintenance
    track! :signup
    UserSession.create(@user)
    flash[:message] = "Thank you for signing up! You are now logged in."
    redirect_back_or_default account_url
  else  
    render :action => :new
  end
end

This defers the transformation of the session for long enough to have Vanity
successfully correllate the experiment and conversion.

[akahn]

Interesting, thanks for that workaround. I'll give that a try.

[sdhull]

The problem is definitely with the vanity_identity. If you are trying to track things that start as an unregistered user and convert as a registered user, then the identity changes from the hex that is assigned to the vanity_id cookie to current_user.id. If you try to track a conversion with an identity that hasn't previously been added as an experiment participant, it will fail silently.

Honestly I'm very surprised you're having this issue, as we seem to be having the opposite issue at my office. Somehow, somewhere, vanity_identity is being called before ab_test is called in our UsersController. vanity_identity memoizes the identity, so even though we call ab_test after current_user returns a valid user object, the hex has already been memoized as the vanity_identity.

I can post some code if you like.

[sdhull]

Hah actually I just discovered that our problem is because of my office's optimization to avoid hitting our DB if the user doesn't have any user_credentials cookie yet (as most of our traffic is first-time visitors).

def current_user
  return nil if cookies[:user_credentials].blank?
  @current_user ||= current_user_session && current_user_session.record
end

Anyway your problem is still a very valid one, and I think that Vanity should transparently convert a user's recorded identity in Redis from what was stored in the cookie to user.id (when it detects that the user has the cookie AND a user object).

Otherwise, vanity_identity could simply check for a cookie first, use that if it's there, then fall back to using the user's id. That seems like maybe the simplest solution.

-Steve

[wmwong]

I agree that there should be some sort of handling of the cookie's vanity_id and the user.id (or any other id we wish to use). I'm not sure simply replacing the identity in the database will work. There needs to be some sort of an alias method. KISSmetrics does this well. You can use several identifiers for the same person by aliasing them together. Whether their identity comes from the vanity_id in the cookie or user.id, they are the same person. They should be treated as the same person.

I can imagine having an experiment occur on a non-logged in page, then tracking a conversion when the person is logged in. Currently, this will not be tracked as shown by the OP. They are seen as 2 different people.

This seems like something people assume Vanity will take care of, but it doesn't. Many people may see errors in their statistics and not know why as this is not posted clearly somewhere. I accidentally came across this bug and now have a trade-off to make which I wasn't aware of before.

Wesley

[tomblomfield-zz]

Hmmm. I'd imagine this is still a fairly big issue for people. Including me.

I may look at it shortly.

[phillbaker]

Closing as this is fixed in #49, will centralize comments there. Also related to #140 .