Are multiple, progressive goals available?

[yjukaku]

I was looking at #109 and it seems like it's been merged into master. From that pull request:

An experiment can have more than one goals. For example, we might want to see how an alternative might affect sign up conversion, as well as purchase conversion. It's possible that an alternative might increase sign-up conversion, but negatively impact purchase conversion.

So that makes it seem like multiple, progressive goals are available. But then if you look at the README, it mentions:

NOTE: This does not mean that a single experiment can have/complete progressive goals.

Good Example: Test if listing Plan A first result in more conversions to Plan A (goal: "plana_conversion") or Plan B (goal: "planb_conversion").

Bad Example: Test if button color increases conversion rate through multiple steps of a funnel. THIS WILL NOT WORK. 

In fact, I am the one who wrote that section in the README. 😆 So what is going on here??

[andrew]

@yjukaku I suspect we forgot to update your part of the readme when we merged #109 but it was a while ago and I can't remember exactly how it works now!

[yjukaku]

Hmm, I don't think so, since #109 was merged in Jan 2013, and I made that commit to the README earlier this year.

I was basing my information on #172 (which was opened about 4 months after #109 was merged). The discussion in #172 implies that one experiment can only have one converted goal, and progressive goals* are not available. It's starting to look like that is false, but it seems to be what a lot of people think.

*To clarify, what I mean by progress goals is something like "Test a blue button vs a red button, to see how many people go to Step 2 of a funnel (first goal), and also how many people go to Step 3 of a funnel (second goal). Then you could compare the effect of a alternative on the entire funnel, instead of just one step of it.

[yjukaku]

Perhaps @coolzilj can enlighten us?

[coolzilj]

@yjukaku Sorry, I can't remember the details since it was a long time ago, but I think progressive goals should works fine though I haven't thought about progressive when I working on this. Maybe you can do some tests.

[losvedir]

Aww, got my hopes up. I can report that, at least on version 0.6.6, progressive goals do not work.

Per the PR comment on #109, in one controller action, I put:

ab_test({"link_color" => ["purchase", "refund"]}, "red", "blue")

In another, I put:

finished({"link_color" => ["purchase"]}, reset: false)

In a third, I put:

finished({"link_color" => ["refund"]}, reset: false)

When I visited the first controller action, it generated an experiment with two goals. When I visited the second controller action, it marked the one goal as "completed" and changed the other goal's "non finished" back to zero. When I visited the third controller action, nothing changed:

In summary: You can't do progressive goals. If you set up multiple goals, as the PR comment states, it will finish the first one you visit. IOW, the README, as currently written, is correct.

[andrew]

Is this something you'd be interested improving on the project? Perhaps adding a spec for the expected behaviour would be a good way to get start.

[benross]

From a user perspective, this would really distinguish split from other ab tests out there. We are almost always interested in a conversion funnel and need at least 2 data points (eg sign ups and upgrades) as multiple goals.

In the meantime I've been trying to come up with a workaround for this. I was thinking we could track the goals either directly to our database or using a third party analytics service (eg tracking an event in google analytics). But to do this I need to access whether the user is in a test, and can't find any documentation on this. Something like a ab_test_value('my_experiment') that returns the current experiment alternative, or nil if the user is not in an experiment. Does this exist in some form?

[losvedir]

@benross - the work around we've been doing is basically setting up multiple experiments, doing ab_test to put someone in one of them, and then, based on the outcome of that, bucketing them in the other experiments with probability 1 or 0. Something like:

bucket = ab_test('xp1', 'a', 'b')
if bucket == 'a'
  ab_test('xp2', {'a' => 1}, {'b' => 0})
else
  ab_test('xp2', {'a' => 0}, {'b' => 1})
end

Someone is then enrolled in both xp1 and xp2 simultaneously, and in either the a or b versions consistently. It's not ideal, but it works.

One thing to watch out for is that the order of hashes is important, and as given above for xp2 the a experiment is "control". You'll need to make sure that all calls to ab_test for xp2 have the same control as the parameter after the experiment name.

[benross]

@losvedir - Brilliant! I had an inkling of doing something like this but was going down a more complicated rabbit hole. Great work around.

And officially offering a beer to anyone who adds this feature to Split :-)

[henrik]

Tried clarifying this limitation in #383.

[benross]

@andrew and @henrik is there a way this can live somewhere as a feature request? In my team's experience this is still the most frustrating part of using the gem :/