Allows An Experiment To Have Multiple Goals

[coolzilj]

This feature comes with pretty substantial changes. Some of those changes possibly kludgy. Please give some feedback : )

Backward compatibility is maintained.

An experiment can have more than one goals. For example, we might want to see how an alternative might affect sign up conversion, as well as purchase conversion. It's possible that an alternative might increase sign-up conversion, but negatively impact purchase conversion.

The API to define goals for an experiment is this:

ab_test({"link_color" => ["purchase", "refund"]}, "red", "blue")

The name of the experiment is link_color, and it has two additional goals: purchase and refund.

To complete a goal conversion, you do it like:

finished({"link_color" => ["purchase"]})

I also thought about a different API design to specify to completion goal using the option, but not sure if it's better.

finished("link_color",:goal => "purchase")

Thoughts?

[andrew]

I'll take a look through it in a few days (first day back at work today), should be fun merging with this: #105

[coolzilj]

#105 is pretty epic.

A lot of the changes to spec is because I changed the method signature for Split::Experiment.find_or_create to not use splat, so it can take goals as the third argument.

Hmmm... maybe I can work around that by sneaking in goals along with experiment name by using a hash {name => goals}

I'll see what I can do to minimize diff to help with the merging.

[andrew]

Thanks!

[feliperaul]

I started using split not longer then 2 weeks ago and I'm loving how awesome it is.

Then I just came here, this second, to post a question if it was possible to track 2 or more metrics with the same experiment, and I see this waiting right in the top of the issue list ... what a coincidence !

[iangreenleaf]

I'm excited to see this ticket, it's a feature I've been wanting since I started using Split.

Let me run one more idea for the API past you: I've always wanted completions to be independent of specific experiments. For example, I'd like to be able to do finished(:email_clickthrough) or finished(:signup) and have it complete any experiments/goals that are using that "metric". This would mesh great with what you've done - ab_test will stay the same, and finished will once again take a single string/symbol of either an experiment name or a goal name.

The only problem with compatibility is there could be a small overlap if someone creates a goal with the same name as an older experiment. That seems easily avoided, though, and easily fixed if anyone runs into it.

[coolzilj]

@iangreenleaf Hmmm... That sounds greate although I don't need the feature right now. I'll see what I can do.

[coolzilj]

@feliperaul What a coincidence! I started using split not longer then 2 weeks ago too. :)

[andrew]

@coolzilj can you add some documentation of this feature to the readme please, also can you rebase this branch against master as it's un-mergable right now.

[coolzilj]

@andrew Yes,of course. I'll be working on that.

[coolzilj]

@andrew It's mergable now. But I don't like the present way to complete a goal conversion like this

finished({"link_color" => ["purchase"]})

I'd like to take @iangreenleaf 's advise to make finished taking a single string/symbol of either an experiment name or a goal name, or a metric name. So there's a little more work to do.

[andrew]

Ok cool, ping me when you are happy :)

[andrew]

Sorry @coolzilj I think you'll need to rebate once more!

[coolzilj]

sorry @andrew , I'm afraid I don't have enough time to change the api recently, so I just leave it like that now. I have rebased and updated Readme. Hope it's mergable right now.

[andrew]

Ok, not a problem, I'll merge it in and we can see how it goes. Thanks!

[iangreenleaf]

Awesome! 👍

Since you liked my idea, I might take a look and see if I can adjust the finished API before this gets officially released.

[coolzilj]

Thanks @iangreenleaf , that would be awesome!

[bilus]

Hi,

An awesome addition. Doesn't work for me though. I define an experiment like this:

  config.experiments = {
    "buy_credits" => {
      alternatives: ["low", "high", "very_high"],
      goals: ["pack1", "pack2", "pack3"]
    }
  }

Obviously, ab_test works but the dashboard shows no completed experiments after a call to:

finished({"buy_credits" => ["pack2"]}, reset: true)

Am I missing something obvious? I'm getting a 'bit' desperate so if you can help with any suggestions, you have a standing invitation to coffee/beer if you're ever in my area.

In Split::Helper::finished Metric::possible_experiments returns an empty array.

UPDATE 1: normalized_experiments == {"buy_credits"=>{:alternatives=>["low", ["high", "very_high"]]}}

UPDATE 2: @andrew Have you updated the gem version number? If updated the gem to the latest one 0.5.0 and configuration.rb still contains https://gist.github.com/4708662 Same after

gem uninstall split
gem install split

I must be missing something obvious here.

[andrew]

@bilus I think there is a fix on master that has not been released yet, might want to try adding this to your Gemfile:

gem 'split', :git => 'git://github.com/andrew/split.git'

[bilus]

@andrew Thank you for getting back so quickly.

I tried this but w/o effect but it must have been something on my system
because pulling from coolzilj's master branch has worked.

Awesome job!

On Mon, Feb 4, 2013 at 8:04 PM, Andrew Nesbitt notifications@github.comwrote:

@bilus https://github.com/bilus I think there is a fix on master that
has not been released yet, might want to try adding this to your Gemfile:

gem 'split', :git => 'git://github.com/andrew/split.git'

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/109#issuecomment-13093168.