/convert all experiments for a single KPI?

[Jud]

We are looking at using sixpack for a/b testing some design changes on our online store. I'd imagine that the flow for a simple button color test would be:
/participate?experiment=button_color&alternatives=black&alternatives=blue&client_id=123

And if the user clicked on the button, we would convert it:
/convert?experiment=button_color&client_id=123&kpi=add_to_cart

If we also wanted to test whether the button color has an effect on revenue, we could do:
/convert?experiment=button_color&client_id=123&kpi=checkout

when a user goes through the checkout flow. The problem is storing all of the currently enrolled experiments for a given client. Seems like this information should be available internally to sixpack.

Is there a standard way to handle this kind of behavior without needing to externally store the enrolled experiments for a client?

[zackkitzmiller]

If I'm understanding this correctly, Sixpack does handle this internally. You don't need to worry about what experiments a user is participating on, you only need to manage client_ids. All Official clients manage client_ids for you.

[Jud]

@zackkitzmiller I could be mistaken, but I don't see a way in the sixpack API (or any of the clients) to convert all experiments without needing to know all of the experiments that a user is enrolled in.

For us, the end goal is to get more users to make purchases. It'd be nice to know the micro and macro effects that an experiment has. E.g. people in with Alternative A add more items to their basket, but people in Alternative B are more likely to actually purchase.

I'm thinking that in our checkout controller, we'd add something like:

$sp = new \SeatGeek\Sixpack\Session\Base(array('clientId' => 1234));
$sp->convert('*', 'Purchase');

[zackkitzmiller]

Ahh, I see. Yes.

You're right at this point, you're going to have to have explicit calls to convert, perhaps in a loop. Let me discus this with @erwaller, but I think we might want to hold on to our current approach.

You're almost certainly going to want some kind of ServiceProvider or similar to get your Session instance, you could theoretically keep track of experiments there in a list, and iterate in your controller.

[Jud]

It seems useful to be able to define experiments in code without needing to duplicate the experiment list that sixpack already keeps up with.

This wouldn't even require a new endpoint, only the ability to specify an experiment wildcard, or omit the experiment variable in the api call.

[armen52]

Updated: On second thought, although this behavior would be best for us in the majority of cases it would present difficulties as soon as we launched any experiment where the "conversion" goal was not the same as the others, e.g. getting users to share a page with others more often. In that case I am not sure how it would be handled, so maybe we're better off as is.

---

I agree that this would be useful functionality. Most of our tests aim for increases in ultimate conversions, and this would make tracking conversion on most of our tests a lot simpler, i.e. only require our developers to implement code for the experiment where the experiment's change or variation occurs in the funnel.

With this change we could simply have a static call to Sixpack to "convert" a user on the last page in our funnel ("success" page), without having to go back and modify it every time we launch a new experiment somewhere upstream in the funnel (to also make a conversion call for the latest new experiment somewhere along the way).

[Jud]

I see how this could get complicated without adding something like namespaces to experiments and allowing you to convert all experiments in a single namespace.

One question is, does calling convert on an experiment that the user isn't enrolled in, automatically enroll them? If it doesn't, then we can just keep track of all available experiments and loop through them.

Edit: Looking at the Experiment.convert method, if a client wasn't participating, a ValueError is thrown, which will return a 400 status code. Barring a better way to use the information that sixpack uses internally, we can just keep track of namespaced experiments ourselves and loop over all experiments in a namespace and call convert

[zackkitzmiller]

Converting a user that hasn't participated does nothing.