Ability to change variants?

[jasonm]

Hello,

I have the following scenario:

• Previously I was testing experiment "landing_page" with variants "original" and "new_copy".
• We decided to run a new test.
• We replaced this test with a new one, same experiment name "landing_page" with variants "original" and "screencast_and_testimonials".
• Split will now only serve variant "original"
• If you visit http://domain.com/?landing_page=screencast_and_testimonials, the screencast_and_testimonials variant is displayed to the user. However, we use the following code to record the AB variant into KISSmetrics, and when we manually request screencast_and_testimonials, it was being recorded as new_copy (variant Resetting an experiment doesn't reset existing participants #2 from the prior experiment):

  <% Split::Experiment.all.each do |experiment| %>
    <% km.set(experiment.name, ab_test(experiment.name, *experiment.alternative_names)) %>
  <% end %>

Is this expected behavior? e.g should we not reuse an experiment name with new sets of variants?

(As an aside, is there a better way to record the variants into KM other than that snippet?)

Thanks,
-Jason

[andrew]

Experiments have the concept of versions, if you change alternatives or reset an experiment it should bump the version of the experiment, this is because the user only has the experiment name and a single alternative stored in their cookie.

With no way to expire their cookie, incrementing the version (which stores the experiment as experiment:version in the cookie) means that the user should receive a new alternative from the second version of the experiment.

I suspect that when you use km in this fashion the experiment version is never used so you only every get version 0, and the alternatives have already been set for that.

I think something like this should work for you:

<% Split::Experiment.all.each do |experiment| %>
  <% km.set(experiment.key, ab_test(experiment.name, *experiment.alternative_names)) %>
<% end %>

[jasonm]

Thanks @andrew - makes sense.