Mutually exclusive experiments

[jdorn]

Ability to mark two or more experiments as mutually exclusive, meaning a single user will never be included in more than one of them. We can do this with a new optional experiment property namespace.

The namespace property is a tuple with 3 parts: id, range_start, and range_end. For example, [pricing, 0, 0.5].

Users are assigned a value of 0 to 1 based on a hash of their id and the namespace id.

The range defines which part of the namespace should be included in the experiment.

The following two experiments would be mutually exclusive.

{
  "key": "test1",
  "variations": [0, 1],
  "namespace": ["pricing", 0, 0.5]
}

{
  "key": "test2",
  "variations": [0, 1],
  "namespace": ["pricing", 0.5, 1]
}

The benefits of this approach are the simplicity and that it doesn't require any API calls or persistence. That means there is minimal impact to complexity, size, and performance.

The biggest downside is that you can't start an experiment at 100% and then decide halfway through that you want to add a new mutually exclusive experiment. You need to plan up front to leave space. This is a little restrictive, but that's not necessarily a bad thing since this way it's much harder to shoot yourself in the foot and cause statistical problems down the road.