A/B tests vs. feature flags - UX issue

[paolodamico]

We're now introducing full A/B testing functionality (experimentation) to PostHog. Some of it will work on top of our FF infrastructure. One key UX problem to figure out is the mental model of FFs vs. experiments. They solve fundamentally different problems (e.g. phased rollouts vs. new feature testing) and are both relevant to developing better products. We want both to coexist.

So, how should experiments be treated in regard to FFs?

• Are experiments a special case of FFs? Or the other way around?
• Are they separate features?

Thoughts? @clarkus @neilkakkar @liyiy @macobo

[paolodamico]

To move things along here's a proposal.

• We treat them as two separate features.
• We have a single "Experiments & FFs" page with two tabs (as with the events & actions page) and you can create either a FF or an experiment.

[macobo]

My proposal:

• Experiments are a special kind of feature flags under the hood for everything that looks outside posthog. Rationale: We want to avoid needing to roll out yet another thing into client libraries and in our backend. Flags side of thing handles the "select the audience" part well and if there's something missing functionality-wise we'll add it there.
• UX-wise experiments are a separate feature/page where they can be created from. This can be a tab as well - we can switch that around as needed.
• We'll have a separate Experiments model which has a flag attached + all the additional metadata on the experiment is gathered there.
• We don't list flags that are part of experiments in the feature flags page

[neilkakkar]

Q: When can you run an experiment without a FF?

The way I see it, FFs can be used for both experiments or rollouts.

[paolodamico]

Not sure I follow @neilkakkar but I agree with @macobo, experiments are a subset of FFs in the sense that all experiments have a FF but not all FF have an experiment.