Add experiment tags to flow events

[davismtl]

If we A/B test 2 different signin/signup flows, we need to tell them apart in our flow events.

We should add a flow id (version) to our flow events.

The IDs could just be numerical and we describe them on a github page or in a spreadsheet for future reference.

[rfk]

Is a simple version sufficient, or do we need to tag them with a list of all active experiments and the group into which the user was allocated? I can imagine us having multiple things being A/B tested at once, that might be useful to track via flow metrics.

[davismtl]

@rfk I'm not sure I understand what you mean by "simple version". Can you clarify?

[rfk]

I meant "an integer version number".

Is "this was flow number 7" sufficiently granular for our purposes, or do we need to capture something like "this flow had the mailcheck experiment turned on, and variant 2 of the new password hint, but the show-password button emoji were not active"? In other words, a combination of individual A/B-testable things, rather than a single identifier.

[davismtl]

Ideally, users don't get bucketed into multiple tests at the same time to be sure that all variations are similar. It's not impossible to bucket people into multiple tests properly but I think it will just be far more complicated than what we need right now.

In Summary:
Tracking a flow number "7" would be enough for the short to medium term tests we plan on doing.

However, if we think that little effort is required to log a combination of test ids (7,9,12) for any given flow ID, then we should support that now and figure out how that works in practice down the road. This could maybe be used for multi-variant testing too.

[philbooth]

However, if we think that little effort is required to log a combination of test ids...

Logging multiple identifiers is no more difficult than logging a single value, although I'm not clear on what the best way to subsequently store them in redshift / query them in redash would be. What did you have in mind @rfk?

Alternatively, if the likely number of experiments is not going to be so large as to make it impractical, we could get the best of both worlds with a bit mask. That way we're exporting a single value for storage but it encodes the on/off state for a whole heap of different experiments.

[rfk]

In my head I was imagining an extra table, along the lines of:

+---------+--------------------+----------+
| flow_id | experiment         | group    |
+---------+--------------------+----------+
| 123456  | email_autocomplete | control  |
| 123456  | mailcheck_v2       | variant1 |
| ABCDEF  | email_autocomplete | enabled  |
| ABCDEF  | mailcheck_v2       | variant2 |
| ...     | ...                | ...      |
+---------+--------------------+----------+

But that could easily be overkill at this stage :-)

FWIW the content-server has some precedent here, with the way it handles multiple live experiments:

https://github.com/mozilla/fxa-content-server/blob/master/app/scripts/lib/experiment.js

And converts them into experiment:X, exp_group:Y tags on events when sending them to datadog:

https://github.com/mozilla/fxa-content-server/blob/master/server/lib/statsd-collector.js#L77

[philbooth]

In my head I was imagining an extra table...

That makes a lot of sense to me and I don't see it as overkill either. Thanks.

[philbooth]

@shane-tomlinson, fyi this is the issue we just mentioned in the meeting.

[shane-tomlinson]

Is a simple version sufficient, or do we need to tag them with a list of all active experiments and the group into which the user was allocated? I can imagine us having multiple things being A/B tested at once, that might be useful to track via flow metrics.

FWIW, for "signup experiments", we only allow a user to be in one experiment at a time. This is so that one experiment does not inadvertently influence another experiment, plus it makes metrics bucketing on DD simpler.

Either proposed solution, having two columns (one for experiment, one for group), or a single column (experiment:group) could work.

[davismtl]

I agree, for signup experiments, best they be in only one at a time.

They could however be bucketed into another experiment after sign-up. (e.g. device manager)

[rfk]

They could however be bucketed into another experiment after sign-up

Flow events as they currently stand will be useless to these experiments, since we only emit flow events during a login flow.

[shane-tomlinson]

Flow events as they currently stand will be useless to these experiments, since we only emit flow events during a login flow.

login meaning signin or signup.

[rfk]

I wish we had a general-purpose word for "signin or signup flow".

[philbooth]

I'm going to pick this up Real Soon Now. Initial plan is to emit experiment events for each experiment that a user is part of.