Experiment from Feature Flags should pick an Assignment Query based on hash attribute (or ask for one to be set)

[lukesonnet]

Summary

When you create an experiment rule in a feature flag, it spins up a linked Experiment, but it comes with no selected experiment assignment query. So every time you do this you need to select the experiment assignment query that maps to the attribute you used for hashing in the feature.

We should consider one of two options:

1. An improved UX to require users to select this assignment table when the experiment is created, or at least give them a better error warning when it is unset
2. a. Have some mapping between hash attributes in features <-> identifier types
   b. Automatically use that mapping to set the exposure query in the experiment (or, if no mapping, prompt experimenters to pick an exposure query)

Specifically, in here:

growthbook/packages/front-end/components/Features/RuleModal.tsx

Lines 296 to 297 in 27198a9

	...getNewExperimentDatasourceDefaults(
	datasources,

We should pass down the values.hashAttribute and hopefully have the experiment assignment queries know which hashAttribute values map to them so we can easily set this when the experiment is created.

[lukesonnet]

The above issue #2105 has a good example of how the status quo manifests in unclear errors.

I've also written more about this in a community PR here, copied below:

First of all, thanks a bunch for trying to close some issues on this project! Unfortunately, I have some qualms about setting a default assignment query. I think it will exacerbate some issues in our UX.

The root of the issue: The Experiment Assignment Table dropdown in the experiment form must be set by users, but it is somewhat hidden and not always clear what should be selected (users have to know: what identifier type and hash attribute does this table map to?). Setting it incorrectly will silently cause "no data found" errors when people run queries for a lot of people, or worse, it could coincidentally match IDs and lead to silent errors.

Therefore, I think we shouldn't be setting a default experiment assignment query unless we slightly retool the new experiment form to:

• have a separate modal page for datasource and modal, like "Data Settings". so that you always have to "accept" this default in some kind of simple Modal that only has (datasource and experiment assignment query) as the options. This way, if we do want defaults, at least we're putting it in front of people's eyes. (edge case: if people want to start an experiment but don't want to set this, we can let them skip it with a warning that they'll have to set them later in "analysis settings")
• furthermore, I think it'd be good to make a hash attribute <-> exposure query mapping Experiment from Feature Flags should pick an Assignment Query based on hash attribute (or ask for one to be set) #1943 that could easily set this default query in the modal proposed above to make it so it's more of a confirmation step rather than a "oh wait which id did I use again?" step.

[cameronmoten]

The main way i think of this is an “Onboarding Guide” for errors.

Your error should kick off a Step by Step way to solve any error. This can be re-used for any issue, and or if someone deletes an experiment assignment you would hit this error as well.

"Unknown experiment assignment table - anonymous_id" currently provides no info, and doesn't even show you where to click to solve the issue.

Suggested:

"No Experiment Assignment Table was selected for this Analysis -> [Click Here] to set it up. [Click Here] to learn more about it / a video guide on it etc.."