Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add snapshot analysis creation/lookup, baseline chooser #1515

Merged
merged 43 commits into from Sep 1, 2023

Conversation

lukesonnet
Copy link
Contributor

@lukesonnet lukesonnet commented Jul 27, 2023

image

Features and Changes

As part of the Snapshot Refactor (#1284), one of the key next steps was enabling creating multiple analyses that rely on one snapshot and set of queries. This would be done through quick calls to gbstats, stored side-by-side, and accessed by their ExperimentSnapshotAnalysisSettings which represents meta-data about the analysis.

This PR is one way to achieve this goal with the new Snapshot model, and use it for a simple feature.

  • Adds POST endpoint for creating snapshot analyses, and createSnapshotAnalyses which can create multiple snapshot analyses. Much of this code duplicates ExperimentQueryRunner so we should consider how to keep this dry.
  • Adds a hook to useSnapshot() to allow for setting the ExperimentSnapshotAnalysisSettings on the fly and to retrieve the right data for a given analysisSettings object.

TODO:

  • Add functionality to reports
  • Use different approach to handling variation order than sorting
  • Actually hook up to new BaselineChooser

Here's the call that this PR used to fire in AnalysisSettingsBar when you messed with the baseline chooser I had built into an old commit (but have ripped out when rebasing on results-table-variation-filtering:

onChange={async (i) => {
  const newSettings: ExperimentSnapshotAnalysisSettings = {
    ...analysis.settings,
    baselineVariationIndex: Number(i),
  };
  if (!getSnapshotAnalysis(snapshot, newSettings)) {
    await apiCall(`/snapshot/${snapshot.id}/analysis`, {
      method: "POST",
      body: JSON.stringify({
        analysisSettings: newSettings,
      }),
    })
      .then(() => {
        track("Experiment Analysis: switch baseline");
      })
      .catch((e) => {
        setRefreshError(e.message);
      });
  }
  setAnalysisSettings(newSettings);
  mutate();
}}

Dependencies

#1631

Testing

TODO

@github-actions
Copy link

github-actions bot commented Jul 27, 2023

Your preview environment pr-1515-bttf has been deployed.

Preview environment endpoints are available at:

@lukesonnet lukesonnet marked this pull request as draft July 27, 2023 17:12
@lukesonnet lukesonnet changed the base branch from main to results-table-variation-filtering August 30, 2023 15:06
@lukesonnet lukesonnet changed the title [WIP] Add snapshot analysis creation/lookup and baseline variation switcher [WIP] Add snapshot analysis creation/lookup Aug 30, 2023
Base automatically changed from results-table-variation-filtering to main August 30, 2023 19:33
@bryce-fitzsimons bryce-fitzsimons marked this pull request as ready for review August 31, 2023 04:34
Copy link
Contributor Author

@lukesonnet lukesonnet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't approve my own PR, but this looks good to me. One comment about the UX when running queries and another tiny comment.

@bryce-fitzsimons bryce-fitzsimons changed the title [WIP] Add snapshot analysis creation/lookup Add snapshot analysis creation/lookup, baseline chooser Aug 31, 2023
@bryce-fitzsimons bryce-fitzsimons merged commit 64c19f5 into main Sep 1, 2023
3 checks passed
@bryce-fitzsimons bryce-fitzsimons deleted the ls/analysis_baseline branch September 1, 2023 04:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants