Skip to content

Latest commit

 

History

History
executable file
·
179 lines (139 loc) · 7.04 KB

README.mdx

File metadata and controls

executable file
·
179 lines (139 loc) · 7.04 KB
Raw
id slug title description date tags
kibCloudExperimentsPlugin
/kibana-dev-docs/key-concepts/cloud-experiments-plugin
Cloud Experiments service
The Cloud Experiments Service provides the necessary APIs to implement A/B testing scenarios, fetching the variations in configuration and reporting back metrics to track conversion rates of the experiments.
2022-09-07
kibana
dev
contributor
api docs
cloud
a/b testing
experiments

Kibana Cloud Experiments Service

The Cloud Experiments Service provides the necessary APIs to implement A/B testing scenarios, fetching the variations in configuration and reporting back metrics to track conversion rates of the experiments.

The cloudExperiments plugin is disabled by default and only enabled on Elastic Cloud deployments.

Public API

If you are developing a feature that needs to use a feature flag, or you are implementing an A/B-testing scenario, this is how you should fetch the value of your feature flags (for either server and browser side code):

First, you should declare the optional dependency on this plugin. Do not list it in your requiredPlugins, as this plugin is disabled by default and only enabled in Cloud deployments. Adding it to your requiredPlugins will cause Kibana to refuse to start by default.

// plugin/kibana.json
{
  "id": "myPlugin",
  "optionalPlugins": ["cloudExperiments"]
}

Please, be aware that your plugin will run even when the cloudExperiment plugin is disabled. Make sure to declare it as an optional dependency in your plugin's TypeScript contract to remind you that it might not always be available.

Fetching the value of the feature flags

First, make sure that your feature flag is listed in FEATURE_FLAG_NAMES. Then, you can fetch the value of your feature flag by using the API cloudExperiments.getVariation as follows:

import type { CoreSetup, CoreStart, Plugin } from '@kbn/core/(public|server)';
import type { 
  CloudExperimentsPluginSetup, 
  CloudExperimentsPluginStart
} from '@kbn/cloud-experiments-plugin/common';

interface SetupDeps {
  cloudExperiments?: CloudExperimentsPluginSetup;
}

interface StartDeps {
  cloudExperiments?: CloudExperimentsPluginStart;
}

export class MyPlugin implements Plugin<void, void, SetupDeps, StartDeps> {
  public setup(core: CoreSetup, deps: SetupDeps) {
    this.doSomethingBasedOnFeatureFlag(deps.cloudExperiments);
  }
  
  public start(core: CoreStart, deps: StartDeps) {
    this.doSomethingBasedOnFeatureFlag(deps.cloudExperiments);
  }
  
  private async doSomethingBasedOnFeatureFlag(cloudExperiments?: CloudExperimentsPluginStart) {
    let myConfig = 'default config';
    if (cloudExperiments) {
      myConfig = await cloudExperiments.getVariation(
        'my-plugin.my-feature-flag', // The key 'my-plugin.my-feature-flag' should exist in FEATURE_FLAG_NAMES
        'default config'
      );
    }
    // do something with the final value of myConfig...
  }
}

Since the getVariation API returns a promise, when using it in a React component, you may want to use the hook useEffect.

import React, { useEffect, useState } from 'react';
import type { 
  CloudExperimentsFeatureFlagNames,
  CloudExperimentsPluginStart
} from '@kbn/cloud-experiments-plugin/common';

interface Props {
  cloudExperiments?: CloudExperimentsPluginStart;
}

const useVariation = <Data>(
  cloudExperiments: CloudExperimentsPluginStart | undefined,
  featureFlagName: CloudExperimentsFeatureFlagNames,
  defaultValue: Data,
  setter: (value: Data) => void
) => {
  useEffect(() => {
    (async function loadVariation() {
      const variationUrl = await cloudExperiments?.getVariation(featureFlagName, defaultValue);
      if (variationUrl) {
        setter(variationUrl);
      }
    })();
  }, [cloudExperiments, featureFlagName, defaultValue, setter]);
};

export const MyReactComponent: React.FC<Props> = ({ cloudExperiments }: Props) => {
  const [myConfig, setMyConfig] = useState('default config');
  useVariation(
    cloudExperiments,
    'my-plugin.my-feature-flag', // The key 'my-plugin.my-feature-flag' should exist in FEATURE_FLAG_NAMES
    'default config',
    setMyConfig
  );
  
  // use myConfig in the component...
}

Reporting metrics

Experiments require feedback to analyze which variation to the feature flag is the most successful. For this reason, we need to report some metrics defined in the success criteria of the experiment (check back with your PM if they are unclear).

Our A/B testing provider allows some high-level analysis of the experiment based on the metrics. It also has some limitations about how it handles some type of metrics like number of objects or size of indices. For this reason, you might want to consider shipping the metrics via our usual telemetry channels (core.analytics for event-based metrics, or ).

However, if our A/B testing provider's analysis tool is good enough for your use case, you can use the api reportMetric as follows.

First, make sure to add the metric name in METRIC_NAMES. Then you can use it like below:

import type { CoreStart, Plugin } from '@kbn/core/(public|server)';
import type { 
  CloudExperimentsPluginSetup, 
  CloudExperimentsPluginStart 
} from '@kbn/cloud-experiments-plugin/common';

interface SetupDeps {
  cloudExperiments?: CloudExperimentsPluginSetup;
}

interface StartDeps {
  cloudExperiments?: CloudExperimentsPluginStart;
}

export class MyPlugin implements Plugin<void, void, SetupDeps, StartDeps> {
  public start(core: CoreStart, deps: StartDeps) {
    // whenever we need to report any metrics: 
    // the user performed some action, 
    // or a metric hit a threshold we want to communicate about
    deps.cloudExperiments?.reportMetric({
      name: 'Something happened', // The key 'Something happened' should exist in METRIC_NAMES
      value: 22, // (optional) in case the metric requires a numeric metric
      meta: { // Optional metadata.
        hadSomething: true,  
        userType: 'type 1',  
        otherNumericField: 1,  
      }
    })
  }
}

Testing

To test your code locally when developing the A/B scenarios, this plugin accepts a custom config to skip the A/B provider calls and return the values. Use the following kibana.dev.yml configuration as an example:

xpack.cloud_integrations.experiments.enabled: true
xpack.cloud_integrations.experiments.flag_overrides:
  "my-plugin.my-feature-flag": "my custom value"

How is my user identified?

The user is automatically identified during the setup phase. It currently uses the ESS deployment ID, meaning all users accessing the same deployment will get the same values for the getVariation requests unless the A/B provider is explicitly configured to randomize it.

If you are curious of the data provided to the identify call, you can see that in the cloud plugin.

Development

See the kibana contributing guide for instructions setting up your development environment.