workshop/3-Experimentation/3.6-Optimizely-AB-Experiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Retail Demo Store Experimentation Workshop - Using Optimizely Full Stack\n",
    "\n",
    "In this exercise we will define, launch, and evaluate the results of an A/B experiment using Optimizely. If you have not already stepped through the [3.1-Overview](./3.1-Overview.ipynb) workshop notebook, please do so now as it provides the foundation built upon in this exercise.\n",
    "\n",
    "Recommended Time: 30 minutes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "In this workshop, you'll see how to use Optimizely to deploy A/B tests to compare different personalized recommendations algorithms as they are used to surface product recommendations to the users of the Retail Demo Store, and then you will observe the results of your A/B tests using the Optimizely console.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Prerequisites\n",
    "\n",
    "It is assumed that you have either completed the [Personalization workshop](../1-Personalization/1.1-Personalize.ipynb) or those resources have been pre-provisioned in your AWS environment. If you are unsure and attending an AWS managed event such as a workshop, check with your event lead.\n",
    "\n",
    "You will also need access to an Optimizely Rollouts account, in order to configure the Optimizely experimentation framework.  You can set up a free account [here](https://www.optimizely.com/rollouts-signup/?utm_source=demo&utm_campagin=aws-personalize-workshop).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why Optimizely?\n",
    "\n",
    "The examples given in sections [3.2](./3.2-AB-Experiment.ipynb), [3.3](./3.3-Interleaving-Experiment.ipynb), [3.4](./3.4-Multi-Armed-Bandit-Experiment.ipynb) detail how to implement a testing framework from scratch. Although implementing a testing framework from scratch will work, significant technical work is required to get up and running. It's very likely that costly errors can be made when trying to solve the fundamentals of an A/B testing framework:\n",
    "\n",
    "- Making variation assignment random\n",
    "- Ensuring traffic is equally split between the variations\n",
    "- Implementing success metrics\n",
    "- Reporting on test outcomes\n",
    "\n",
    "Once the logic to run and measure your experiment is built, you may run into additional problems and features to improve. For example:\n",
    "\n",
    "- What if you want to limit the percentage of users that get into your experiment?\n",
    "- How do you make changes to a running test?\n",
    "- How do you pause a test once it’s complete?\n",
    "- How do you make sure your test findings are accurate?\n",
    "- How will you ensure that the home-built solution scales and is reliable?\n",
    "\n",
    "Requirements for robust experimentation can quickly snowball into a lot of work. Rather than tackling these problems as they arise, another option is to use or build on top of an existing external platform. This workshop covers using Optimizely’s Full Stack Experimentation product to more easily and quickly execute an A/B test.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Optimizely Full Stack?\n",
    "\n",
    "Optimizely Full Stack is a progressive delivery and experimentation platform for product and engineering teams, which allows you to both progressively release new features to your users as well as experiment with different versions of the features. Full Stack is implemented directly within client-side or server-side code via a Software Development Kit (SDK) available in [many languages](https://docs.developers.optimizely.com/full-stack/docs/quickstarts), or as its own standalone service using [Optimizely Agent](https://docs.developers.optimizely.com/full-stack/docs/use-optimizely-agent).\n",
    "\n",
    "An implementation of Optimizely Full Stack provides users with a suite of functions that enable delivery of new application features, as well as experimentation on existing features:\n",
    "\n",
    "- **Feature Flags**: The ability to remotely toggle features on/off\n",
    "- **Feature Rollouts**: Increase or decrease the percentage of users exposed to a feature flag at any point in time\n",
    "- **Feature Variables**: Add a variable to any feature, allowing for aspects of a feature to be remotely configured at any time\n",
    "- **Feature Experimentation**: Create variations of any existing feature and measure the impact of feature changes on conversion metrics\n",
    "- **Audience Attributes**: Create user attributes for results segmentation and audience targeting use-cases\n",
    "- **Event Tracking**: Track interactions to measure user impact\n",
    "\n",
    "To visualize these features watch this [4 minute video](https://www.youtube.com/watch?v=DVjnOYi4214).\n",
    "\n",
    "The easiest way to get started with Optimizely is to use one of our SDKs for the language in which your application is written.  In this workshop, you will integrate Optimizely Full Stack, using the [Python SDK](https://optimizely-python-sdk.readthedocs.io/).  The Python SDK will be used in the Recommendations service of the Retail Demo Store to run experiments across the three personalization use cases of the Retail Demo Store.  In your own environments, you can also deploy Optimizely as a service using [Optimizely Agent](https://docs.developers.optimizely.com/full-stack/docs/use-optimizely-agent), which is ideal for microservice environments or environments where Optimizely does not have an SDK for a particular language. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How Optimizely Works\n",
    "\n",
    "The diagram below illustrates how the Optimizely service works.\n",
    "\n",
    "![Optimizely Architecture](./images/optimizely/architecture.png)\n",
    "\n",
    "1. The Optimizely Application (on the left-hand-side) allows you to create/launch feature configurations and tests from the Optimizely Application, as well as create metrics to measure experiment outcomes.\n",
    "2. All the relevant experiment information you've saved in Optimizely is populated to a condensed JSON format called the datafile. The datafile contains information such as:\n",
    "   1. Experiments that are running\n",
    "   2. Variations of the variables in those running experiments\n",
    "   3. Traffic allocation percentages of the experiment variations\n",
    "3. Optimizely’s SDKs use the project datafile to create an Optimizely instance, either client-side or server-side, that buckets users into the A/B test.\n",
    "4. Interaction events are tracked asynchronously back to Optimizely’s results pages, allowing users to measure the impact of their experiments \n",
    "\n",
    "Optimizely is integrated into the Retail Demo Store, and with Amazon Personalize as shown in the architecture diagram below:\n",
    "\n",
    "![Optimizely in Retail Demo Store Architecture](./images/optimizely/retail-demo-store-architecture.png)\n",
    "\n",
    "#### How Optimizely Buckets Users\n",
    "\n",
    "Optimizely Full Stack uses a hashing algorithm that combines the user ID, feature flag key, and traffic allocation percentage to determine user bucketing. This allows bucketing to occur in the SDK code, rather than requiring the SDK to communicate with an external API. This method ensures that bucketing decisions happen in matter of microseconds, minimizing the performance impact on your application. It also ensures that users receive the same variation in an experiment, even if traffic allocation is increased. More information available [here](https://docs.developers.optimizely.com/full-stack/docs/how-bucketing-works#:~:text=Bucketing%20is%20the%20process%20of,which%20variation%20they%20should%20see.)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Steps\n",
    "\n",
    "### 1. Understand your Hypothesis\n",
    "\n",
    "Your goal is to increase revenue from your e-commerce site.\n",
    "\n",
    "Website analytics have shown that user sessions frequently end on the home page for your e-commerce site, the Retail Demo Store. Furthermore, when users do make a purchase, most purchases are for a single product. Currently on our home page we are using a basic approach of recommending featured products.\n",
    "\n",
    "You hypothesize that adding personalized recommendations to the home page will result in increasing the click-through rate of products by 25%. The current click-through rate is 15%.\n",
    "\n",
    "\n",
    "To verify your hypothesis in a data-driven manner, you've decided to run an A/B test against different personalization strategies.\n",
    "\n",
    "In the next steps, you'll implement this A/B test, a similar A/B test as in [section 3.2](./3.2-AB-Experiment.ipynb), comparing the [Default Product Resolver](https://github.com/aws-samples/retail-demo-store/blob/master/src/recommendations/src/recommendations-service/experimentation/resolvers.py#L382) with the [Personalize Resolver](https://github.com/aws-samples/retail-demo-store/blob/master/src/recommendations/src/recommendations-service/experimentation/resolvers.py#L384).\n",
    "\n",
    "This time; however, instead of building an A/B test platform from the ground-up, you'll leverage Optimizely's platform to run an A/B test on top of the configuration of a [feature flag](https://www.optimizely.com/optimization-glossary/feature-flags/).\n",
    "\n",
    "Let's get started!\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Log in to your Optimizely Account\n",
    "\n",
    "If you have not yet done so, create a free Optimizely Full Stack account, called Rollouts, [here](https://www.optimizely.com/rollouts-signup/?utm_source=demo&utm_campagin=aws-personalize-workshop).  If you already have an account, log in now [here](https://app.optimizely.com).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Create a Feature Flag\n",
    "\n",
    "In the Optimizely user interface of a new account, you will be prompted to create a new feature flag. Let's create a feature flag to control the product recommendations on the home page:\n",
    "\n",
    "- Enter the name `home_product_recs`\n",
    "- Click \"Create New Feature\"\n",
    "- Click \"Close Quickstart Guide\"\n",
    "- Click \"Return to Dashboard\"\n",
    "\n",
    "In the Optimizely user interface of an existing account:\n",
    "\n",
    "- Navigate to the \"Features\" dashboard\n",
    "- Click \"New Feature\"\n",
    "- Enter the name `home_product_recs`\n",
    "- Click \"Create Feature\"\n",
    "![Optimizely Create Feature Flag Button](./images/optimizely/create-feature-flag.png)\n",
    "![Optimizely Create Feature Flag Dialog](./images/optimizely/create-feature-flag2.png)\n",
    "\n",
    "Great. You've now created a feature flag. Optimizely makes it easy to build A/B tests off of the configuration of a [feature flag](https://www.optimizely.com/optimization-glossary/feature-flags/). \n",
    "\n",
    "In the next step, we'll configure our home_product_recs feature flag by adding variables.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Add Feature Variables\n",
    "\n",
    "Variables allow you to pass information about the feature flag to the user interface and configure parameters for a feature test.  In this step, you are going to add two variables to control which algorithm will be used for recommendations and the configuration of that algorithm.\n",
    "\n",
    "First, add a feature variable called `algorithm_type` to control the type of algorithm that powers our recommendations. To do this, we'll first use a string variable.\n",
    "\n",
    "- Click on your newly created `home_product_recs` feature\n",
    "- Click on \"Variable Keys & Types\" in the left navigation\n",
    "- Click the \"Select Variable Type\" dropdown\n",
    "- Choose \"String\"\n",
    "- For the variable key, type in the name of the variable `algorithm_type`\n",
    "- For the default value, we'll use `product` to match the [DefaultProductResolver](https://github.com/aws-samples/retail-demo-store/blob/master/src/recommendations/src/recommendations-service/experimentation/resolvers.py#L354-L361)\n",
    "\n",
    "```python\n",
    "\n",
    "    # from src/recommendations/src/recommendations-service/experimentation/resolvers.py\n",
    "\n",
    "    class ResolverFactory:\n",
    "        \"\"\" Provides resolver instance given a type and initialization arguments \"\"\"\n",
    "        TYPE_HTTP = 'http'\n",
    "        TYPE_PRODUCT = 'product'\n",
    "        TYPE_SIMILAR = 'similar'\n",
    "        TYPE_PERSONALIZE_RECOMMENDATIONS = 'personalize-recommendations'\n",
    "        TYPE_PERSONALIZE_RANKING = 'personalize-ranking'\n",
    "        TYPE_RANKING_NO_OP = 'ranking-no-op'\n",
    "\n",
    "```\n",
    "\n",
    "Since each algorithm has different configuration options, you will also add a variable `algorithm_config` to control the configuration of our recommendation algorithm.\n",
    "\n",
    "- Click on \"Variable Keys & Types\" in the left navigation\n",
    "- Click the plus icon on the far right\n",
    "- Click the \"Select Variable Type\" dropdown\n",
    "- Choose \"JSON\"\n",
    "- For the variable key, type in the name of the variable `algorithm_config`\n",
    "- For the default value, we'll use an empty object `{}`, since the parameters for the [DefaultProductResolver](https://github.com/aws-samples/retail-demo-store/blob/master/src/recommendations/src/recommendations-service/experimentation/resolvers.py#L29-L37) are optional.\n",
    "\n",
    "![Optimizely Add Feature Variable algorithm_config](./images/optimizely/add-feature-variable.png)\n",
    "\n",
    "Nice. Your feature flag now has variables which we can remotely control from Optimizely. In the next steps, we will connect Optimizely to the code so that changes in Optimizely can be seen in your Retail Demo Store.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Find and Deploy Your Optimizely SDK Key\n",
    "\n",
    "If you have already deployed the Retail Demo Store with your Optimizely key, you can skip this step, if not, find your Optimizely SDK key to connect your Optimizely Application to your Retail Demo Store instance.  \n",
    "\n",
    "In the Optimizely console:\n",
    "\n",
    "- Navigate to the far left 'Settings'\n",
    "- Click 'Environments'\n",
    "- Copy your Development SDK key value\n",
    "\n",
    "![Optimizely SDK Key](./images/optimizely/sdk-key.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Once you have your SDK key, paste it in the cell below, where it says `optimizely_sdk_key` below and run the following code to set the `OPTIMIZELY_SDK_KEY` environment variable.  You will need to force a re-deploy of your Retail Demo Store instance so that the Optimizely Application is connected to the Retail Demo Store (this is also optional for the purposes of this workshop) and you can see interact with the feature tests you are configuring in the user interface.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the Optimizely SDK key in the optimizely_sdk_key string below, and run this cell.  \n",
    " \n",
    "# THIS IS ONLY REQUIRED IF YOU DID NOT SET THE SDK KEY IN YOUR ORIGINAL DEPLOYMENT\n",
    "\n",
    "optimizely_sdk_key = ''\n",
    " \n",
    "if optimizely_sdk_key:\n",
    "    response = ssm.put_parameter(\n",
    "        Name='retaildemostore-optimizely-sdk-key',\n",
    "        Value='{}'.format(optimizely_sdk_key),\n",
    "        Type='String',\n",
    "        Overwrite=True\n",
    "    )\n",
    " \n",
    "    optimizely_config = {\n",
    "        'sdk_key': '{}'.format(optimizely_sdk_key)\n",
    "    }\n",
    " \n",
    "    print(optimizely_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Read the SSM Parameters for the Optimizely SDK Key\n",
    "\n",
    "If you already deployed the Retail Demo Store with your Optimizely SDK key, run the following cell to read the Optimizely key from the SSM repository that was deployed in your AWS account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the SSM parameter for the Optimizely SDK key, if it was set a deploy time\n",
    "import boto3\n",
    "\n",
    "ssm = boto3.client('ssm')\n",
    "\n",
    "response = ssm.get_parameter(\n",
    "    Name='retaildemostore-optimizely-sdk-key'\n",
    ")\n",
    "\n",
    "optimizely_config = {\n",
    "    'sdk_key': response['Parameter']['Value']     # Do Not Change\n",
    "}\n",
    "\n",
    "print(optimizely_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Using the Optimizely SDK\n",
    "\n",
    "Now you will walk through the installation and initialization of the Optimizely SDK so we can implement our A/B test.  The Retail Demo Store environment provides pre-configured code that gives you an example implementation of the Optimizely SDK; in this step we will review what is required to deploy the Optimizely libraries in order to connect an application to Optimizely.\n",
    "\n",
    "Optimizely Full Stack comes with SDKs in [several languages](https://docs.developers.optimizely.com/full-stack/docs/quickstarts). Since the Retail Demo Store backend is built in Python, we'll use the Optimizely Python SDK. The SDK provides a set of APIs to evaluate the state of a feature flag or A/B test as well as track analytics for reporting the results of an experiment.\n",
    "\n",
    "Your Retail Demo Store already has the Python Optimizely SDK installed. If you were installing it from scratch, you could use pip from the command line:\n",
    "\n",
    "```bash\n",
    "pip install optimizely-sdk\n",
    "```\n",
    "\n",
    "To initialize the Optimizely SDK, you would pass in your SDK key environment variable from the previous step to the Optimizely constructor:\n",
    "\n",
    "```python\n",
    "    # from src/recommendations/src/recommendations-service/experimentation/experiment_optimizely.py\n",
    "\n",
    "    from optimizely import optimizely\n",
    "    \n",
    "    ...\n",
    "    \n",
    "    optimizely_sdk = optimizely.Optimizely(sdk_key=os.environ.get('OPTIMIZELY_SDK_KEY'))\n",
    "```\n",
    "\n",
    "We've already done this at the top of the [experiment_optimizely.py](https://github.com/aws-samples/retail-demo-store/blob/master/src/recommendations/src/recommendations-service/experimentation/experiment_optimizely.py#L11) file which will be used in the subsequent steps to control the algorithm powering our recommendations.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Implement the Feature Variables\n",
    "\n",
    "To get and use the values of the variables that we created in the Optimizely Application in step 2 and 3 above, we'll use the get_feature_variable_string and get_feature_variable_json SDK APIs:\n",
    "\n",
    "```python\n",
    "algorithm_type = optimizely_sdk.get_feature_variable_string('home_product_recs', 'algorithm_type', user_id)\n",
    "\n",
    "algorithm_config = optimizely_sdk.get_feature_variable_json('home_product_recs', 'algorithm_config', user_id)\n",
    "```\n",
    "\n",
    "The above two APIs take the same parameters:\n",
    "- **feature_key** (ex: 'home_product_recs'): the feature key that we created in the Optimizely Application as a string\n",
    "- **variable_key** (ex: 'algorithm_type'): the variable key that we created in the Optimizely Application as a string\n",
    "- **user_id** (ex: 'user123'): a string that Optimizely uses to randomly bucket different users into different variations of the experiment. This can be any string that uniquely identifies a particular user.\n",
    "\n",
    "By passing the algorithm_type and algorithm_config variable values to the `ResolverFactory.get` [method](https://github.com/aws-samples/retail-demo-store/blob/master/src/recommendations/src/recommendations-service/experimentation/experiment_optimizely.py#L19-L21), you can control which `ProductResolver` is used to power the featured products listings.\n",
    "\n",
    "```python\n",
    "\n",
    "    # from src/recommendations/src/recommendations-service/experimentation/experiment_optimizely.py\n",
    "    \n",
    "    algorithm_type = optimizely_sdk.get_feature_variable_string(self.feature, 'algorithm_type', user_id=user_id)\n",
    "    algorithm_config = optimizely_sdk.get_feature_variable_json(self.feature, 'algorithm_config', user_id=user_id)\n",
    "    resolver = resolvers.ResolverFactory.get(type=algorithm_type, **algorithm_config)\n",
    "    \n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Setup the A/B Test\n",
    "\n",
    "Now that we've implemented our feature and variables to control how we are populating our featured product listings, we can go to the Optimizely interface to setup an A/B test to experiment with which product recommendations algorithm will drive the most up-lift and click-throughs to our products.\n",
    "\n",
    "We will do so by creating a \"Feature Test\" which allows us to experiment with the feature flag we created above:\n",
    "\n",
    "- Navigate to the far left 'Experiments'\n",
    "- Click \"Create New Feature Test\"\n",
    "- Click \"Search by name or key\"\n",
    "- Select your home_product_recs feature\n",
    "- Name the experiment \"Homepage Recommendations Test\"\n",
    "- Click \"Create Feature Test\"\n",
    "\n",
    "![Create A/B Test Button](./images/optimizely/create-feature-test.png)\n",
    "![Create A/B Test Modal](./images/optimizely/create-feature-test2.png)\n",
    "\n",
    "The Optimizely Application will automatically create two variations, one with the feature flag off and one with the feature flag on. This default is useful if you wanted to run an A/B test comparing the Retail Demo Store with and without homepage product recommendations. In this workshop, we don't want to turn off product recommendations entirely but rather compare two different algorithms powering the product recommendations:\n",
    "\n",
    "For variation_1:\n",
    "- Toggle the home_product_recs feature On\n",
    "- Leave the default variable value for algorithm_type as `product`\n",
    "- Leave the default variable value for algorithm_config as an empty object `{}`\n",
    "\n",
    "For variation 2, we'll setup the AWS Personalize Recommendations. To do that we'll need to lookup the Amazon Personalize campaign ARN for product recommendations. This is the campaign that was created in the [Personalization workshop](../1-Personalization/1.1-Personalize.ipynb) (or was pre-built for you depending on your workshop event). Run the following code to get the campaign ARN:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = ssm.get_parameter(Name = 'retaildemostore-product-recommendation-campaign-arn')\n",
    "\n",
    "campaign_arn = response['Parameter']['Value']   # Do Not Change\n",
    "print('Personalize product recommendations ARN: ' + campaign_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For variation_2:\n",
    "- Ensure the home_product_recs toggle is also On\n",
    "- Change the variable value for algorithm_type to \"personalize-recommendations\"\n",
    "- Change the variable value for `algorithm_config` to include your campaign ARN, printed by the code above:\n",
    "\n",
    "```json\n",
    "{ \"campaign_arn\": \"Your_campaign_ARN\" }\n",
    "```\n",
    "- Scroll down and click save\n",
    "\n",
    "![Optimizely Setup Variations](./images/optimizely/setup-variations.png)\n",
    "\n",
    "You've now setup an A/B test on top of your feature flag (a.k.a a Feature Test) using Optimizely!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10. Instrument Experiment Metrics\n",
    "\n",
    "Before we start our A/B test in Optimizely, we need to make sure we're tracking metrics to evaluate which variation is performing better.\n",
    "\n",
    "Using the Optimizely Application, let's create an event:\n",
    "- Navigate to the far left \"Events\"\n",
    "- Click \"New Event\"\n",
    "- Use the event key \"ProductAdded\"\n",
    "- Click JavaScript SDK and note the code sample necessary to implement this event\n",
    "\n",
    "![Create Event Dashboard](./images/optimizely/create-event.png)\n",
    "![Create Event Modal](./images/optimizely/create-event2.png)\n",
    "\n",
    "Let's implement this tracking call in our Retail Demo Store.\n",
    "\n",
    "We've done this in the Retail Demo Store [frontend](https://github.com/aws-samples/retail-demo-store/blob/master/src/web-ui/src/analytics/AnalyticsHandler.js#L164-L226) already, but if you did it from scratch, you would install a client-side Optimizely SDK and make a client-side tracking call when the user clicks through to a product.\n",
    "\n",
    "```javascript\n",
    "\n",
    "     // from src/web-ui/src/analytics/AnalyticsHandler.js\n",
    "\n",
    "     productAddedToCart(user, cart, product, quantity, feature, experimentCorrelationId) {\n",
    "         ...\n",
    "         const userId = user.id.toString();\n",
    "         optimizelyClientInstance.track('ProductAdded', userId);\n",
    "         ...\n",
    "     }\n",
    "```\n",
    "\n",
    "As mentioned, we've already implemented this in the [AnalyticsHandler](https://github.com/aws-samples/retail-demo-store/blob/master/src/web-ui/src/analytics/AnalyticsHandler.js#L218-L225) portion of the frontend on the Retail Demo Store.\n",
    "\n",
    "Now that we've created and implemented our event, we'll want to add it to our experiment:\n",
    "\n",
    "- Navigate to the far left \"Experiments\"\n",
    "- Click your \"Homepage Recommendations Test\"\n",
    "- Click \"Metrics\"\n",
    "- Click our newly created \"ProductAdded\" event\n",
    "- Click \"Save to Experiment\"\n",
    "\n",
    "![Optimizely Add Metric](./images/optimizely/add-metric.png)\n",
    "\n",
    "Now that you've setup events, we're ready to run your experiment.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11. Run the Experiment & Simulate User Traffic\n",
    "\n",
    "Normally when you start an experiment, you run it for some time until it hits statistical significance. This can take some time depending on how many visitors are exposed to your experiment.\n",
    "\n",
    "Since this is a workshop that's tight on time, let's simulate traffic to the Retail Demo Store to see what it would be like to analyze the results of running the experiment in Optimizely.\n",
    "\n",
    "First, be sure to start the experiment in Optimizely before you simulate traffic to it:\n",
    "\n",
    "- Navigate to the far left \"Experiments\"\n",
    "- Click the \"Run\" button for the development environment matching the key of the environment you used in step 5\n",
    "- Wait 1 min\n",
    "\n",
    "![Optimizely Run Experiment](./images/optimizely/run-experiment.png)\n",
    "\n",
    "We wait 1 minute after saving to ensure that your action to start the Optimizely experiment has taken affect before simulating traffic below.\n",
    "\n",
    "Run the following code to hit the Optimizely API in a similar way that your application would to send data to Optimizely's results so we can simulate what it would be like to run this experiment to statistical significance.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install optimizely-sdk\n",
    "\n",
    "import logging\n",
    "import random\n",
    "\n",
    "from optimizely import optimizely, logger\n",
    "\n",
    "OPTIMIZELY_SDK_KEY = optimizely_config['sdk_key']\n",
    "\n",
    "optimizely_sdk = optimizely.Optimizely(\n",
    "  sdk_key=OPTIMIZELY_SDK_KEY,\n",
    "  logger=logger.SimpleLogger(min_level=logging.WARNING)\n",
    ")\n",
    "\n",
    "NUM_USERS = 500\n",
    "\n",
    "CONVERSION_RATES = {\n",
    "  'product': 0.5,\n",
    "  'personalize-recommendations': 0.75,\n",
    "}\n",
    "\n",
    "print(\"Simulating Traffic of %d users\" % NUM_USERS)\n",
    "print(\"Legend:\")\n",
    "print(\"  '.' == visitor\")\n",
    "print(\"  '!' == visitor converted\")\n",
    "\n",
    "for i in range(NUM_USERS):\n",
    "\n",
    "  user_id = 'user%d' % i\n",
    "  print(\".\", end=\"\")\n",
    "  enabled = optimizely_sdk.is_feature_enabled('home_product_recs', user_id=user_id)\n",
    "\n",
    "  if enabled:\n",
    "    algorithm_type = optimizely_sdk.get_feature_variable_string('home_product_recs', 'algorithm_type', user_id=user_id)\n",
    "\n",
    "    conversion_rate = CONVERSION_RATES.get(algorithm_type, 0.25)\n",
    "\n",
    "    if conversion_rate > random.uniform(0, 1):\n",
    "      optimizely_sdk.track('ProductAdded', user_id)\n",
    "      print(\"!\", end=\"\")\n",
    "\n",
    "\n",
    "print(\" \")\n",
    "print(\"Traffic Simulation Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you see the following text after a few minutes with no warnings or errors then you're good to go to the next step:\n",
    "\n",
    "```\n",
    "Traffic Simulation Done!\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 12. Analyze Your Test Results\n",
    "\n",
    "Optimizely knows the importance of seeing the results of an experiment as soon as possible. That's why we provide results in as real-time as possible. \n",
    "\n",
    "You can see experiment results in the Optimizely Application:\n",
    "\n",
    "- Navigate to the far left \"Experiments\"\n",
    "- Click the \"Results\" button on the far right of the \"Home Page Recommendations\" \n",
    "\n",
    "![View Results](./images/optimizely/view-results.png)\n",
    "\n",
    "Once results have been calculated for your simulated traffic, you will be able to analyze how the different variations performed.\n",
    "\n",
    "In the below results page, you'll see that `variation_1` which corresponds to the default product recommendations had around a 50% conversion rate of people adding products.\n",
    "\n",
    "In contrast, you'll see `variation_2` which corresponds to the AWS Personalize recommendations has lead to around a 75% conversion rate of people adding products!\n",
    "\n",
    "Both of these conversion rates are higher than the baseline conversion rate you observed on your website, but with Optimizely, we can confidently say that not only are product recommendations causing more product click throughs, but the AWS Personalize recommendations are performing the best with statistical confidence!\n",
    "\n",
    "![Optimizely Results Analysis](./images/optimizely/results-analysis.png)\n",
    "\n",
    "Congrats on running your first A/B test end-to-end using Optimizely!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Congrats! 🎉\n",
    "\n",
    "You have now completed the Optimizely workshop. \n",
    "\n",
    "You've seen how to install and leverage Optimizely's experimentation platform to optimize your personalized recommendations for your users, ultimately having a positive impact on your business!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3-final"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}