Basic_AB_Testing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Basic AB Testing.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/meln-ds/Basic-A-B-Testing/blob/master/Basic_AB_Testing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zxjxGS2gKAAG",
        "colab_type": "code",
        "outputId": "f4987ad8-3951-4cb4-94be-14ac14b216ed",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 309
        }
      },
      "source": [
        "%matplotlib inline\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "from scipy.stats import ttest_ind\n",
        "\n",
        "\n",
        "url = 'https://raw.githubusercontent.com/meln-ds/Basic-A-B-Testing/master/marketing_new.csv'\n",
        "marketing = pd.read_csv(url)\n",
        "\n",
        "# Inspect the columns available\n",
        "marketing.head()"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>user_id</th>\n",
              "      <th>date_served</th>\n",
              "      <th>marketing_channel</th>\n",
              "      <th>variant</th>\n",
              "      <th>converted</th>\n",
              "      <th>language_displayed</th>\n",
              "      <th>language_preferred</th>\n",
              "      <th>age_group</th>\n",
              "      <th>date_subscribed</th>\n",
              "      <th>date_canceled</th>\n",
              "      <th>subscribing_channel</th>\n",
              "      <th>is_retained</th>\n",
              "      <th>DoW</th>\n",
              "      <th>channel_code</th>\n",
              "      <th>is_correct_lang</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>a100000029</td>\n",
              "      <td>2018-01-01</td>\n",
              "      <td>House Ads</td>\n",
              "      <td>personalization</td>\n",
              "      <td>True</td>\n",
              "      <td>English</td>\n",
              "      <td>English</td>\n",
              "      <td>0-18 years</td>\n",
              "      <td>2018-01-01</td>\n",
              "      <td>NaN</td>\n",
              "      <td>House Ads</td>\n",
              "      <td>True</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>a100000030</td>\n",
              "      <td>2018-01-01</td>\n",
              "      <td>House Ads</td>\n",
              "      <td>personalization</td>\n",
              "      <td>True</td>\n",
              "      <td>English</td>\n",
              "      <td>English</td>\n",
              "      <td>19-24 years</td>\n",
              "      <td>2018-01-01</td>\n",
              "      <td>NaN</td>\n",
              "      <td>House Ads</td>\n",
              "      <td>True</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>a100000031</td>\n",
              "      <td>2018-01-01</td>\n",
              "      <td>House Ads</td>\n",
              "      <td>personalization</td>\n",
              "      <td>True</td>\n",
              "      <td>English</td>\n",
              "      <td>English</td>\n",
              "      <td>24-30 years</td>\n",
              "      <td>2018-01-01</td>\n",
              "      <td>NaN</td>\n",
              "      <td>House Ads</td>\n",
              "      <td>True</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>a100000032</td>\n",
              "      <td>2018-01-01</td>\n",
              "      <td>House Ads</td>\n",
              "      <td>personalization</td>\n",
              "      <td>True</td>\n",
              "      <td>English</td>\n",
              "      <td>English</td>\n",
              "      <td>30-36 years</td>\n",
              "      <td>2018-01-01</td>\n",
              "      <td>NaN</td>\n",
              "      <td>House Ads</td>\n",
              "      <td>True</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>a100000033</td>\n",
              "      <td>2018-01-01</td>\n",
              "      <td>House Ads</td>\n",
              "      <td>personalization</td>\n",
              "      <td>True</td>\n",
              "      <td>English</td>\n",
              "      <td>English</td>\n",
              "      <td>36-45 years</td>\n",
              "      <td>2018-01-01</td>\n",
              "      <td>NaN</td>\n",
              "      <td>House Ads</td>\n",
              "      <td>True</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>Yes</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Unnamed: 0     user_id date_served  ...  DoW channel_code  is_correct_lang\n",
              "0           0  a100000029  2018-01-01  ...  0.0          1.0              Yes\n",
              "1           1  a100000030  2018-01-01  ...  0.0          1.0              Yes\n",
              "2           2  a100000031  2018-01-01  ...  0.0          1.0              Yes\n",
              "3           3  a100000032  2018-01-01  ...  0.0          1.0              Yes\n",
              "4           4  a100000033  2018-01-01  ...  0.0          1.0              Yes\n",
              "\n",
              "[5 rows x 16 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XtMyzMnA6QD_",
        "colab_type": "text"
      },
      "source": [
        "#### **1. Test allocation**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XhbrW7T6KXer",
        "colab_type": "code",
        "outputId": "65fa900d-ade7-46c6-d100-e2829266fb1f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "# For this A/B testing, users whose marketing channel is email were segregated into two groups: control and treatment group.\n",
        "# Treatment group received a personalized email. The classification is stored in the variant column\n",
        "\n",
        "email_df = marketing[marketing['marketing_channel'] == 'Email']\n",
        "user_alloc = email_df.groupby('variant')['user_id'].nunique()\n",
        "user_alloc"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "variant\n",
              "control            270\n",
              "personalization    284\n",
              "Name: user_id, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5L3e3sXd8AGx",
        "colab_type": "code",
        "outputId": "5e871894-a484-46fb-8aac-b51dd9d78c2d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 336
        }
      },
      "source": [
        "# Visualize the user allocation. I cleaned up the viz a bit, remove the frame and move the data labels into within the bars themselves\n",
        "plt.figure(figsize=(4,5))\n",
        "bars = plt.bar(['control','treatment'],user_alloc, width=0.5)\n",
        "plt.title('Number of users in control vs treatment group')\n",
        "plt.ylabel('# of users')\n",
        "plt.box(False)\n",
        "plt.tick_params(left=False, bottom=False, labelleft=False)\n",
        "plt.xticks(rotation=0)\n",
        "for bar in bars: # direct label each bar with Y axis values\n",
        "    plt.gca().text(bar.get_x() + bar.get_width()/2, bar.get_height() - 15, str(int(bar.get_height())), \n",
        "                 ha='center', color='w', fontsize=11)\n",
        "\n",
        "plt.show()\n"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAASgAAAE/CAYAAAD11F46AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAX50lEQVR4nO3deZRdVZ238eeXgZABEgmDQELCJGMQ\nBVvBaAMyyCAoMtgNioDIS9sO9FIRGhGVVtq2G+W1W6FfX6OggKDYCiqDjAFZYSZMCoZAMBHIRMhA\nptr9x9kVTsqaMtxb+xbPZ6276p5x73P3Pt86+9x7qyKlhCSVaEBfV0CSumJASSqWASWpWAaUpGIZ\nUJKKZUBJKlZxARURkyLigj4qOyLiBxExLyKm9EUd1lRE/CYiTurreqxPEXF+RFze1/VQ3+sxoCJi\nekS8GBHDa/M+FhG3NbRmfWMicBAwJqX0N31dmd5IKR2aUvphX9ejXauES+7XBzZw//tFxPON2n8v\nyr8tIj7WV+WvL729ghoIfLqRFWmEiBi4hpuMA6anlBY1oj69ERGD+qrsZshXqcVduXfU39thTfXZ\n65FS6vYBTAe+AMwFRuV5HwNuy8/HAwkYVNvmNuBj+flHgbuAi4D5wDRg3zx/BvAicFJt20nA94Cb\ngFeA24FxteU752VzgT8Ax3XY9rvAr4FFwIGdHM9WwC/z9k8Dp+X5pwKvAiuBhcCXO9n2fODy2vRq\nx56PaVqu9zPACbV1TwGeAOYBN3Q4pgR8Angqbxf59XoRWABMBXbvon06vtaTgW/mcp4BDu2mbccC\nPwdeAuYA38nzBwDnAs/mOvwIGNnhmE8CngNmA/+cl70XWAYsz6/hw7U6/kvuB0uAHbpqh85e5w51\nfgI4ojY9KNf/rcCGwOX5WOYD9wJbdLKPy4C2XJeFwOdrx3VqPq478rrvAO7O+3sY2K+2n5NzfV7J\n7X56nj8877st739hPt7zgatzHV/J7fom4Oz8Os8ADq7tfyTwfWAW8GfgAmBgT22dX+uVVP15YXu7\ndvI6fCS38Rzgi1Tn+oG1Nrgm13UB1Tk/BPgWMDM/vgUMqdenw/4TsENvzusu+2gvA+pAqo58wVoG\n1IrcmAPzi/wc8J/5gA/OFR5RO5BXgHfn5d9uP/Dc8DPyvgYBb6E6QXatbfsy8E6qk2zDTo7nDuC/\nqDrznlSd+4CuXuTeBlSu2wJgp7xsS2C3/PwoqpNwl7zuucDdHRryJmATYChwCHA/MIoqrHYBtuxl\nQC0HTsuv9Rm5I0Un2w2kOuEuynXfEJhYC9Onge2AEbntL+twzP+d6/pmYCmwS1fhkuv4HLBbPv7B\nPbTDX+2jtq/zgB/Xpg8HnsjPTwd+BQzLx7cXsHF3/bqTtvxRfj2GAltTnbyHUfWng/L0ZrWyt89t\n9LfAYuCtedl+wPOd9J9Xc/sOymU9A/xzfk1OA56prX8tcEmuz+bAFF4LwW7but4vujj+XanCayKw\nAVXQLWf1gFoOvD8f+1DgK8A9uS6bUQX3V9cgoDo9r9dXQO1OdfJvxpoH1FO1ZRPy+lvU5s0B9qwd\nyJW1ZSOofhuMBY4H7uxQv0uAL9W2/VE3xzI272uj2ryvA5PWU0DNBz4IDO2w3W+AU2vTA6g687ha\nQx5QW34A8Eeq394Demifjq/107Vlw/K+39jJdvtQhcKgTpb9DviH2vROVJ11UO2Yx9SWTwE+1ENA\nfWUN2uGv9lFbbweqjj4sT/8YOC8/P4XqpNmjt/26k7bcrjbvLHIw1+bdQO2Kv8OyXwCfzs/3o/OA\nuqk2/T6qkGi/Ktoo12EUsAVV8A+trf93wK29aWt6DqjzgCs6bL+M1QPqjg7b/Ak4rDZ9CNUtkfb6\n9BRQnZ7X3bVTr+8FpJQeBa6jGu6tqRdqz5fk/XWcN6I2PaNW7kKqYcBWVPeI3h4R89sfwAnAGzvb\nthNbAXNTSq/U5j1L9ZtynaTqvtXxwP8BZkXE9RGxc148Dvh2rc5zqX7r1sutH/MtwHeorjJfjIhL\nI2LjXlblL7X9LM5PR3Sy3ljg2ZTSik6WbUX1urR7liqctuisHKqw7ayMunq7rHU7pJSephpWvS8i\nhgFHAj/Jiy+jCpArI2JmRHwjIgb3tM9u6jkOOLZDf5tIdXVMRBwaEfdExNy87DBg0x7237Hfz04p\nraxNQ/VajqO6qppVK/sSqquXdr1t685sxep9bjHVhUJdx3Ops36xVS/LW21/Hc7rLq3pzcovUV1S\n1jtS+w3lYbV59cBYG2Pbn0TECKqhz0yqA7w9pTSq9hiRUjqjtm3qZr8zgU0iYqPavG2oxve9sYhu\njjOldENK6SCqDvwk1TCIXO/TO9R7aErp7q7qnVK6OKW0F9Wl+JuAz/Wyjr01A9imi5ufM6lOkHbb\nUA3TX+hk3Y66ev3r89e1Ha6gupo4Cng8hxYppeUppS+nlHalus95BNV9lrWt5wyqK6h6uw1PKV0Y\nEUOAn1ENjbZIKY2iuvcZPey/t2ZQXUFtWit745TSbr3cvqfyZwFj2iciYigwuod9dNYvZubnq50b\nEdFZBnR1XndpjQIqd4SrgE/V5r1E1bFOjIiBEXEK1bh8XRwWERMjYgPgq8A9KaUZVFdwb4qID0fE\n4Px4W0Ts0sv6z6AaAnw9IjaMiD2obor29m3xh4B3R8Q2ETGS6uYmABGxRUQclT+OsZTq0r0tL/4e\ncHZE7JbXHRkRx3ZVSD6mt+ff/ouo7lu0dbX+WppC1UkvjIjh+fV4Z152BXBmRGybO9LXgKu6uNrq\n6AVgfHfv1K2HdriS6t7lGbx29URE7B8RE/K7twuohqVdvW4vUN1j687lVFdqh+S+vWH++MAYqvs2\nQ6iGySsi4tBcp/r+R+d+ssZSSrOAG4F/j4iNI2JARGwfEX/by130dHzXUB3bvvk8O5/XwrUrVwDn\nRsRmEbEp1TCxvc0eBnaLiD0jYsO8v466Oq+7tDZv936F6n5L3WlUv+HnUN0IvbvjRmvoJ1RXa3Op\nbnSeCJCHBAcDH6JK3r8A/0rVUXrr76juN8ykugn5pZTSzb3ZMKV0E1VAP0J1E/u62uIBwD/l/c6l\numl6Rt7u2lzPKyNiAfAocGg3RW1MdfU1j9feZfm3Xh1dL+Vhxfuo7uk8BzxPNUQF+P9Uw6U7qG7i\nvgp8spe7vjr/nBMRD3Sz3rq0wyzg91RXSVfVFr2R6sRbQDUMvD0fR2e+TnWyzY+Iz3ZRzgyqq7Rz\nqIJoBlU/H5D74qeAn1K1099TvSvZvu2TVCf0tFzGmgyF2n2EKggfz2VcQx5e9sK3gWPyh44v7uTY\nHqNq0yupflEtpHoncWk3+7wAuI+q/08FHsjzSCn9kSobbqZ6N3pyJ9t3el53p/2Ov6TXsXylPB/Y\nMaX0TAP2P4nqTYNz12S74j8wJ6kxIuJ9ETEs35b4JtVV0fS+rdXqDCjp9esoXvvQ5Y5UHxUpakjl\nEE9SsbyCklQsA0pSsQwoScUyoCQVy4CSVCwDSlKxDChJxTKgJBXLgJJULANKUrEMKEnFMqAkFcuA\nklQsA0pSsQwoScXy3zurXxr/hev9Q2c10y88vKd/iFAkr6AkFcuAklQsA0pSsQwoScUyoCQVy3fx\npLUwathgLjpuT7YZPYzlK9uYPnsx51w7lbmLlnHs3mM4deK2tLUlVrQlvnrd49w7fd5q23/jmD04\nbu+x7Hreb1m8bGUfHUX5DChpLaQEl9zxJ+6ZNheAsw/dmbPeuzNf/80TnHfEruz/zduYvXAZB+6y\nOV/7wAQOuuiOVdu+Z5fN8b+99Y5DPGktvLxk+apwAnhwxny2HjWUACKC4UOq3/0bDx3MrJdfXbXe\nqGGD+fR7duSC6x5vdpVbkldQ0jqKgBPfPo6bn3iBeYuXc87Pp3LdJyey4NUVDAj40KX3rFr3q0ft\nzkU3PcUrS1f0YY1bhwElraMvH7kbi5et4Ie/n86IIYM4ad/xHPWdu5g2exGHT9iSSz68F+/91p0c\nPmFLlq1s49Y/vNjXVW4ZDvGkdXDOYbswfvRw/vEnD5ISvGvHTVmwZDnTZi8C4Pqpsxi3yXA2Gb4B\n79huNPtuP5rJZ+3P5LP2B+DGM9/NDpuP6MtDKJpXUNJa+twhOzFh65GcPGkKy1a2ATBj3mJ233pj\nRg/fgDmLlrHPdqNZuHQ5cxct44v/8yhf/J/Xtp9+4eEcfNEdvovXjZYOKL8Q+ppW/TJoq9px8xF8\nYv8d+NNLC/n5GfsCMGPeEk6/7H6+d/s0rjp9H5avbGPpijbOuPyBPq5t62rpgJL6ylMvLmT8F67v\ndNn3Jz/D9yc/0+M+utper/EelKRiGVCSimVASSqWASWpWN4k70ZXXwgdP3o4F7x/t1XrjR4xhJde\nWcoR/3cyAG8ZO4qvHT2BIYMG8Py8JZx51UPMWbSsrw5DalkGVDe6+kLoWT97hMMunrxqvUs/vNeq\nb6tHwEXH78lnr36Y+56dxz8esANnHbozn7/mkT45BqmVOcTrRldfCK0bPXwD3rXjZlz74PMATNh6\nJEtXtHHfs1Vg/fieZzl8wpbNq7TUjxhQvVT/Qmjd0W8dw51PvcTshdUQbqtRQ/nz/CWrls9bvJwB\nEYwcOrip9ZX6AwOql+pfCK07du8x/PS+GX1SJ6m/M6B6oeMXQtu9ZewoRg0dzK1/eGnVvJnzl6w2\nDHzDsMG0pcTLS5Y3s8pSv2BA9aD9C6Efv+y+VV8IbXfs3mP5+YN/ZmXba6k19c8vs+HgAew97g0A\nnPCOcfx66qym1lnqL3wXrxvdfSF0yKABHPHmLfnAf9612jYpwZlXPczXjt6dIYMG8vy8xZx51UN9\nUX2p5RlQ3ejuC6FLV7Sxx/k3drrsgefm8d5v3dnIqkmvCw7xJBXLgJJULANKUrEMKEnFMqAkFcuA\nklQsA0pSsQwoScUyoCQVy4CSVCwDSlKxDChJxTKgJBXLgJJULANKUrEMKEnFMqAkFcuAklQsA0pS\nsQwoScUyoCQVy4CSVCwDSlKxDChJxTKgJBXLgJJULANKUrEMKEnFMqAkFcuAklQsA0pSsQwoScUy\noCQVy4CSVCwDSlKxDChJxTKgJBXLgJJULANKUrEMKEnFMqAkFcuAklQsA0pSsQwoScUyoCQVy4CS\nVCwDSlKxDChJxTKgJBXLgJJULANKUrEMKEnFMqAkFcuAklQsA0pSsQwoScUyoCQVy4CSVCwDSlKx\nDChJxTKgJBXLgJJULANKUrEMKEnFMqAkFcuAklQsA0pSsQwoScUyoCQVy4CSVCwDSlKxGh5QEfHp\niNg4Kt+PiAci4uBGlyup9TXjCuqUlNIC4GDgDcCHgQubUK6kFteMgIr88zDgspTSY7V5ktSlZgTU\n/RFxI1VA3RARGwFtTShXUosb1MidR0QA5wGbAdNSSosjYjRwciPLldQ/NDSgUkopIn6dUppQmzcH\nmNPIciX1D80Y4j0QEW9rQjmS+pmGXkFlbwdOjIjpwCKqG+QppbRHE8qW1MKaEVCHNKEMSf1Qw4d4\nKaVngbHAAfn54maUK6n1NeOT5F8CzgLOzrMGA5c3ulxJra8ZVzIfAI6kuv9ESmkmsFETypXU4poR\nUMtSSglIABExvAllSuoHmhFQP42IS4BREXEacDPw300oV1KLa/i7eCmlb0bEQcACYCfgvJTSTY0u\nV1Lra3hA5SHdLSmlmyJiJ2CniBicUlre6LIltbZmDPHuAIZExNbAb6n+3MqkJpQrqcU15c+tpJQW\nA0cD300pHQvs1oRyJbW4pgRUROwDnABcn+cNbEK5klpcMwLqM1Qf0rw2pfRYRGwH3NqEciW1uGa8\ni3c7cHttehrwqUaXK6n1NeNdvFvJH9KsSykd0OiyJbW2Zvw1g8/Wnm8IfBBY0YRyJbW4Zgzx7u8w\n666ImNLociW1vmYM8TapTQ4A9gJGNrpcSa2vGUO8+6nuQQXV0O4Z4NQmlCupxTVjiLdto8uQ1D/5\nly0lFcuAklSshgVURLwz/xzSqDIk9W+NvIK6OP/8fQPLkNSPNfIm+fKIuBTYOiIu7rgwpeTXXSR1\nq5EBdQRwINX/xev4YU1J6lHDAiqlNBu4MiKeSCk93KhyJPVfzXgXb05EXBsRL+bHzyJiTBPKldTi\nmhFQPwB+CWyVH7/K8ySpW80IqM1TSj9IKa3Ij0nAZk0oV1KLa0ZAzY6IEyNiYH6cCMxpQrmSWlwz\nAuoU4DjgL8As4Bjg5CaUK6nFNePLws8CRza6HEn9j9/Fk1QsA0pSsQwoScVqeEBFxLm15/5lA0m9\n1sg/t3JW/o/Cx9Rm+5cNJPVaI9/FexI4FtguIu7M06MjYqeU0h8aWK6kfqKRQ7z5wDnA08B+wLfz\n/C9ExN0NLFdSP9HIK6hDgPOA7YH/AB4BFqWU/JCmpF5p2BVUSumclNJ7gOnAZcBAYLOImBwRv2pU\nuZL6j2b8X7wbUkr3AfdFxBkppYkRsWkTypXU4hr+MYOU0udrkx/N82Y3ulxJra+pH9T0L2tKWhN+\nklxSsQwoScUyoCQVy4CSVCwDSlKxDChJxTKgJBXLgJJULANKUrEMKEnFMqAkFcuAklQsA0pSsQwo\nScUyoCQVy4CSVCwDSlKxDChJxTKgJBXLgJJULANKUrEMKEnFMqAkFcuAklQsA0pSsQwoScUyoCQV\ny4CSVCwDSlKxDChJxTKgJBXLgJJULANKUrEMKEnFMqAkFcuAklQsA0pSsQwoScUyoCQVy4CSVCwD\nSlKxDChJxTKgJBXLgJJULANKUrEMKEnFMqAkFcuAklQsA0pSsQwoScUyoCQVy4CSVCwDSlKxDChJ\nxTKgJBXLgJJULANKUrEMKEnFMqAkFcuAklQsA0pSsQwoScUyoCQVy4CSVCwDSlKxDChJxTKgJBXL\ngJJULANKUrEMKEnFMqAkFcuAklQsA0pSsQwoScUyoCQVy4CSVCwDSlKxDChJxTKgJBXLgJJULANK\nUrEMKEnFMqAkFcuAklQsA0pSsQwoScUyoCQVy4CSVCwDSlKxDChJxTKgJBXLgJJULANKUrEMKEnF\nMqAkFcuAklQsA0pSsQwoScUyoCQVy4CSVCwDSlKxDChJxTKgJBXLgJJULANKUrEMKEnFMqAkFcuA\nklSsSCn1dR0kqVNeQUkqlgElqVgGlKRiGVCSimVASSqWASWpWAbUehAR4yPi79dyu0cbUSf1LCJG\nRcQ/rMf9fSYihq2v/fV1OSUwoNaP8UCnARURg5pbFa2BUcBfBdQ6tNlngGYER7PK6XMGFBARH4mI\nRyLi4Yi4LF/Z3JLn/S4itsnrTYqIiyPi7oiYFhHH5F1cCLwrIh6KiDMj4qMR8cuIuAX4XVT+LSIe\njYipEXF8nx2s6i4Ets/tdm9E3BkRvwQeB4iIEyNiSl5+SUQMzPO/GxH3RcRjEfHlPO9TwFbArRFx\na563MLf7YxFxc0T8TUTclvvOkXmdgXmde3N/Oz3P3y+ve01EPBkRP8796K/K6ddSSq/rB7Ab8Edg\n0zy9CfAr4KQ8fQrwi/x8EnA1VbDvCjyd5+8HXFfb50eB54FN8vQHgZuAgcAWwHPAllRXXo/29Wvw\nen3UX//chouAbfP0LrkfDM7T/wV8pL2P5J8DgduAPfL09PZ+lKcTcGh+fi1wIzAYeDPwUJ7/ceDc\n/HwIcB+wba7Py8CY3N9+D0zsrJz+/HD4AQcAV6eUZgOklOZGxD7A0Xn5ZcA3auv/IqXUBjweEVt0\ns9+bUkpz8/OJwBUppZXACxFxO/A24JH1eSBaZ1NSSs/k5+8B9gLujQiAocCLedlxEfFxYBDVL5pd\n6bwtlwG/zc+nAktTSssjYipVOAIcDOxRuxofCeyYt52SUnoeICIeyttMXvfDbB0G1JpbWnse3ay3\nqNEV0XpXb7MAfphSOru+QkRsC3wWeFtKaV5ETAI27GJ/y1O+5AHayH0npdRWu88VwCdTSjd0KGc/\nVu9rK3kdnq/eg4JbgGMjYjRARGwC3A18KC8/Abizh328AmzUzfI7gePz/YbNgHcDU9ap1lofumu3\n3wHHRMTmUPWLiBgHbEwVZC/nK+hDe7m/rtwAnBERg3M5b4qI4etQ737ldZfIHaWUHouIfwFuj4iV\nwIPAJ4EfRMTngJeAk3vYzSPAyoh4mOo+1bwOy68F9gEeprov8fmU0l8iYvz6Og6tuZTSnIi4K3/U\nYwnwQm3Z4xFxLnBjRAwAlgOfSCndExEPAk8CM4C7aru8FPhtRMxMKe3fy2r8P6qh2wNRjSVfAt7f\nwzZrU05L8s+tSCqWQzxJxTKgJBXLgJJULANKUrEMKEnFMqAkFcuAklQsA0pSsf4XUxDskKgdP9oA\nAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 288x360 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XSm7L8-bKl2F",
        "colab_type": "text"
      },
      "source": [
        "> As we can observe there isn't a big difference between the two groups. However, in real life, experimenters will need to bear in mind that having the same numbers of users in these groups may not guarantee a meaningful results. Randomness for instance is very important in allocating your users and without it your groups might be very skewed and not an accurate representation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XeotK4TTDU0U",
        "colab_type": "text"
      },
      "source": [
        "#### **2. Lift and Statistical Significance**\n",
        "\n",
        "> Lift is the first question to ask in A/B testing. It is the % difference between your control page and a successful test variant. In this context, the question is - Is my conversion rate under the treatment group higher than the control group, and by how much?\n",
        "\n",
        "> \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2lHm5FDwNTbn",
        "colab_type": "code",
        "outputId": "ed98ca2d-8048-4426-eb59-bfbecf0049f3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# Extract conversion results of control and treatment groups\n",
        "\n",
        "subscriber_df = email_df.groupby(['user_id','variant'])['converted'].max().unstack(level=1)\n",
        "control = subscriber_df['control'].dropna()\n",
        "treatment = subscriber_df['personalization'].dropna()\n",
        "\n",
        "# Calculate lift\n",
        "control_conv = np.mean(control)\n",
        "treatment_conv = np.mean(treatment)\n",
        "lift = (treatment_conv - control_conv)/control_conv\n",
        "\n",
        "print('Control conversion rate: '+'{:.2%}'.format(control_conv)) \n",
        "print('Treatment conversion rate: '+'{:.2%}'.format(treatment_conv)) \n",
        "print('Lift: '+'{:.2%}'.format(lift)) "
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Control conversion rate: 28.15%\n",
            "Treatment conversion rate: 39.08%\n",
            "Lift: 38.85%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GGDyjHXdMH4L",
        "colab_type": "text"
      },
      "source": [
        "> 38.85% seems to be a pretty good result, but how certain are we that the result obtained under our treatment group did not just happen by chance? Statistical significance is a validator of our calculated lift. To measure this, we employ the scipy stats's independent t-test, with significance level = 5%. I'm not going into details of what t-test is exactly, but there are plenty of resources online that you can look into (Khan Academy AP Stats is a good start). \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9L2G6FDdLiNU",
        "colab_type": "code",
        "outputId": "c88733be-5cb8-42de-d067-3bceb80e8bb2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "ttest = ttest_ind(control,treatment)\n",
        "print(ttest)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Ttest_indResult(statistic=-2.7343299447505074, pvalue=0.006451487844694175)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9vy5m2UVPN2H",
        "colab_type": "text"
      },
      "source": [
        "> Our p-value is less than our significance level of 5%, which means we can reject the null hypothesis, and that our result is statistically significant.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B5DlswQdPjh9",
        "colab_type": "text"
      },
      "source": [
        "#### 3. A/B testing and segmentation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3rtVYZrnuMDj",
        "colab_type": "text"
      },
      "source": [
        "> Overall, the A/B testing result indicates that personalized emails lead to a higher conversion rate compared to a normal email. However, is this the case for all customer segments ? Bear in mind that different customer segments may react differently to a product functionality or marketing tactics.\n",
        "\n",
        "> In this section, we will break down the A/B testing result further to different customer segments. It would not be ideal to rewrite all the code from scratch for each segment, so we will define a function that can be reuseable, knowing that our dataset format will remain the same."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RKsAPXTyMq1S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Build a function for calculating lift\n",
        "\n",
        "def AB_lift(control,treatment):\n",
        "  control_conv = np.mean(control)\n",
        "  treatment_conv = np.mean(treatment)\n",
        "  return (treatment_conv - control_conv)/control_conv\n",
        "  \n",
        "# Build a function for calculating p-value\n",
        "\n",
        "def AB_ttest(control,treatment):\n",
        "  return ttest_ind(control,treatment)[1]\n",
        "\n",
        "# Build a function to go through each customer subsegment from a segment column specified by the user\n",
        "\n",
        "def AB_test(segment):\n",
        "  # Build a for loop for each subsegment\n",
        "  for subsegment in np.unique(marketing[segment].values):\n",
        "    print(subsegment)\n",
        "\n",
        "    # Filter users reached via email and belonging to relevant subsegment\n",
        "    marketing_filtered = marketing[(marketing['marketing_channel'] == 'Email') & (marketing[segment] == subsegment)]\n",
        "\n",
        "    # Repeat the same steps in section 2\n",
        "    subscriber_df = marketing_filtered.groupby(['user_id','variant'])['converted'].max().unstack(level=1)\n",
        "    control = subscriber_df['control'].dropna()\n",
        "    treatment = subscriber_df['personalization'].dropna()\n",
        "\n",
        "    # Return lift and p-value as a tuple\n",
        "    print('Lift: '+ str(AB_lift(control,treatment)) + ', '+'p-value: '+ str(AB_ttest(control,treatment)))\n",
        "\n",
        "\n",
        "\n",
        "  \n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w-WQpBP40qRq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        },
        "outputId": "01cda5c3-8364-462a-bfce-4716118a7a6e"
      },
      "source": [
        "# Run our function on language displayed\n",
        "\n",
        "AB_test('language_displayed')"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Arabic\n",
            "Lift: 0.4999999999999999, p-value: 0.5795840000000001\n",
            "English\n",
            "Lift: 0.3900293255131965, p-value: 0.026991701290720815\n",
            "German\n",
            "Lift: -0.016203703703703685, p-value: 0.8494394170062678\n",
            "Spanish\n",
            "Lift: 1.6666666666666667, p-value: 0.040156718110477524\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "olDOLfO-1-3b",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# It looks like our personalized emails worked very well for Spanish audience, but not so much for German audience."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KdKt9F5p02Mg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 255
        },
        "outputId": "99fc7c85-c1c9-4b5f-cd85-63787d005d59"
      },
      "source": [
        "# And another one on age group\n",
        "\n",
        "AB_test('age_group')"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0-18 years\n",
            "Lift: 1.2140000000000002, p-value: 0.0038724494391297226\n",
            "19-24 years\n",
            "Lift: 1.0624151967435547, p-value: 0.0030623836114689134\n",
            "24-30 years\n",
            "Lift: 1.611851851851852, p-value: 0.00018743381094867337\n",
            "30-36 years\n",
            "Lift: -1.0, p-value: 0.0023238487431765137\n",
            "36-45 years\n",
            "Lift: -0.8522727272727273, p-value: 0.01797568600978829\n",
            "45-55 years\n",
            "Lift: -0.7222222222222223, p-value: 0.043062339688201196\n",
            "55+ years\n",
            "Lift: -1.0, p-value: 0.0016358623456360435\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0pTDGF-U11-v",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Results suggest our younger audience responded more positively to the personlized emails, whereas the older audience were less welcoming.\n",
        "# Perhaps they thought it sounded like an invasion of privacy ?\n",
        "# Either way, segmentation helps the marketing team drill down further and tweak their experiments accordingly."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DbjILaOU2oAm",
        "colab_type": "text"
      },
      "source": [
        "**Potential extensions:**\n",
        "\n",
        "The AB test analysis above is still fairly basic, and there is a lot of room for improvements. For example, one can improve the AB_test function to include a bar chart of the lift values for the segments. This should be relatively easy. There are also more AB tests that the team could run, for example, sending emails on a weekday vs a weekend. The result can be analyzed in a similar fashion.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GLzpF6ln2iFD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}