docs-samples/data-science/ai-samples/python/AIsample - Uplift Modelling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "af6c52ee-d000-4328-83c3-897c06ca276b",
   "metadata": {},
   "source": [
    "# Create, evaluate, and score an uplift model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cfd8e82-be33-4c90-aac7-47d397862ebc",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## Introduction\n",
    "\n",
    "This notebook demonstrates the data science work flow with an end-to-end example. The notebook creates, trains, and evaluates an uplift model that refers to a collection of machine learning techniques. These techniques are used to estimate the incremental effects of a treatment on the behavior of an individual or a sub-group. \n",
    "\n",
    "This notebook covers these topics:\n",
    "\n",
    "1. Load the data\n",
    "2. Understand and process the data through exploratory data analysis\n",
    "3. Train a machine learning model with an uplift model\n",
    "4. Save the final machine learning model\n",
    "5. Load the machine learning model for scoring and making predictions\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "- [Add a lakehouse](https://aka.ms/fabric/addlakehouse) to this notebook. You will download data from a public blob, then store the data in the lakehouse resource."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58090d36-f361-4a8e-8a93-80fba833a43e",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "### What is uplift modeling?\n",
    "\n",
    "Uplift modeling relies on a set of causal inference techniques that utilize machine learning models to identify the most suitable treatment, from a group of possible treatments, that leads to the most desirable outcome. Study of both a treatment group - individuals who undergo a treatment - and a control group - individuals who do not undergo that treatment - determines outcome desirability. This approach classifies individuals into four main subgroups within a population:\n",
    "\n",
    "- **Persuadables**: This group of individuals will respond positively only when given the treatment.\n",
    "- **Sleeping-dogs**: This group of individuals will show a strong negative response to the treatment.\n",
    "- **Lost Causes**: This group of individuals will never achieve the desired outcome, even with the treatment.\n",
    "- **Sure Things**: This group of individuals will achieve the outcome regardless of whether they receive the treatment or not.\n",
    "\n",
    "For its primary objective, uplift modeling identifies the **\"persuadables\"**, since a focus on them can lead to a more favorable outcome. Conversely, efforts directed toward the **\"lost causes\"** and **\"sure things\"** groups would become ineffective, and the **\"sleeping dogs\"** group could potentially result in negative outcomes. Uplift modeling finds practical applications in various fields\n",
    "\n",
    "- marketing\n",
    "- healthcare\n",
    "- social sciences\n",
    "- etc.\n",
    "\n",
    "where it optimizes decision-making and it enhances the effectiveness of interventions.\n",
    "\n",
    "### Uplift modeling\n",
    "\n",
    "The main techniques used for uplift modeling include:\n",
    "\n",
    "- Meta Learner, to predict the difference between an individual's behavior when the individual undergoes a treatment, and when not undergoing a treatment.\n",
    "\n",
    "- Uplift Tree, a tree-based algorithm that combines both treatment/control group assignment information and response information directly into decisions about splitting criterion for a node.\n",
    "\n",
    "- Neural Network-based Model, which usually works with observational data. It uses deep learning to help determine the distribution of a latent variable. The latent variable represents the co-founder in the uplift modeling.\n",
    "\n",
    "### Uplift modeling offers benefits\n",
    "\n",
    "- For marketing applications, uplift modeling can help identify persuadables who can potentially be swayed to try the treatment. The outreach to identify persuadables could involve a coupon or an online advertisement, for example.\n",
    "- For medical treatments, uplift modeling can help measure how a particular treatment can impact distinct groups. This allows for optimized target selection, to maximize the impact.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10b9dec8-622d-4f6c-8b4c-23dc3505ede9",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## Step 1: Load the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84827e55-bd4e-4ec4-b29f-54d866ac71af",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### Dataset\n",
    "\n",
    "The Criteo AI Lab created the dataset. That dataset has 13M rows. Each row represents one user. Each row has twelve features, a treatment indicator, and two binary labels that include visit and conversion.\n",
    "\n",
    "|f0|f1|f2|f3|f4|f5|f6|f7|f8|f9|f10|f11|treatment|conversion|visit|\n",
    "|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n",
    "\n",
    "- **f0 - f11**: feature values (dense, floating values)\n",
    "- **treatment**: whether or not a user was randomly targeted for treatment (e.g., advertising) (1 = treatment, 0 = control)\n",
    "- **conversion**: whether a conversion occurred (e.g., made a purchase) for a user (binary, label)\n",
    "- **visit**: whether a visit occurred (e.g., visit the online store) for a user (binary, label)\n",
    "\n",
    "#### Citation\n",
    "\n",
    "- Dataset homepage: https://ailab.criteo.com/criteo-uplift-prediction-dataset/\n",
    "\n",
    "The dataset used for this notebook requires this BibTex citation:\n",
    "\n",
    "    @inproceedings{Diemert2018,\n",
    "    author = {{Diemert Eustache, Betlei Artem} and Renaudin, Christophe and Massih-Reza, Amini},\n",
    "    title={A Large Scale Benchmark for Uplift Modeling},\n",
    "    publisher = {ACM},\n",
    "    booktitle = {Proceedings of the AdKDD and TargetAd Workshop, KDD, London,United Kingdom, August, 20, 2018},\n",
    "    year = {2018}\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "961661cc-00ad-461a-a882-13047b7bfe92",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "> [!TIP]\n",
    "> By defining the following parameters, you can apply this notebook on different datasets easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "daaeb770-009e-4924-85d2-57c7b6a874d5",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:19:54.0020297Z",
       "execution_start_time": "2023-09-08T23:19:51.842555Z",
       "livy_statement_state": "available",
       "parent_msg_id": "b2d44fdf-1a04-44c6-8c2c-2996e2c58b4d",
       "queued_time": "2023-09-08T23:19:41.8542719Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": "2023-09-08T23:19:41.9610298Z",
       "spark_jobs": {
        "jobs": [],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 0,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 3
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 3, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "IS_CUSTOM_DATA = False  # if True, dataset has to be uploaded manually by user\n",
    "DATA_FOLDER = \"Files/uplift-modelling\"\n",
    "DATA_FILE = \"criteo-research-uplift-v2.1.csv\"\n",
    "\n",
    "FEATURE_COLUMNS = [f\"f{i}\" for i in range(12)]\n",
    "TREATMENT_COLUMN = \"treatment\"\n",
    "LABEL_COLUMN = \"visit\"\n",
    "\n",
    "EXPERIMENT_NAME = \"aisample-upliftmodelling\"  # MLflow experiment name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb5b5a1c-ff44-4706-9c7d-e51c161930aa",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### Import libraries\n",
    "\n",
    "Before processing, you must import required Spark and SynapseML libraries. You must also import a data visualization library - for example, Seaborn, a Python data visualization library. A data visualization library provides a high-level interface to build visual resources on DataFrames and arrays. Learn more about [Spark](https://spark.apache.org/), [SynapseML](https://aka.ms/AboutSynapseML) and [Seaborn](https://seaborn.pydata.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "723d006f-fc57-4207-a5e5-b604bff6df98",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:20:02.8182102Z",
       "execution_start_time": "2023-09-08T23:19:54.2727167Z",
       "livy_statement_state": "available",
       "parent_msg_id": "3f2ee30d-1c04-40d2-a3af-dd82249dcf21",
       "queued_time": "2023-09-08T23:19:41.8549165Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 0,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 4
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 4, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import os\n",
    "import gzip\n",
    "\n",
    "import pyspark.sql.functions as F\n",
    "from pyspark.sql.window import Window\n",
    "from pyspark.sql.types import *\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.style as style\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "from synapse.ml.featurize import Featurize\n",
    "from synapse.ml.core.spark import FluentAPI\n",
    "from synapse.ml.lightgbm import *\n",
    "from synapse.ml.train import ComputeModelStatistics\n",
    "\n",
    "import mlflow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81d10e4c-eb33-41e1-a3ef-4ee9c4e481b2",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### Download a dataset and upload to lakehouse\n",
    "\n",
    "This code downloads a publicly available version of the dataset, and then stores that data resource in a Fabric lakehouse.\n",
    "\n",
    "> [!IMPORTANT]\n",
    "> **Make sure you [add a lakehouse](https://aka.ms/fabric/addlakehouse) to the notebook before running it. Failure to do so will result in an error.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bd24839-523a-4424-be50-ace09179278e",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:20:04.0712302Z",
       "execution_start_time": "2023-09-08T23:20:03.097957Z",
       "livy_statement_state": "available",
       "parent_msg_id": "a3c13bc2-7945-4ac3-be81-82df2411480c",
       "queued_time": "2023-09-08T23:19:41.8557708Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 0,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 5
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 5, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloaded demo data files into lakehouse.\n"
     ]
    }
   ],
   "source": [
    "if not IS_CUSTOM_DATA:\n",
    "    # Download demo data files into lakehouse if not exist\n",
    "    import os, requests\n",
    "\n",
    "    remote_url = \"http://go.criteo.net/criteo-research-uplift-v2.1.csv.gz\"\n",
    "    download_file = \"criteo-research-uplift-v2.1.csv.gz\"\n",
    "    download_path = f\"/lakehouse/default/{DATA_FOLDER}/raw\"\n",
    "\n",
    "    if not os.path.exists(\"/lakehouse/default\"):\n",
    "        raise FileNotFoundError(\"Default lakehouse not found, please add a lakehouse and restart the session.\")\n",
    "    os.makedirs(download_path, exist_ok=True)\n",
    "    if not os.path.exists(f\"{download_path}/{DATA_FILE}\"):\n",
    "        r = requests.get(f\"{remote_url}\", timeout=30)\n",
    "        with open(f\"{download_path}/{download_file}\", \"wb\") as f:\n",
    "            f.write(r.content)\n",
    "        with gzip.open(f\"{download_path}/{download_file}\", \"rb\") as fin:\n",
    "            with open(f\"{download_path}/{DATA_FILE}\", \"wb\") as fout:\n",
    "                fout.write(fin.read())\n",
    "    print(\"Downloaded demo data files into lakehouse.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bb4bcc6-6c72-4370-9c00-2257432a2861",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Start recording the runtime of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b28508a8-71c5-4351-a0ad-9df05fcf8b8a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:20:04.6942959Z",
       "execution_start_time": "2023-09-08T23:20:04.3545298Z",
       "livy_statement_state": "available",
       "parent_msg_id": "60655387-8bf6-4f65-8df8-743d0e0e1056",
       "queued_time": "2023-09-08T23:19:41.8565475Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 0,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 6
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 6, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Record the notebook running time\n",
    "import time\n",
    "\n",
    "ts = time.time()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9af9f97-a377-4508-af1a-6aad555e8f21",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### Set up the MLflow experiment tracking\n",
    "\n",
    "To extend the MLflow logging capabilities, autologging automatically captures the values of input parameters and output metrics of a machine learning model during its training. This information is then logged to the workspace, where the MLflow APIs or the corresponding experiment in the workspace can access and visualize it. Visit [this resource](https://aka.ms/fabric-autologging) for more information about autologging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab02f4a4-fa06-4e06-992f-82c3e3c731fc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:20:07.5388883Z",
       "execution_start_time": "2023-09-08T23:20:04.9579369Z",
       "livy_statement_state": "available",
       "parent_msg_id": "62bbb9e6-2b00-4fbb-9e53-15d0d004542c",
       "queued_time": "2023-09-08T23:19:41.8575066Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 0,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 7
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 7, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Set up the MLflow experiment\n",
    "import mlflow\n",
    "\n",
    "mlflow.set_experiment(EXPERIMENT_NAME)\n",
    "mlflow.autolog(disable=True)  # Disable MLflow autologging"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3819a24e-6a70-40b0-999b-4b07b93c83d2",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "> [!NOTE]\n",
    "> If you want to disable Microsoft Fabric autologging in a notebook session, call `mlflow.autolog()` and set `disable=True`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfda728c-6966-44e6-9af5-2ed83447aac7",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### Read data from the lakehouse\n",
    "\n",
    "Read raw data from the lakehouse **Files** section and add additional columns for different date parts. The same information will be used to create a partitioned delta table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b4bb2f6-71db-457f-b702-9b9bf8e303c0",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:20:39.312766Z",
       "execution_start_time": "2023-09-08T23:20:07.8106883Z",
       "livy_statement_state": "available",
       "parent_msg_id": "b61e2571-9204-4681-88d9-c2864d64f2d0",
       "queued_time": "2023-09-08T23:19:41.8586182Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [
         {
          "completionTime": "2023-09-08T23:20:38.664GMT",
          "dataRead": 3249688085,
          "dataWritten": 0,
          "description": "Job group for statement 8:\nraw_df = spark.read.csv(f\"{DATA_FOLDER}/raw/{DATA_FILE}\", header=True, inferSchema=True).cache()",
          "displayName": "csv at NativeMethodAccessorImpl.java:0",
          "jobGroup": "8",
          "jobId": 9,
          "killedTasksSummary": {},
          "name": "csv at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 13979593,
          "stageIds": [
           13
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:20:10.676GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:20:10.543GMT",
          "dataRead": 65536,
          "dataWritten": 0,
          "description": "Job group for statement 8:\nraw_df = spark.read.csv(f\"{DATA_FOLDER}/raw/{DATA_FILE}\", header=True, inferSchema=True).cache()",
          "displayName": "csv at NativeMethodAccessorImpl.java:0",
          "jobGroup": "8",
          "jobId": 8,
          "killedTasksSummary": {},
          "name": "csv at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 1,
          "stageIds": [
           12
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:20:10.053GMT",
          "usageDescription": ""
         }
        ],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 2,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 8
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 8, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "raw_df = spark.read.csv(f\"{DATA_FOLDER}/raw/{DATA_FILE}\", header=True, inferSchema=True).cache()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce193d37-2b0d-4839-b9c0-66ddc2a47f89",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## Step 2: Exploratory data analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6490f765",
   "metadata": {},
   "source": [
    "Use the `display` command to view high-level statistics about the dataset. You can also show the Chart views to easily visualize subsets of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b1e69b2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:21:07.843199Z",
       "execution_start_time": "2023-09-08T23:20:39.5768741Z",
       "livy_statement_state": "available",
       "parent_msg_id": "f7d88ddd-571b-46a1-8a04-aae94ed0adec",
       "queued_time": "2023-09-08T23:19:41.8596072Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [
         {
          "completionTime": "2023-09-08T23:21:06.600GMT",
          "dataRead": 0,
          "dataWritten": 0,
          "description": "Job group for statement 9:\ndisplay(raw_df.limit(20))",
          "displayName": "getRowsInJsonString at Display.scala:452",
          "jobGroup": "9",
          "jobId": 11,
          "killedTasksSummary": {},
          "name": "getRowsInJsonString at Display.scala:452",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 0,
          "stageIds": [
           15,
           16
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:06.539GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:21:06.457GMT",
          "dataRead": 3249688085,
          "dataWritten": 18525,
          "description": "Job group for statement 9:\ndisplay(raw_df.limit(20))",
          "displayName": "getRowsInJsonString at Display.scala:452",
          "jobGroup": "9",
          "jobId": 10,
          "killedTasksSummary": {},
          "name": "getRowsInJsonString at Display.scala:452",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 13980092,
          "stageIds": [
           14
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:20:39.847GMT",
          "usageDescription": ""
         }
        ],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 2,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 9
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 9, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.synapse.widget-view+json": {
       "widget_id": "4b4c30fd-819e-47ac-b5b4-8bbc70921bbc",
       "widget_type": "Synapse.DataFrame"
      },
      "text/plain": [
       "SynapseWidget(Synapse.DataFrame, 4b4c30fd-819e-47ac-b5b4-8bbc70921bbc)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(raw_df.limit(20))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34c36b4c-88d9-49e9-8ffe-87867a93dd7f",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Examine the percentage of the users that visit, the percentage of users that convert, and the percentage of the visitors that convert."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1774905-cbef-4f25-9df8-4b7a11262cbe",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:21:09.050238Z",
       "execution_start_time": "2023-09-08T23:21:08.1199607Z",
       "livy_statement_state": "available",
       "parent_msg_id": "b8c0b967-153d-44d0-8a86-906deb79f003",
       "queued_time": "2023-09-08T23:19:41.8610177Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [
         {
          "completionTime": "2023-09-08T23:21:08.830GMT",
          "dataRead": 1964,
          "dataWritten": 0,
          "description": "Job group for statement 10:\nraw_df.select(\n    F.mean(\"visit\").alias(\"Percentage of users that visit\"),\n    F.mean(\"conversion\").alias(\"Percentage of users that convert\"),\n    (F.sum(\"conversion\") / F.sum(\"visit\")).alias(\"Percentage of visitors that convert\"),\n).show()\n",
          "displayName": "showString at NativeMethodAccessorImpl.java:0",
          "jobGroup": "10",
          "jobId": 13,
          "killedTasksSummary": {},
          "name": "showString at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 25,
          "stageIds": [
           19,
           18
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:08.771GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:21:08.733GMT",
          "dataRead": 1361805488,
          "dataWritten": 1964,
          "description": "Job group for statement 10:\nraw_df.select(\n    F.mean(\"visit\").alias(\"Percentage of users that visit\"),\n    F.mean(\"conversion\").alias(\"Percentage of users that convert\"),\n    (F.sum(\"conversion\") / F.sum(\"visit\")).alias(\"Percentage of visitors that convert\"),\n).show()\n",
          "displayName": "showString at NativeMethodAccessorImpl.java:0",
          "jobGroup": "10",
          "jobId": 12,
          "killedTasksSummary": {},
          "name": "showString at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 1429,
          "stageIds": [
           17
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:08.300GMT",
          "usageDescription": ""
         }
        ],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 2,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 10
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 10, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------------------+--------------------------------+-----------------------------------+\n",
      "|Percentage of users that visit|Percentage of users that convert|Percentage of visitors that convert|\n",
      "+------------------------------+--------------------------------+-----------------------------------+\n",
      "|          0.046992000911042324|            0.002916680257907384|                0.06206759025709019|\n",
      "+------------------------------+--------------------------------+-----------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "raw_df.select(\n",
    "    F.mean(\"visit\").alias(\"Percentage of users that visit\"),\n",
    "    F.mean(\"conversion\").alias(\"Percentage of users that convert\"),\n",
    "    (F.sum(\"conversion\") / F.sum(\"visit\")).alias(\"Percentage of visitors that convert\"),\n",
    ").show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37ad065d-2341-4e68-9c57-50911d7fc0c3",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "This analysis indicates that **4.7%** of all users visited the online store. Only **0.29%** converted (e.g., made a purchase). This means that the conversion rate of visitors is **6.2%**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d5be3f7-ac9c-4927-b81b-a53f673a64a6",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Additionally, examine the overall average treatment effect on visits and conversions. A treatment of 1 indicates the user was offered or exposed to advertising (e.g., discount coupons, gifts, or other privileges). A treatment of 0 indicates that the user was not offered or exposed to any form of advertising."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18807dbf-51a4-4c11-a914-5cc48f9766d4",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:21:11.8980335Z",
       "execution_start_time": "2023-09-08T23:21:09.3271199Z",
       "livy_statement_state": "available",
       "parent_msg_id": "b30fd0a5-c320-4a88-9993-f2a28967ea2c",
       "queued_time": "2023-09-08T23:19:41.861598Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [
         {
          "completionTime": "2023-09-08T23:21:10.904GMT",
          "dataRead": 2755,
          "dataWritten": 0,
          "description": "Job group for statement 11:\nraw_df.groupby(\"treatment\").agg(\n    F.mean(\"visit\").alias(\"Percentage of users that visit\"),\n    F.mean(\"conversion\").alias(\"Percentage of users that convert\"),\n    (F.sum(\"conversion\") / F.sum(\"visit\")).alias(\"Percentage of visitors that convert\"),\n).show()\n",
          "displayName": "showString at NativeMethodAccessorImpl.java:0",
          "jobGroup": "11",
          "jobId": 15,
          "killedTasksSummary": {},
          "name": "showString at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 32,
          "stageIds": [
           21,
           22
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:10.821GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:21:10.737GMT",
          "dataRead": 1361805488,
          "dataWritten": 2755,
          "description": "Job group for statement 11:\nraw_df.groupby(\"treatment\").agg(\n    F.mean(\"visit\").alias(\"Percentage of users that visit\"),\n    F.mean(\"conversion\").alias(\"Percentage of users that convert\"),\n    (F.sum(\"conversion\") / F.sum(\"visit\")).alias(\"Percentage of visitors that convert\"),\n).show()\n",
          "displayName": "showString at NativeMethodAccessorImpl.java:0",
          "jobGroup": "11",
          "jobId": 14,
          "killedTasksSummary": {},
          "name": "showString at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 1436,
          "stageIds": [
           20
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:09.617GMT",
          "usageDescription": ""
         }
        ],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 2,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 11
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 11, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+------------------------------+--------------------------------+-----------------------------------+\n",
      "|treatment|Percentage of users that visit|Percentage of users that convert|Percentage of visitors that convert|\n",
      "+---------+------------------------------+--------------------------------+-----------------------------------+\n",
      "|        1|          0.048543360048743316|            0.003089461067412...|                0.06364332968115058|\n",
      "|        0|           0.03820095691954503|            0.001937588015281...|                0.05072092878097497|\n",
      "+---------+------------------------------+--------------------------------+-----------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "raw_df.groupby(\"treatment\").agg(\n",
    "    F.mean(\"visit\").alias(\"Percentage of users that visit\"),\n",
    "    F.mean(\"conversion\").alias(\"Percentage of users that convert\"),\n",
    "    (F.sum(\"conversion\") / F.sum(\"visit\")).alias(\"Percentage of visitors that convert\"),\n",
    ").show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "769f766e-8e17-4ecc-9f19-72efb7cb44e7",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "The analysis indicates that **4.9%** of users from the treatment group - users that received the treatment, or advertising - visited the online store. Only **3.8%** of users from the control group - users that did not receive the treatment, or were not offered or exposed to advertising - did the same. Additionally, **0.31%** of all users from the treatment group converted, or made a purchase - while only **0.19%** of users from the control group did so. As a result, the conversion rate of visitors that made a purchase, who were also members of treatment group, is **6.36%**, compared to only **5.07%** for users of the control group. This means that the treatment can potentially improve the visit rate by about **1%**, and the conversion rate of visitors by about **1.3%**. This is a significant improvement."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bed56605-0882-4cf0-8731-3830a912b990",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## Step 3: Define the model for training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6db58b1e-5b13-4751-91cd-bb33efce0b3c",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### Prepare the training and test the datasets\n",
    "\n",
    "You'll fit a Featurize transformer to the `raw_df` DataFrame, to extract features from the specified input columns and output those features to a new column named `features`.\n",
    "\n",
    "The resulting DataFrame is stored in a new DataFrame named `df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88aa91c1-149c-47f3-9385-1eb3e0fc3db4",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:21:14.751905Z",
       "execution_start_time": "2023-09-08T23:21:12.1676493Z",
       "livy_statement_state": "available",
       "parent_msg_id": "3811d08f-54d0-46b1-a4f2-4f6aeb55b716",
       "queued_time": "2023-09-08T23:19:41.8630546Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [
         {
          "completionTime": "2023-09-08T23:21:13.881GMT",
          "dataRead": 4514,
          "dataWritten": 0,
          "description": "Job group for statement 12:\ntransformer = Featurize().setOutputCol(\"features\").setInputCols(FEATURE_COLUMNS).fit(raw_df)\ndf = transformer.transform(raw_df)",
          "displayName": "collect at CleanMissingData.scala:124",
          "jobGroup": "12",
          "jobId": 17,
          "killedTasksSummary": {},
          "name": "collect at CleanMissingData.scala:124",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 25,
          "stageIds": [
           24,
           25
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:13.832GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:21:13.777GMT",
          "dataRead": 1361805488,
          "dataWritten": 4514,
          "description": "Job group for statement 12:\ntransformer = Featurize().setOutputCol(\"features\").setInputCols(FEATURE_COLUMNS).fit(raw_df)\ndf = transformer.transform(raw_df)",
          "displayName": "collect at CleanMissingData.scala:124",
          "jobGroup": "12",
          "jobId": 16,
          "killedTasksSummary": {},
          "name": "collect at CleanMissingData.scala:124",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 1429,
          "stageIds": [
           23
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:12.950GMT",
          "usageDescription": ""
         }
        ],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 2,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 12
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 12, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "transformer = Featurize().setOutputCol(\"features\").setInputCols(FEATURE_COLUMNS).fit(raw_df)\n",
    "df = transformer.transform(raw_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fcc0e16-0f6b-49ba-9d0b-0f7647d163d3",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:21:40.8575403Z",
       "execution_start_time": "2023-09-08T23:21:15.0095088Z",
       "livy_statement_state": "available",
       "parent_msg_id": "b78257b2-b6a6-4aa3-b385-e6350836de12",
       "queued_time": "2023-09-08T23:19:41.8637115Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [
         {
          "completionTime": "2023-09-08T23:21:39.886GMT",
          "dataRead": 1983,
          "dataWritten": 0,
          "description": "Job group for statement 13:\n# Split the DataFrame into training and test sets, with a 80/20 ratio and a seed of 42\ntrain_df, test_df = df.randomSplit([0.8, 0.2], seed=42)\n\n# Print the training and test dataset sizes\nprint(\"Size of train dataset: %d\" % train_df.count())\nprint(\"Size of test dataset: %d\" % test_df.count())\n\n# Group the training dataset by the treatment column, and count the number of occurrences of each value\ntrain_df.groupby(TREATMENT_COLUMN).count().show()",
          "displayName": "showString at NativeMethodAccessorImpl.java:0",
          "jobGroup": "13",
          "jobId": 23,
          "killedTasksSummary": {},
          "name": "showString at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 32,
          "stageIds": [
           33,
           34
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:39.837GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:21:39.790GMT",
          "dataRead": 1361805488,
          "dataWritten": 1983,
          "description": "Job group for statement 13:\n# Split the DataFrame into training and test sets, with a 80/20 ratio and a seed of 42\ntrain_df, test_df = df.randomSplit([0.8, 0.2], seed=42)\n\n# Print the training and test dataset sizes\nprint(\"Size of train dataset: %d\" % train_df.count())\nprint(\"Size of test dataset: %d\" % test_df.count())\n\n# Group the training dataset by the treatment column, and count the number of occurrences of each value\ntrain_df.groupby(TREATMENT_COLUMN).count().show()",
          "displayName": "showString at NativeMethodAccessorImpl.java:0",
          "jobGroup": "13",
          "jobId": 22,
          "killedTasksSummary": {},
          "name": "showString at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 1436,
          "stageIds": [
           32
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:32.101GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:21:31.678GMT",
          "dataRead": 1475,
          "dataWritten": 0,
          "description": "Job group for statement 13:\n# Split the DataFrame into training and test sets, with a 80/20 ratio and a seed of 42\ntrain_df, test_df = df.randomSplit([0.8, 0.2], seed=42)\n\n# Print the training and test dataset sizes\nprint(\"Size of train dataset: %d\" % train_df.count())\nprint(\"Size of test dataset: %d\" % test_df.count())\n\n# Group the training dataset by the treatment column, and count the number of occurrences of each value\ntrain_df.groupby(TREATMENT_COLUMN).count().show()",
          "displayName": "count at NativeMethodAccessorImpl.java:0",
          "jobGroup": "13",
          "jobId": 21,
          "killedTasksSummary": {},
          "name": "count at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 25,
          "stageIds": [
           30,
           31
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:31.654GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:21:31.637GMT",
          "dataRead": 1361805488,
          "dataWritten": 1475,
          "description": "Job group for statement 13:\n# Split the DataFrame into training and test sets, with a 80/20 ratio and a seed of 42\ntrain_df, test_df = df.randomSplit([0.8, 0.2], seed=42)\n\n# Print the training and test dataset sizes\nprint(\"Size of train dataset: %d\" % train_df.count())\nprint(\"Size of test dataset: %d\" % test_df.count())\n\n# Group the training dataset by the treatment column, and count the number of occurrences of each value\ntrain_df.groupby(TREATMENT_COLUMN).count().show()",
          "displayName": "count at NativeMethodAccessorImpl.java:0",
          "jobGroup": "13",
          "jobId": 20,
          "killedTasksSummary": {},
          "name": "count at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 1429,
          "stageIds": [
           29
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:24.029GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:21:23.686GMT",
          "dataRead": 1475,
          "dataWritten": 0,
          "description": "Job group for statement 13:\n# Split the DataFrame into training and test sets, with a 80/20 ratio and a seed of 42\ntrain_df, test_df = df.randomSplit([0.8, 0.2], seed=42)\n\n# Print the training and test dataset sizes\nprint(\"Size of train dataset: %d\" % train_df.count())\nprint(\"Size of test dataset: %d\" % test_df.count())\n\n# Group the training dataset by the treatment column, and count the number of occurrences of each value\ntrain_df.groupby(TREATMENT_COLUMN).count().show()",
          "displayName": "count at NativeMethodAccessorImpl.java:0",
          "jobGroup": "13",
          "jobId": 19,
          "killedTasksSummary": {},
          "name": "count at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 25,
          "stageIds": [
           27,
           28
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:23.648GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:21:23.619GMT",
          "dataRead": 1361805488,
          "dataWritten": 1475,
          "description": "Job group for statement 13:\n# Split the DataFrame into training and test sets, with a 80/20 ratio and a seed of 42\ntrain_df, test_df = df.randomSplit([0.8, 0.2], seed=42)\n\n# Print the training and test dataset sizes\nprint(\"Size of train dataset: %d\" % train_df.count())\nprint(\"Size of test dataset: %d\" % test_df.count())\n\n# Group the training dataset by the treatment column, and count the number of occurrences of each value\ntrain_df.groupby(TREATMENT_COLUMN).count().show()",
          "displayName": "count at NativeMethodAccessorImpl.java:0",
          "jobGroup": "13",
          "jobId": 18,
          "killedTasksSummary": {},
          "name": "count at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 1429,
          "stageIds": [
           26
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:15.337GMT",
          "usageDescription": ""
         }
        ],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 6,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 13
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 13, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of train dataset: 11182965\n",
      "Size of test dataset: 2796627\n",
      "+---------+-------+\n",
      "|treatment|  count|\n",
      "+---------+-------+\n",
      "|        1|9505415|\n",
      "|        0|1677550|\n",
      "+---------+-------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Split the DataFrame into training and test sets, with a 80/20 ratio and a seed of 42\n",
    "train_df, test_df = df.randomSplit([0.8, 0.2], seed=42)\n",
    "\n",
    "# Print the training and test dataset sizes\n",
    "print(\"Size of train dataset: %d\" % train_df.count())\n",
    "print(\"Size of test dataset: %d\" % test_df.count())\n",
    "\n",
    "# Group the training dataset by the treatment column, and count the number of occurrences of each value\n",
    "train_df.groupby(TREATMENT_COLUMN).count().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af7aa060-edbd-4bca-9d89-14e25df92d42",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### Prepare the treatment and control datasets\n",
    "\n",
    "After you create the training and test datasets, you must also form the treatment and control datasets, to train the machine learning models to measure the uplift."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3dd54f60-4e77-479e-ac91-89e68fbfaa4a",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:21:41.4501488Z",
       "execution_start_time": "2023-09-08T23:21:41.1192741Z",
       "livy_statement_state": "available",
       "parent_msg_id": "22043324-7923-4417-916e-c4652f463878",
       "queued_time": "2023-09-08T23:19:41.8644191Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 0,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 14
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 14, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Extract the treatment and control DataFrames\n",
    "treatment_train_df = train_df.where(f\"{TREATMENT_COLUMN} > 0\")\n",
    "control_train_df = train_df.where(f\"{TREATMENT_COLUMN} = 0\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97a88ea8-1981-403c-b0e0-c8a67392d1c8",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Now that you prepared your data, you can proceed to train a model with LightGBM."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30553ba4-005b-4c4f-8723-a4b8b7822645",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### Uplift modeling: T-Learner with LightGBM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3328c276-86f9-4b3b-802c-33b19d76b78e",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Meta-learners are a set of algorithms, built on top of machine learning algorithms like LightGBM, Xgboost, etc. They help estimate conditional average treatment effect, or **CATE**.\n",
    "T-learner is a meta-learner that doesn't use a single model. Instead, T-learner uses one model per treatment variable. Therefore, two models are developed and hence it is referred to as T-learner. T-learner uses multiple machine learning models to overcome the problem of entirely discarding the treatment, by forcing the learner to first split on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b7e2bef-b1f2-4a98-9b02-394f09c89f2c",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:21:42.6334411Z",
       "execution_start_time": "2023-09-08T23:21:41.7182486Z",
       "livy_statement_state": "available",
       "parent_msg_id": "32faf333-8c2e-438b-96b4-367f83fe413c",
       "queued_time": "2023-09-08T23:19:41.8651385Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 0,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 15
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 15, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023/09/08 23:21:41 INFO mlflow.tracking.fluent: Autologging successfully enabled for statsmodels.\n",
      "2023/09/08 23:21:41 INFO mlflow.tracking.fluent: Autologging successfully enabled for pyspark.ml.\n"
     ]
    }
   ],
   "source": [
    "mlflow.autolog(exclusive=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1586aff-d7de-4569-b250-949d40076f29",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:22:53.6265322Z",
       "execution_start_time": "2023-09-08T23:21:42.8971537Z",
       "livy_statement_state": "available",
       "parent_msg_id": "3ee53e1e-9334-48a5-b9b4-5c193a61103e",
       "queued_time": "2023-09-08T23:19:41.8658513Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [
         {
          "completionTime": "2023-09-08T23:22:47.787GMT",
          "dataRead": 0,
          "dataWritten": 817,
          "description": "Job group for statement 16:\nclassifier = (\n    LightGBMClassifier()\n    .setFeaturesCol(\"features\")  # Set the column name for features\n    .setNumLeaves(10)  # Set the number of leaves in each decision tree\n    .setNumIterations(100)  # Set the number of boosting iterations\n    .setObjective(\"binary\")  # Set the objective function for binary classification\n    .setLabelCol(LABEL_COLUMN)  # Set the column name for the label\n)\n\n# Start a new MLflow run with the name \"uplift\"\nactive_run = mlflow.start_run(run_name=\"uplift\")\n\n# Start a new nested MLflow run with the name \"treatment\"\nwith mlflow.start_run(run_name=\"treatment\", nested=True) as treatment_run:\n    treatment_run_id = treatment_run.info.run_id  # Get the ID of the treatment run\n    treatment_model = classifier.fit(treatment_train_df)  # Fit the classifier on the treatment training data\n\n# Start a new nested MLflow run with the name \"control\"\nwith mlflow.start_run(run_name=\"control\", nested=True) as control_run:\n    control_run_id = control_run.info.run_id  # Get the ID of the co...",
          "displayName": "runJob at SparkHadoopWriter.scala:85",
          "jobGroup": "16",
          "jobId": 31,
          "killedTasksSummary": {},
          "name": "runJob at SparkHadoopWriter.scala:85",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 1,
          "stageIds": [
           42
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:22:47.528GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:22:47.249GMT",
          "dataRead": 0,
          "dataWritten": 226,
          "description": "Job group for statement 16:\nclassifier = (\n    LightGBMClassifier()\n    .setFeaturesCol(\"features\")  # Set the column name for features\n    .setNumLeaves(10)  # Set the number of leaves in each decision tree\n    .setNumIterations(100)  # Set the number of boosting iterations\n    .setObjective(\"binary\")  # Set the objective function for binary classification\n    .setLabelCol(LABEL_COLUMN)  # Set the column name for the label\n)\n\n# Start a new MLflow run with the name \"uplift\"\nactive_run = mlflow.start_run(run_name=\"uplift\")\n\n# Start a new nested MLflow run with the name \"treatment\"\nwith mlflow.start_run(run_name=\"treatment\", nested=True) as treatment_run:\n    treatment_run_id = treatment_run.info.run_id  # Get the ID of the treatment run\n    treatment_model = classifier.fit(treatment_train_df)  # Fit the classifier on the treatment training data\n\n# Start a new nested MLflow run with the name \"control\"\nwith mlflow.start_run(run_name=\"control\", nested=True) as control_run:\n    control_run_id = control_run.info.run_id  # Get the ID of the co...",
          "displayName": "runJob at SparkHadoopWriter.scala:85",
          "jobGroup": "16",
          "jobId": 30,
          "killedTasksSummary": {},
          "name": "runJob at SparkHadoopWriter.scala:85",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 1,
          "stageIds": [
           41
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:22:46.994GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:22:46.449GMT",
          "dataRead": 1202720770,
          "dataWritten": 0,
          "description": "Job group for statement 16:\nclassifier = (\n    LightGBMClassifier()\n    .setFeaturesCol(\"features\")  # Set the column name for features\n    .setNumLeaves(10)  # Set the number of leaves in each decision tree\n    .setNumIterations(100)  # Set the number of boosting iterations\n    .setObjective(\"binary\")  # Set the objective function for binary classification\n    .setLabelCol(LABEL_COLUMN)  # Set the column name for the label\n)\n\n# Start a new MLflow run with the name \"uplift\"\nactive_run = mlflow.start_run(run_name=\"uplift\")\n\n# Start a new nested MLflow run with the name \"treatment\"\nwith mlflow.start_run(run_name=\"treatment\", nested=True) as treatment_run:\n    treatment_run_id = treatment_run.info.run_id  # Get the ID of the treatment run\n    treatment_model = classifier.fit(treatment_train_df)  # Fit the classifier on the treatment training data\n\n# Start a new nested MLflow run with the name \"control\"\nwith mlflow.start_run(run_name=\"control\", nested=True) as control_run:\n    control_run_id = control_run.info.run_id  # Get the ID of the co...",
          "displayName": "collect at LightGBMBase.scala:623",
          "jobGroup": "16",
          "jobId": 29,
          "killedTasksSummary": {},
          "name": "collect at LightGBMBase.scala:623",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 8,
          "numCompletedStages": 1,
          "numCompletedTasks": 8,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 8,
          "rowCount": 1255,
          "stageIds": [
           40
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:22:34.283GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:22:34.226GMT",
          "dataRead": 168688192,
          "dataWritten": 0,
          "description": "Job group for statement 16:\nclassifier = (\n    LightGBMClassifier()\n    .setFeaturesCol(\"features\")  # Set the column name for features\n    .setNumLeaves(10)  # Set the number of leaves in each decision tree\n    .setNumIterations(100)  # Set the number of boosting iterations\n    .setObjective(\"binary\")  # Set the objective function for binary classification\n    .setLabelCol(LABEL_COLUMN)  # Set the column name for the label\n)\n\n# Start a new MLflow run with the name \"uplift\"\nactive_run = mlflow.start_run(run_name=\"uplift\")\n\n# Start a new nested MLflow run with the name \"treatment\"\nwith mlflow.start_run(run_name=\"treatment\", nested=True) as treatment_run:\n    treatment_run_id = treatment_run.info.run_id  # Get the ID of the treatment run\n    treatment_model = classifier.fit(treatment_train_df)  # Fit the classifier on the treatment training data\n\n# Start a new nested MLflow run with the name \"control\"\nwith mlflow.start_run(run_name=\"control\", nested=True) as control_run:\n    control_run_id = control_run.info.run_id  # Get the ID of the co...",
          "displayName": "first at LightGBMBase.scala:481",
          "jobGroup": "16",
          "jobId": 28,
          "killedTasksSummary": {},
          "name": "first at LightGBMBase.scala:481",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 174,
          "stageIds": [
           39
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:22:29.870GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:22:20.877GMT",
          "dataRead": 0,
          "dataWritten": 817,
          "description": "Job group for statement 16:\nclassifier = (\n    LightGBMClassifier()\n    .setFeaturesCol(\"features\")  # Set the column name for features\n    .setNumLeaves(10)  # Set the number of leaves in each decision tree\n    .setNumIterations(100)  # Set the number of boosting iterations\n    .setObjective(\"binary\")  # Set the objective function for binary classification\n    .setLabelCol(LABEL_COLUMN)  # Set the column name for the label\n)\n\n# Start a new MLflow run with the name \"uplift\"\nactive_run = mlflow.start_run(run_name=\"uplift\")\n\n# Start a new nested MLflow run with the name \"treatment\"\nwith mlflow.start_run(run_name=\"treatment\", nested=True) as treatment_run:\n    treatment_run_id = treatment_run.info.run_id  # Get the ID of the treatment run\n    treatment_model = classifier.fit(treatment_train_df)  # Fit the classifier on the treatment training data\n\n# Start a new nested MLflow run with the name \"control\"\nwith mlflow.start_run(run_name=\"control\", nested=True) as control_run:\n    control_run_id = control_run.info.run_id  # Get the ID of the co...",
          "displayName": "runJob at SparkHadoopWriter.scala:85",
          "jobGroup": "16",
          "jobId": 27,
          "killedTasksSummary": {},
          "name": "runJob at SparkHadoopWriter.scala:85",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 1,
          "stageIds": [
           38
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:22:20.622GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:22:20.280GMT",
          "dataRead": 0,
          "dataWritten": 226,
          "description": "Job group for statement 16:\nclassifier = (\n    LightGBMClassifier()\n    .setFeaturesCol(\"features\")  # Set the column name for features\n    .setNumLeaves(10)  # Set the number of leaves in each decision tree\n    .setNumIterations(100)  # Set the number of boosting iterations\n    .setObjective(\"binary\")  # Set the objective function for binary classification\n    .setLabelCol(LABEL_COLUMN)  # Set the column name for the label\n)\n\n# Start a new MLflow run with the name \"uplift\"\nactive_run = mlflow.start_run(run_name=\"uplift\")\n\n# Start a new nested MLflow run with the name \"treatment\"\nwith mlflow.start_run(run_name=\"treatment\", nested=True) as treatment_run:\n    treatment_run_id = treatment_run.info.run_id  # Get the ID of the treatment run\n    treatment_model = classifier.fit(treatment_train_df)  # Fit the classifier on the treatment training data\n\n# Start a new nested MLflow run with the name \"control\"\nwith mlflow.start_run(run_name=\"control\", nested=True) as control_run:\n    control_run_id = control_run.info.run_id  # Get the ID of the co...",
          "displayName": "runJob at SparkHadoopWriter.scala:85",
          "jobGroup": "16",
          "jobId": 26,
          "killedTasksSummary": {},
          "name": "runJob at SparkHadoopWriter.scala:85",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 1,
          "stageIds": [
           37
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:22:19.670GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:22:18.762GMT",
          "dataRead": 1320160872,
          "dataWritten": 0,
          "description": "Job group for statement 16:\nclassifier = (\n    LightGBMClassifier()\n    .setFeaturesCol(\"features\")  # Set the column name for features\n    .setNumLeaves(10)  # Set the number of leaves in each decision tree\n    .setNumIterations(100)  # Set the number of boosting iterations\n    .setObjective(\"binary\")  # Set the objective function for binary classification\n    .setLabelCol(LABEL_COLUMN)  # Set the column name for the label\n)\n\n# Start a new MLflow run with the name \"uplift\"\nactive_run = mlflow.start_run(run_name=\"uplift\")\n\n# Start a new nested MLflow run with the name \"treatment\"\nwith mlflow.start_run(run_name=\"treatment\", nested=True) as treatment_run:\n    treatment_run_id = treatment_run.info.run_id  # Get the ID of the treatment run\n    treatment_model = classifier.fit(treatment_train_df)  # Fit the classifier on the treatment training data\n\n# Start a new nested MLflow run with the name \"control\"\nwith mlflow.start_run(run_name=\"control\", nested=True) as control_run:\n    control_run_id = control_run.info.run_id  # Get the ID of the co...",
          "displayName": "collect at LightGBMBase.scala:623",
          "jobGroup": "16",
          "jobId": 25,
          "killedTasksSummary": {},
          "name": "collect at LightGBMBase.scala:623",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 8,
          "numCompletedStages": 1,
          "numCompletedTasks": 8,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 8,
          "rowCount": 1393,
          "stageIds": [
           36
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:48.077GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:21:47.939GMT",
          "dataRead": 56187832,
          "dataWritten": 0,
          "description": "Job group for statement 16:\nclassifier = (\n    LightGBMClassifier()\n    .setFeaturesCol(\"features\")  # Set the column name for features\n    .setNumLeaves(10)  # Set the number of leaves in each decision tree\n    .setNumIterations(100)  # Set the number of boosting iterations\n    .setObjective(\"binary\")  # Set the objective function for binary classification\n    .setLabelCol(LABEL_COLUMN)  # Set the column name for the label\n)\n\n# Start a new MLflow run with the name \"uplift\"\nactive_run = mlflow.start_run(run_name=\"uplift\")\n\n# Start a new nested MLflow run with the name \"treatment\"\nwith mlflow.start_run(run_name=\"treatment\", nested=True) as treatment_run:\n    treatment_run_id = treatment_run.info.run_id  # Get the ID of the treatment run\n    treatment_model = classifier.fit(treatment_train_df)  # Fit the classifier on the treatment training data\n\n# Start a new nested MLflow run with the name \"control\"\nwith mlflow.start_run(run_name=\"control\", nested=True) as control_run:\n    control_run_id = control_run.info.run_id  # Get the ID of the co...",
          "displayName": "first at LightGBMBase.scala:481",
          "jobGroup": "16",
          "jobId": 24,
          "killedTasksSummary": {},
          "name": "first at LightGBMBase.scala:481",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 58,
          "stageIds": [
           35
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:21:45.662GMT",
          "usageDescription": ""
         }
        ],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 8,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 16
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 16, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023/09/08 23:22:18 WARNING mlflow.pyspark.ml: Model inputs contain unsupported Spark data types: [StructField('features', VectorUDT(), True)]. Model signature is not logged.\n",
      "2023/09/08 23:22:26 WARNING mlflow.utils.environment: Encountered an unexpected error while inferring pip requirements (model URI: /tmp/tmpack5q2cd/model, flavor: spark), fall back to return ['pyspark==3.3.1']. Set logging level to DEBUG to see the full traceback.\n",
      "2023/09/08 23:22:26 WARNING mlflow.utils.autologging_utils: MLflow autologging encountered a warning: \"/home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.\"\n",
      "2023/09/08 23:22:50 WARNING mlflow.utils.environment: Encountered an unexpected error while inferring pip requirements (model URI: /tmp/tmpuocncyyt/model, flavor: spark), fall back to return ['pyspark==3.3.1']. Set logging level to DEBUG to see the full traceback.\n"
     ]
    }
   ],
   "source": [
    "classifier = (\n",
    "    LightGBMClassifier()\n",
    "    .setFeaturesCol(\"features\")  # Set the column name for features\n",
    "    .setNumLeaves(10)  # Set the number of leaves in each decision tree\n",
    "    .setNumIterations(100)  # Set the number of boosting iterations\n",
    "    .setObjective(\"binary\")  # Set the objective function for binary classification\n",
    "    .setLabelCol(LABEL_COLUMN)  # Set the column name for the label\n",
    ")\n",
    "\n",
    "# Start a new MLflow run with the name \"uplift\"\n",
    "active_run = mlflow.start_run(run_name=\"uplift\")\n",
    "\n",
    "# Start a new nested MLflow run with the name \"treatment\"\n",
    "with mlflow.start_run(run_name=\"treatment\", nested=True) as treatment_run:\n",
    "    treatment_run_id = treatment_run.info.run_id  # Get the ID of the treatment run\n",
    "    treatment_model = classifier.fit(treatment_train_df)  # Fit the classifier on the treatment training data\n",
    "\n",
    "# Start a new nested MLflow run with the name \"control\"\n",
    "with mlflow.start_run(run_name=\"control\", nested=True) as control_run:\n",
    "    control_run_id = control_run.info.run_id  # Get the ID of the control run\n",
    "    control_model = classifier.fit(control_train_df)  # Fit the classifier on the control training data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41aeeb3b-05ed-47c8-bd27-16717c8fc9b4",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### Use the test dataset for a prediction\n",
    "\n",
    "You'll use the `treatment_model` and `control_model`, both defined earlier, to transform the `test_df` test dataset. Then, you'll calculate the predicted uplift. You'll define the predicted uplift as the difference between the predicted treatment outcome and the predicted control outcome. The greater this predicted uplift difference, the greater the effectiveness of the treatment (e.g., advertising) on an individual or a sub-group.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37ee9071-8119-4ca0-b213-78f6b47e12a9",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:23:31.1820478Z",
       "execution_start_time": "2023-09-08T23:22:53.8890825Z",
       "livy_statement_state": "available",
       "parent_msg_id": "1f88948d-e954-4e33-8f6b-e31be63ae913",
       "queued_time": "2023-09-08T23:19:41.8665932Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [
         {
          "completionTime": "2023-09-08T23:23:29.884GMT",
          "dataRead": 0,
          "dataWritten": 0,
          "description": "Job group for statement 17:\ngetPred = F.udf(lambda v: float(v[1]), FloatType())\n\n# Cache the resulting DataFrame for easier access\ntest_pred_df = (\n    test_df.mlTransform(treatment_model)\n    .withColumn(\"treatment_pred\", getPred(\"probability\"))\n    .drop(\"rawPrediction\", \"probability\", \"prediction\")\n    .mlTransform(control_model)\n    .withColumn(\"control_pred\", getPred(\"probability\"))\n    .drop(\"rawPrediction\", \"probability\", \"prediction\")\n    .withColumn(\"pred_uplift\", F.col(\"treatment_pred\") - F.col(\"control_pred\"))\n    .select(TREATMENT_COLUMN, LABEL_COLUMN, \"treatment_pred\", \"control_pred\", \"pred_uplift\")\n    .cache()\n)\n\n# Display the first twenty rows of the resulting DataFrame\ndisplay(test_pred_df.limit(20))",
          "displayName": "getRowsInJsonString at Display.scala:452",
          "jobGroup": "17",
          "jobId": 33,
          "killedTasksSummary": {},
          "name": "getRowsInJsonString at Display.scala:452",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 0,
          "stageIds": [
           45,
           44
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:23:29.851GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:23:29.788GMT",
          "dataRead": 1361805488,
          "dataWritten": 5830,
          "description": "Job group for statement 17:\ngetPred = F.udf(lambda v: float(v[1]), FloatType())\n\n# Cache the resulting DataFrame for easier access\ntest_pred_df = (\n    test_df.mlTransform(treatment_model)\n    .withColumn(\"treatment_pred\", getPred(\"probability\"))\n    .drop(\"rawPrediction\", \"probability\", \"prediction\")\n    .mlTransform(control_model)\n    .withColumn(\"control_pred\", getPred(\"probability\"))\n    .drop(\"rawPrediction\", \"probability\", \"prediction\")\n    .withColumn(\"pred_uplift\", F.col(\"treatment_pred\") - F.col(\"control_pred\"))\n    .select(TREATMENT_COLUMN, LABEL_COLUMN, \"treatment_pred\", \"control_pred\", \"pred_uplift\")\n    .cache()\n)\n\n# Display the first twenty rows of the resulting DataFrame\ndisplay(test_pred_df.limit(20))",
          "displayName": "getRowsInJsonString at Display.scala:452",
          "jobGroup": "17",
          "jobId": 32,
          "killedTasksSummary": {},
          "name": "getRowsInJsonString at Display.scala:452",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 1904,
          "stageIds": [
           43
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:22:54.586GMT",
          "usageDescription": ""
         }
        ],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 2,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 17
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 17, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.synapse.widget-view+json": {
       "widget_id": "2b05e279-7816-43b3-857c-cb58b1bb8f4d",
       "widget_type": "Synapse.DataFrame"
      },
      "text/plain": [
       "SynapseWidget(Synapse.DataFrame, 2b05e279-7816-43b3-857c-cb58b1bb8f4d)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "getPred = F.udf(lambda v: float(v[1]), FloatType())\n",
    "\n",
    "# Cache the resulting DataFrame for easier access\n",
    "test_pred_df = (\n",
    "    test_df.mlTransform(treatment_model)\n",
    "    .withColumn(\"treatment_pred\", getPred(\"probability\"))\n",
    "    .drop(\"rawPrediction\", \"probability\", \"prediction\")\n",
    "    .mlTransform(control_model)\n",
    "    .withColumn(\"control_pred\", getPred(\"probability\"))\n",
    "    .drop(\"rawPrediction\", \"probability\", \"prediction\")\n",
    "    .withColumn(\"pred_uplift\", F.col(\"treatment_pred\") - F.col(\"control_pred\"))\n",
    "    .select(TREATMENT_COLUMN, LABEL_COLUMN, \"treatment_pred\", \"control_pred\", \"pred_uplift\")\n",
    "    .cache()\n",
    ")\n",
    "\n",
    "# Display the first twenty rows of the resulting DataFrame\n",
    "display(test_pred_df.limit(20))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f783613-8e24-4cda-b6d9-3a06aad7b650",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### Perform model evaluation\n",
    "\n",
    "Since actual uplift cannot be observed for each individual, you need to measure the uplift over a group of individuals. You'll use an Uplift Curve that plots the real, cumulative uplift across the population.\n",
    "\n",
    "![criteo_uplift_curve.jpeg](https://mmlspark.blob.core.windows.net/graphics/notebooks/criteo_uplift_curve.jpeg)\n",
    "\n",
    "The x-axis represents the ratio of the population selected for the treatment. A value of 0 suggests no treatment group - no one is exposed to, or offered, the treatment. A value of 1 suggests a full treatment group - everyone is exposed to, or offered, the treatment. The y-axis shows the uplift measure. The aim is to find the size of the treatment group, or the percentage of the population that would be offered or exposed to the treatment (e.g., advertising). This optimizes the target selection, to maximize its impact.\n",
    "\n",
    "First, rank the test DataFrame order by the predicted uplift. The predicted uplift is the difference between the predicted treatment outcome and the predicted control outcome."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78061a75-5f32-422b-9555-314d4a638306",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:23:34.0296887Z",
       "execution_start_time": "2023-09-08T23:23:31.4452702Z",
       "livy_statement_state": "available",
       "parent_msg_id": "16ff6fd9-0fe3-4f6d-b45b-cc3eeb62114e",
       "queued_time": "2023-09-08T23:19:41.8672767Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [
         {
          "completionTime": "2023-09-08T23:23:33.725GMT",
          "dataRead": 24381543,
          "dataWritten": 0,
          "description": "Job group for statement 18:\n# Compute the percentage rank of the predicted uplift values in descending order, and display the top twenty rows\ntest_ranked_df = test_pred_df.withColumn(\"percent_rank\", F.percent_rank().over(Window.orderBy(F.desc(\"pred_uplift\"))))\n\ndisplay(test_ranked_df.limit(20))",
          "displayName": "getRowsInJsonString at Display.scala:452",
          "jobGroup": "18",
          "jobId": 35,
          "killedTasksSummary": {},
          "name": "getRowsInJsonString at Display.scala:452",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 2796627,
          "stageIds": [
           48,
           47
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:23:32.291GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:23:32.151GMT",
          "dataRead": 35972800,
          "dataWritten": 24381543,
          "description": "Job group for statement 18:\n# Compute the percentage rank of the predicted uplift values in descending order, and display the top twenty rows\ntest_ranked_df = test_pred_df.withColumn(\"percent_rank\", F.percent_rank().over(Window.orderBy(F.desc(\"pred_uplift\"))))\n\ndisplay(test_ranked_df.limit(20))",
          "displayName": "getRowsInJsonString at Display.scala:452",
          "jobGroup": "18",
          "jobId": 34,
          "killedTasksSummary": {},
          "name": "getRowsInJsonString at Display.scala:452",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 2796918,
          "stageIds": [
           46
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:23:31.701GMT",
          "usageDescription": ""
         }
        ],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 2,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 18
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 18, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.synapse.widget-view+json": {
       "widget_id": "e9b3eb0d-ff8b-4654-b0d8-c62ab740be69",
       "widget_type": "Synapse.DataFrame"
      },
      "text/plain": [
       "SynapseWidget(Synapse.DataFrame, e9b3eb0d-ff8b-4654-b0d8-c62ab740be69)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Compute the percentage rank of the predicted uplift values in descending order, and display the top twenty rows\n",
    "test_ranked_df = test_pred_df.withColumn(\"percent_rank\", F.percent_rank().over(Window.orderBy(F.desc(\"pred_uplift\"))))\n",
    "\n",
    "display(test_ranked_df.limit(20))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ba35815-5909-4feb-bb9f-25fc03c44a62",
   "metadata": {},
   "source": [
    "Next, calculate the cumulative percentage of visits in both the treatment and control groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40f3209a-647b-4047-a204-22971b41d7f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:23:38.672177Z",
       "execution_start_time": "2023-09-08T23:23:34.2910596Z",
       "livy_statement_state": "available",
       "parent_msg_id": "0f6f16a6-a189-429f-8682-473d389c1521",
       "queued_time": "2023-09-08T23:19:41.8679054Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [
         {
          "completionTime": "2023-09-08T23:23:38.325GMT",
          "dataRead": 24381543,
          "dataWritten": 0,
          "description": "Job group for statement 19:\n# Calculate the number of control and treatment samples\nC = test_ranked_df.where(f\"{TREATMENT_COLUMN} == 0\").count()\nT = test_ranked_df.where(f\"{TREATMENT_COLUMN} != 0\").count()\n\n# Add columns to the DataFrame to calculate the control and treatment cumulative sum\ntest_ranked_df = (\n    test_ranked_df.withColumn(\n        \"control_label\",\n        F.when(F.col(TREATMENT_COLUMN) == 0, F.col(LABEL_COLUMN)).otherwise(0),\n    )\n    .withColumn(\n        \"treatment_label\",\n        F.when(F.col(TREATMENT_COLUMN) != 0, F.col(LABEL_COLUMN)).otherwise(0),\n    )\n    .withColumn(\n        \"control_cumsum\",\n        F.sum(\"control_label\").over(Window.orderBy(\"percent_rank\")) / C,\n    )\n    .withColumn(\n        \"treatment_cumsum\",\n        F.sum(\"treatment_label\").over(Window.orderBy(\"percent_rank\")) / T,\n    )\n)\n\n# Display the first 20 rows of the dataframe\ndisplay(test_ranked_df.limit(20))",
          "displayName": "getRowsInJsonString at Display.scala:452",
          "jobGroup": "19",
          "jobId": 41,
          "killedTasksSummary": {},
          "name": "getRowsInJsonString at Display.scala:452",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 2796627,
          "stageIds": [
           56,
           57
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:23:36.112GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:23:36.045GMT",
          "dataRead": 35972800,
          "dataWritten": 24381543,
          "description": "Job group for statement 19:\n# Calculate the number of control and treatment samples\nC = test_ranked_df.where(f\"{TREATMENT_COLUMN} == 0\").count()\nT = test_ranked_df.where(f\"{TREATMENT_COLUMN} != 0\").count()\n\n# Add columns to the DataFrame to calculate the control and treatment cumulative sum\ntest_ranked_df = (\n    test_ranked_df.withColumn(\n        \"control_label\",\n        F.when(F.col(TREATMENT_COLUMN) == 0, F.col(LABEL_COLUMN)).otherwise(0),\n    )\n    .withColumn(\n        \"treatment_label\",\n        F.when(F.col(TREATMENT_COLUMN) != 0, F.col(LABEL_COLUMN)).otherwise(0),\n    )\n    .withColumn(\n        \"control_cumsum\",\n        F.sum(\"control_label\").over(Window.orderBy(\"percent_rank\")) / C,\n    )\n    .withColumn(\n        \"treatment_cumsum\",\n        F.sum(\"treatment_label\").over(Window.orderBy(\"percent_rank\")) / T,\n    )\n)\n\n# Display the first 20 rows of the dataframe\ndisplay(test_ranked_df.limit(20))",
          "displayName": "getRowsInJsonString at Display.scala:452",
          "jobGroup": "19",
          "jobId": 40,
          "killedTasksSummary": {},
          "name": "getRowsInJsonString at Display.scala:452",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 2796918,
          "stageIds": [
           55
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:23:35.764GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:23:35.410GMT",
          "dataRead": 1472,
          "dataWritten": 0,
          "description": "Job group for statement 19:\n# Calculate the number of control and treatment samples\nC = test_ranked_df.where(f\"{TREATMENT_COLUMN} == 0\").count()\nT = test_ranked_df.where(f\"{TREATMENT_COLUMN} != 0\").count()\n\n# Add columns to the DataFrame to calculate the control and treatment cumulative sum\ntest_ranked_df = (\n    test_ranked_df.withColumn(\n        \"control_label\",\n        F.when(F.col(TREATMENT_COLUMN) == 0, F.col(LABEL_COLUMN)).otherwise(0),\n    )\n    .withColumn(\n        \"treatment_label\",\n        F.when(F.col(TREATMENT_COLUMN) != 0, F.col(LABEL_COLUMN)).otherwise(0),\n    )\n    .withColumn(\n        \"control_cumsum\",\n        F.sum(\"control_label\").over(Window.orderBy(\"percent_rank\")) / C,\n    )\n    .withColumn(\n        \"treatment_cumsum\",\n        F.sum(\"treatment_label\").over(Window.orderBy(\"percent_rank\")) / T,\n    )\n)\n\n# Display the first 20 rows of the dataframe\ndisplay(test_ranked_df.limit(20))",
          "displayName": "count at NativeMethodAccessorImpl.java:0",
          "jobGroup": "19",
          "jobId": 39,
          "killedTasksSummary": {},
          "name": "count at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 25,
          "stageIds": [
           53,
           54
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:23:35.392GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:23:35.374GMT",
          "dataRead": 35972800,
          "dataWritten": 1472,
          "description": "Job group for statement 19:\n# Calculate the number of control and treatment samples\nC = test_ranked_df.where(f\"{TREATMENT_COLUMN} == 0\").count()\nT = test_ranked_df.where(f\"{TREATMENT_COLUMN} != 0\").count()\n\n# Add columns to the DataFrame to calculate the control and treatment cumulative sum\ntest_ranked_df = (\n    test_ranked_df.withColumn(\n        \"control_label\",\n        F.when(F.col(TREATMENT_COLUMN) == 0, F.col(LABEL_COLUMN)).otherwise(0),\n    )\n    .withColumn(\n        \"treatment_label\",\n        F.when(F.col(TREATMENT_COLUMN) != 0, F.col(LABEL_COLUMN)).otherwise(0),\n    )\n    .withColumn(\n        \"control_cumsum\",\n        F.sum(\"control_label\").over(Window.orderBy(\"percent_rank\")) / C,\n    )\n    .withColumn(\n        \"treatment_cumsum\",\n        F.sum(\"treatment_label\").over(Window.orderBy(\"percent_rank\")) / T,\n    )\n)\n\n# Display the first 20 rows of the dataframe\ndisplay(test_ranked_df.limit(20))",
          "displayName": "count at NativeMethodAccessorImpl.java:0",
          "jobGroup": "19",
          "jobId": 38,
          "killedTasksSummary": {},
          "name": "count at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 316,
          "stageIds": [
           52
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:23:35.066GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:23:34.808GMT",
          "dataRead": 1424,
          "dataWritten": 0,
          "description": "Job group for statement 19:\n# Calculate the number of control and treatment samples\nC = test_ranked_df.where(f\"{TREATMENT_COLUMN} == 0\").count()\nT = test_ranked_df.where(f\"{TREATMENT_COLUMN} != 0\").count()\n\n# Add columns to the DataFrame to calculate the control and treatment cumulative sum\ntest_ranked_df = (\n    test_ranked_df.withColumn(\n        \"control_label\",\n        F.when(F.col(TREATMENT_COLUMN) == 0, F.col(LABEL_COLUMN)).otherwise(0),\n    )\n    .withColumn(\n        \"treatment_label\",\n        F.when(F.col(TREATMENT_COLUMN) != 0, F.col(LABEL_COLUMN)).otherwise(0),\n    )\n    .withColumn(\n        \"control_cumsum\",\n        F.sum(\"control_label\").over(Window.orderBy(\"percent_rank\")) / C,\n    )\n    .withColumn(\n        \"treatment_cumsum\",\n        F.sum(\"treatment_label\").over(Window.orderBy(\"percent_rank\")) / T,\n    )\n)\n\n# Display the first 20 rows of the dataframe\ndisplay(test_ranked_df.limit(20))",
          "displayName": "count at NativeMethodAccessorImpl.java:0",
          "jobGroup": "19",
          "jobId": 37,
          "killedTasksSummary": {},
          "name": "count at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 25,
          "stageIds": [
           51,
           50
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:23:34.788GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:23:34.768GMT",
          "dataRead": 35972800,
          "dataWritten": 1424,
          "description": "Job group for statement 19:\n# Calculate the number of control and treatment samples\nC = test_ranked_df.where(f\"{TREATMENT_COLUMN} == 0\").count()\nT = test_ranked_df.where(f\"{TREATMENT_COLUMN} != 0\").count()\n\n# Add columns to the DataFrame to calculate the control and treatment cumulative sum\ntest_ranked_df = (\n    test_ranked_df.withColumn(\n        \"control_label\",\n        F.when(F.col(TREATMENT_COLUMN) == 0, F.col(LABEL_COLUMN)).otherwise(0),\n    )\n    .withColumn(\n        \"treatment_label\",\n        F.when(F.col(TREATMENT_COLUMN) != 0, F.col(LABEL_COLUMN)).otherwise(0),\n    )\n    .withColumn(\n        \"control_cumsum\",\n        F.sum(\"control_label\").over(Window.orderBy(\"percent_rank\")) / C,\n    )\n    .withColumn(\n        \"treatment_cumsum\",\n        F.sum(\"treatment_label\").over(Window.orderBy(\"percent_rank\")) / T,\n    )\n)\n\n# Display the first 20 rows of the dataframe\ndisplay(test_ranked_df.limit(20))",
          "displayName": "count at NativeMethodAccessorImpl.java:0",
          "jobGroup": "19",
          "jobId": 36,
          "killedTasksSummary": {},
          "name": "count at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 316,
          "stageIds": [
           49
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:23:34.517GMT",
          "usageDescription": ""
         }
        ],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 6,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 19
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 19, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.synapse.widget-view+json": {
       "widget_id": "ad5b29a3-0feb-4bd7-a223-a7aeb494bf08",
       "widget_type": "Synapse.DataFrame"
      },
      "text/plain": [
       "SynapseWidget(Synapse.DataFrame, ad5b29a3-0feb-4bd7-a223-a7aeb494bf08)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Calculate the number of control and treatment samples\n",
    "C = test_ranked_df.where(f\"{TREATMENT_COLUMN} == 0\").count()\n",
    "T = test_ranked_df.where(f\"{TREATMENT_COLUMN} != 0\").count()\n",
    "\n",
    "# Add columns to the DataFrame to calculate the control and treatment cumulative sum\n",
    "test_ranked_df = (\n",
    "    test_ranked_df.withColumn(\n",
    "        \"control_label\",\n",
    "        F.when(F.col(TREATMENT_COLUMN) == 0, F.col(LABEL_COLUMN)).otherwise(0),\n",
    "    )\n",
    "    .withColumn(\n",
    "        \"treatment_label\",\n",
    "        F.when(F.col(TREATMENT_COLUMN) != 0, F.col(LABEL_COLUMN)).otherwise(0),\n",
    "    )\n",
    "    .withColumn(\n",
    "        \"control_cumsum\",\n",
    "        F.sum(\"control_label\").over(Window.orderBy(\"percent_rank\")) / C,\n",
    "    )\n",
    "    .withColumn(\n",
    "        \"treatment_cumsum\",\n",
    "        F.sum(\"treatment_label\").over(Window.orderBy(\"percent_rank\")) / T,\n",
    "    )\n",
    ")\n",
    "\n",
    "# Display the first 20 rows of the dataframe\n",
    "display(test_ranked_df.limit(20))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3564e70-25e8-4aa1-baf6-19bb0fa683e1",
   "metadata": {},
   "source": [
    "Finally, at each percentage, calculate the uplift of the group as the difference between the cumulative percentage of visits between the treatment and control groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4fa91d5-da05-4269-b42f-05fd8dfbe9b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:23:45.5821115Z",
       "execution_start_time": "2023-09-08T23:23:38.9280503Z",
       "livy_statement_state": "available",
       "parent_msg_id": "a6c07c13-ac42-4ef4-9266-3bb8ee5b2c7b",
       "queued_time": "2023-09-08T23:19:41.8685574Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [
         {
          "completionTime": "2023-09-08T23:23:44.219GMT",
          "dataRead": 60354343,
          "dataWritten": 24381543,
          "description": "Job group for statement 20:\ntest_ranked_df = test_ranked_df.withColumn(\"group_uplift\", F.col(\"treatment_cumsum\") - F.col(\"control_cumsum\")).cache()\ndisplay(test_ranked_df.limit(20))",
          "displayName": "getRowsInJsonString at Display.scala:452",
          "jobGroup": "20",
          "jobId": 42,
          "killedTasksSummary": {},
          "name": "getRowsInJsonString at Display.scala:452",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 26,
          "numCompletedStages": 2,
          "numCompletedTasks": 26,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 26,
          "rowCount": 5593545,
          "stageIds": [
           58,
           59
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:23:39.404GMT",
          "usageDescription": ""
         }
        ],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 1,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 20
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 20, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.synapse.widget-view+json": {
       "widget_id": "ace1f9a9-9e27-41a3-829e-75f6320315c3",
       "widget_type": "Synapse.DataFrame"
      },
      "text/plain": [
       "SynapseWidget(Synapse.DataFrame, ace1f9a9-9e27-41a3-829e-75f6320315c3)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_ranked_df = test_ranked_df.withColumn(\"group_uplift\", F.col(\"treatment_cumsum\") - F.col(\"control_cumsum\")).cache()\n",
    "display(test_ranked_df.limit(20))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e9bac10-1ae6-4051-b1ad-97ab170190d3",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Now, plot the uplift curve for the test dataset prediction. Note that you must convert the PySpark DataFrame to a Pandas DataFrame before plotting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09c53ccf-b86b-421b-aa67-63f15f767202",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:23:52.6142735Z",
       "execution_start_time": "2023-09-08T23:23:45.8477135Z",
       "livy_statement_state": "available",
       "parent_msg_id": "cb2affc0-e4af-4627-bbe0-142778c884a8",
       "queued_time": "2023-09-08T23:19:41.8705408Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [
         {
          "completionTime": "2023-09-08T23:23:46.880GMT",
          "dataRead": 127674616,
          "dataWritten": 0,
          "description": "Job group for statement 21:\ndef uplift_plot(uplift_df):\n    \"\"\"\n    Plot the uplift curve\n    \"\"\"\n    gain_x = uplift_df.percent_rank\n    gain_y = uplift_df.group_uplift\n    # Plot the data\n    fig = plt.figure(figsize=(10, 6))\n    mpl.rcParams[\"font.size\"] = 8\n\n    ax = plt.plot(gain_x, gain_y, color=\"#2077B4\", label=\"Normalized Uplift Model\")\n\n    plt.plot(\n        [0, gain_x.max()],\n        [0, gain_y.max()],\n        \"--\",\n        color=\"tab:orange\",\n        label=\"Random Treatment\",\n    )\n    plt.legend()\n    plt.xlabel(\"Porportion Targeted\")\n    plt.ylabel(\"Uplift\")\n    plt.grid(b=True, which=\"major\")\n\n    return fig, ax\n\n\ntest_ranked_pd_df = test_ranked_df.select([\"pred_uplift\", \"percent_rank\", \"group_uplift\"]).toPandas()\nfig, ax = uplift_plot(test_ranked_pd_df)\n\nmlflow.log_figure(fig, \"UpliftCurve.png\")\n",
          "displayName": "toPandas at /tmp/ipykernel_10137/3726734023.py:28",
          "jobGroup": "21",
          "jobId": 43,
          "killedTasksSummary": {},
          "name": "toPandas at /tmp/ipykernel_10137/3726734023.py:28",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 280,
          "stageIds": [
           60,
           61
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:23:46.077GMT",
          "usageDescription": ""
         }
        ],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 1,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 21
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 21, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_10137/3726734023.py:23: MatplotlibDeprecationWarning: The 'b' parameter of grid() has been renamed 'visible' since Matplotlib 3.5; support for the old name will be dropped two minor releases later.\n",
      "  plt.grid(b=True, which=\"major\")\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1QAAAIFCAYAAADVxLy7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAACDt0lEQVR4nOzdd3hUVf7H8ffMpBGS0ENNCDURKQEiSpHeVVaEdXVtKEVE1/XHKoqKggq66q667iIqWFZcdxUWXJXeVBQFRKRJCSQk1NCTkDaTub8/LpkQahKSuTPJ5/U8PNxz752ZzzDHON+cc8+1GYZhICIiIiIiIiVmtzqAiIiIiIiIv1JBJSIiIiIiUkoqqEREREREREpJBZWIiIiIiEgpqaASEREREREpJRVUIiIiIiIipaSCSkREREREpJRUUImIiIiIiJRSgNUBfIXb7ebAgQOEh4djs9msjiMiIiIiIhYxDIOMjAwaNGiA3X7pMSgVVGccOHCAqKgoq2OIiIiIiIiPSE1NpVGjRpc8RwXVGeHh4YD5jxYREWFxGnA6nSxZsoT+/fsTGBhodRzxceovUlLqM1JS6jNSUuozUlK+1GfS09OJiory1AiXooLqjIJpfhERET5TUIWGhhIREWF5hxLfp/4iJaU+IyWlPiMlpT4jJeWLfaY4lwJpUQoREREREZFS0ghVMbndbvLy8rz2ek6nk4CAAHJycsjPz/fa64p/Kugv+fn5PvMbHREREZHKQAVVMeTl5ZGUlITb7fbaaxqGQb169UhNTdWqg3JZBf1lz5491KhRg3r16qnfiIiIiHiBCqrLMAyDgwcP4nA4iIqKuuyyiWXF7XaTmZlJWFiY115T/Jfb7SYjIwO73c7Ro0cBqF+/vsWpRERERCo+FVSX4XK5yMrKokGDBoSGhnrtdQumGIaEhKigkssq6C8RERHY7XbS0tKIjIzE4XBYHU1ERESkQtM39csouH4pKCjI4iQixVNQ+DudTouTiIiIiFR8KqiKSdejiL9QXxURERHxHhVUfigmJoa4uDhcLpdnX0JCAqtWrbIsk81mIzMzE4DBgweze/fucnnus02ePJlHH320yL4PPviA4cOHX/Y5zz5v/fr13HHHHZ5jb7/9NnFxccTHx/PTTz/x6aefXvJ5bDYbr7/+epH9PXr0uGjuS0lOTqZ27dqXPW/VqlUkJCSU6LlFREREpOypoPJTubm5zJo164qe4+yCrCwtWLCAZs2alctzl4eEhAQ+/vhjT/v111/no48+YuPGjWzevPmSBRVAhw4deP/99z3txMREsrOzyy2viIiIiPgOFVQlYBgGWXmucv9jGMZls0yZMoXnn3+erKys844dPnyYoUOH0qZNG1q3bs0777zjORYTE8PUqVPp1asX99xzD5MnT+b222/nxhtvpHnz5tx66638/PPP9O7dm6ZNmzJ+/HjPY//6179yzTXX0L59ezp16sSPP/54wWwxMTFs2bKFtLQ04uPjPX9q167NvffeC8CuXbu44YYbuOaaa2jXrh3Tp0/3PP6///0vcXFxdO7cmeeff77Yn8+5PvjgA/r168ewYcOIj4+nR48epKSknHfe2aM9w4cPZ/fu3dx111307t2bZ555hmXLlhEfH8/YsWMv+DpNmjShVq1arFu3DoD33nvP8z4LrF+/ns6dO9O2bVs6derEd9995zn2j3/8g+bNm3P99dczc+bMIo9bvHgx3bp1o2PHjlx77bV88803pf73EBEREZGyp1X+SiDbmc/Vzywu99fZ+twAQgIuXet26NCB7t2789prr/HUU08VOfbwww8TFxfHvHnzSEtLo2PHjsTHx9OpUycAUlJSWLFiBTabjcmTJ7N+/XrWr19PWFgYHTp04IknnmDhwoW4XC6aNGnC2LFjadmyJXfddZenwPrhhx8YOXIkW7ZsuWjGyMhINm7caL6nrVu54YYbGD9+PPn5+fz+97/no48+Ii4ujqysLK677jquu+46GjVqxOjRo/n++++JjY3l5ZdfvoJ/SVi9ejUbN270PNfYsWNZsGDBRc+fM2cOMTExzJkzh9atW/PBBx/w5ZdfMmfOnEu+zn333cd7771Hhw4d+Oyzz1i3bh3jxo0DzPuY3XLLLbz77rsMGDCA1atXM3z4cBITE9m9ezdTp07l559/pm7dup7HAOzZs4cpU6awaNEiIiIiSExMpEePHiQnJ1/Rv4mIiIiIlB2NUPmxF154gddff51jx44V2b9s2TIefPBBwCxqbrnlFpYvX+45fu+99xZZuGDAgAFUq1YNh8NB27Zt6devH8HBwVStWpXY2Fj27NkDwM8//0yPHj1o3bo1Y8eOZdu2beTl5V0254EDB7j55pt57733aNOmDTt27GDr1q3cdtttxMfH06VLFzIyMti2bRs//PADHTp0IDY2FoAxY8Zc9HkvtvjC2fu7detW5LlWrlxZrBHAkho2bBgLFixg3rx5dOrUierVq3uO7dixg6CgIAYMGODJFBkZyaZNm1i1ahU33HADdevW9WQssGjRIhITE+nevTvx8fGea75SU1PLPL+IiIiIlI5GqEqgSqCDrc8N8MrrFOdLf9OmTbn99tt54YUXzjt2brFxdjssLKzIsZCQEM+2w+E4r+1yucjLy2PYsGGsWrWKjh07kp6eTrVq1cjLy7vkkvIZGRnceOONPPvss/Tu3Rswp07Wrl3bM3p1ts8///zSb/osderUISkpqci+o0ePEhkZWeznKCtVqlRh0KBBPPDAA/z73/8ucswwjAsWfzab7ZKfs2EYDBw4kH/+85/nHbvQ1EURERER8T6NUJWAzWYjNCig3P+UZNnrSZMmMXv2bA4cOODZ17dvX891U0eOHGHevHmeYqa0cnJycDqdREVFAfDmm29e9jEul4vhw4czfPhw7rzzTs/+2NhYQkNDixQKiYmJHD9+nM6dO/Pzzz+zc+dOgPOuKTpb7969Wbx4sae4SE9P5+OPP6Z///6ec7777rsiz9W7d+8S/ftGRERw6tSpYp37pz/9iccff/y8f+u4uDhyc3NZsWIFAN9//z1paWm0adOGXr16sWDBAtLS0gCKLDTSv39/Fi1aVGRa5dq1a4udXURERETKnwoqP1enTh0efvhhDh486Nn3t7/9jU2bNtG2bVt69erFU0895bl+qrQiIiJ47rnn6NSpE927dyc4OPiyj/nuu+9YtmwZn376qWdhimeeeYaAgAC++OILPv30U9q2bcvVV1/NqFGjyM7OJjIyknfeeYebbrqJLl26YLdfvIvGxcXx5ptvcssttxAfH8/111/PnXfeyW9+8xvPOT169GDy5MnEx8fzxRdfFFn8ojj69OnD6dOnadeu3UUXpSjQokULHn300fMKtqCgIObOnctTTz1F27ZteeSRR/jss8+oWrUqbdu25cknn6RLly5069aNBg0aFHm+2bNnM2rUKNq1a8dVV13FG2+8UaL8IiIiIlK+bEZ5XFDihwqmsJ06dYqIiAjP/pycHJKSkmjSpEmRqXDlze12k56eTkRExCWLCrm44i4oURGc3V/y8vIs6bPiX5xOJwsWLGDw4MEEBgZaHUf8gPqMlJT6jJSUL/WZi9UGF6Jv6iIiIiIiYol8t8GfF25nY+pJcPrnfTy1KIVUWCNGjGDEiBFWxxARERGRi2j+5ALCOU3d756hecRWHC2etTpSiamgEhERERERSwSTx+Lgx6lvOw5ZUPfkBmCo1bFKRFP+RERERESkTOQ481m1I43svPzLnvvtriPkEsTn+V1JdDfgzryJ3Lrjei+kLFsaoRIRERERkVI5fjqPjs8vBaBRjSrsO1H0OqiExjX47IEuhTuyjsOKF1hT/Qbu/jIHgNddw/grv8Xpp6WJRqhERERERKTEZv+w11NMAecVUwDr956gyRNf8fWvB3Gtew/e7AjrZxG05HHAXGw8l6AixVSO8/KjW77EP8tAERERERGxxNiP1rN46+Finx9vS6T6J08TYN8DwHZ3FK84bwPMe3c67Da+fLgbg17/FoBD6TmEh/rPrV9UUPmhmJgYQkJCCAkJITs7m3vvvZcnnniiTF9jxIgRJCQk8NBDD5Xp8wI888wz/O9//wMgMTGRyMhIz/r+//nPf4iNjb2i51+1ahV5eXn079//irMW18aNG9m5cye33nqr115TRERExNuaPPFVsc+tSToTAv7N7wJWAZBuVOE113Bm5/fDdVYZsmvqIGw2G32visR18hANq1cp69jlSgVVaeSdvvgxmwMCQ4p5rh0CqxQ9N6hqsSLMmTOH1q1bc+DAAVq1akXv3r3p1KlTsR5rteeee47nnnsOgJ49e/Loo49y4403FjnH7XYDlOqmxqtWrSIzM9PrBdWXX36pgkpERET8kmGY0+82pp4EIDEtE8OAG9vV5y+Ld/Led0kXfey8B7tSMzSIT9encn+Ppvy09wT3vr+OAY51nmJqTn53/uy8naNU8zxuxwuDCAoo/K43/ffxLFiwgECHf12VpIKqNKY1uPixFv3hjs8K2680B2fWhc9t3A3uPavKf70NTNhToigNGjQgNjaWvXv30qlTJ5YvX87TTz9NTk4OTqeTP/3pT9x7772AWbxce+21fP/99xw4cIB+/foxY8YMAPbv38/dd9/NkSNHaNKkCfn5hXNXDx8+zNixY0lMTMQwDB5++GHGjBkDmKNld999N8uWLWPfvn1MmzaNY8eO8fHHH3Ps2DFmzZpFz549i/VeJk+ezO7duzl9+jSJiYksXLiQLVu28Pzzz5OdnU1AQACvvPIK3bt359ChQ9x+++2kp6eTk5NDnz59eOONN/jll1+YMWMGbrebZcuWccstt3D33XeTkJDA2LFj+eqrr8jOzmb27Nm88847/PDDD4SEhDB//nwaNDA/11dffZVPP/0Ul8tFvXr1ePvtt4mKimLy5Mns3LmTjIwMdu/eTb169ZgzZw4ul4tnnnmG9PR04uPjue666zz/riIiIiLedPx0HtWrBGK328h3G2TluQgLDuDAqRwMw+DBjzdQNyKEalUCmfPTvss+3+NzN1302Gu/i+fm9g097UcHxEJuBj1jI0l66QZwD8T4PIvtDYfx2H/zijx2yf91L1JM+TMVVH5u+/btHD161FO0dOjQgdWrV+NwODh+/DgdOnRg4MCB1K9fH4Ddu3d7psS1atWKNWvW0LlzZx5++GG6d+/Os88+y549e2jXrh0DBw4E4OGHHyYuLo558+aRlpZGx44diY+P94yIZWdn8/3337Nu3Tp69OjBq6++ytq1a/n000958skn+f7774v9flauXMmGDRuIjIxkz549TJkyhUWLFhEREUFiYiI9evQgOTmZ6tWr88UXXxAWFkZ+fj6/+c1vmDt3LsOHD2fs2LFkZmby6quvApCcnMyxY8fo3LkzL7zwAq+88gp9+/Zl1apVvPPOO4wbN46///3vTJs2jX/961/s3LmTNWvW4HA4+Oijj3jooYf4/PPPAfjxxx9Zt24dNWvW5LbbbuPtt99m4sSJPPfcc3z55ZfMmTOnrD5aERERkQvKcebz68F0svLy2Xcii8VbDrFyx5ESPMOpK3r9mNpVWfloz6I7Mw7Dsmdh7/fw4I/mLCy7A9vQGVwF7LnGoOnEBQDceV1jWtQNv6IMvkQFVWk8eeDix2yOou3HEi9x7jlV+SObix1h+PDh2Gw2duzYwWuvvUadOnUAOHbsGCNHjmTnzp0EBARw9OhRtm7d6imobrvtNhwOB1WqVCE+Pp7du3fTuXNnVq5cyd/+9jcAmjZtSp8+fTyvtWzZMn755RcAIiMjueWWW1i+fLmnoPrd734HmMVcdna2Z9pbx44d2bOnZCNuN954I5GRkQAsWrSIxMREunfvXuSc1NRU6tWrx+OPP87q1asxDIO0tDTi4+MZPnz4BZ83LCyMG264wZOzUaNGxMfHe3IuXWquUDN//nzWr19Px44dAcjPz8fhKPxMBw0aRM2aNQHo3LkzmzcX/zMTERG5lOOn80jLyMEw4MzsK8DwbBuY+40zK6OZ22eOFT7ggud4nsMwOOupL/Jc5v6zX/dkVh5HM3JpUqcqDrsdzjyPYYDbMM56ncLnc7pcbDxmw7blEHaH48zrmee4z7zGySwnby7fxVX1zWupbbaCcDbPdsEum8121nbhMVvhgzzHCs4sep7nmT07z91nPrbw9QC++OUAAXYbo7s3pV61ELbsM4sRtwH5hoHbbeA2DM92vhtcbjcb9p4grn4ENsBus2G3m89pP/M+7DazfTIrjxynmwbVq+A2zjyXu/DvfDfku91sSDlJUIAdh81GdjmugtexcQ1+2nuC134Xz//9Z+N5x3dOHVR0Sl6+E9a+C6tehNx0wAa7V0Lc4CKPs9ls5qhVBaSCqjSKeZ1TeZ5bcA3VsmXLuOmmm+jduzdt2rRh7Nix3HTTTcydOxebzUaHDh3IycnxPC4kpPD6LofDgcvlKtbrnf/DqrBd8JwFhcfZ7eI+f4GwsDDPtmEYDBw4kH/+85/nnffCCy9w7NgxfvzxR0JCQhg/fnyR93mu4OBgz7bD4bjov4NhGDz99NPcd999F3ye0v77iYiI7Duexbq9xzEMqBMeTHZePmuTjpPvNli45SCH03OtjlgOHHyw8+JTxgp8v/uYF7KUnstt8Naq3SV+3Nqk48U+d9vB9Muek+dyF2k3rhXK3mOFl5YMbd+QxwbGMv/n/by8aAe3dGjI7Z2iCQ1yUCc8hLDgAEIC7bgNs3A8kpmL0+WmUc3QC77e2dP5LijpW1g4AdK2me0GHWDwq9Co42XfS0WigsrP9e3blwceeICnn36azz//nBMnTtC4cWNsNhvffPONZ2Tpcnr37s17773HpEmTSE5OZvny5fTt29fzGu+88w5TpkzhyJEjzJs3zytT2/r378+UKVPYsmULrVu3BmDt2rV06tSJEydOUK9ePUJCQjh8+DCfffaZZ6QsIiKC/fv3l+o1hwwZwhtvvMHNN99MzZo1cTqdbNmyhfbt21/ycREREZw6dWXD5yIiFUlWnousvPwzv2E3/7jc5m/wT+e5qFk1iFpVg8nMdVI9NIhAh90zunHuL/FKKj3bya60DE6cdpLvNnC63bjyDfJcbjbvP4Uz331m9OXMSMmZ0ZK1ScfZdyKbYR0acTI7jyqBDtxnRmA487f7rNGbgpEZt2GQcjyLvceycNhtOOw2As76+0SWs0T5a4cFnzNScoERlDP/RkVGXi503plG0RGawn/j80ZuLnLO0cxc0jLMgq9V/Qjz+W3myAsUjh4VvK75WIMTx09Qq1ZNc1TmTD7bmZGZgnP3HsuiaZ2qDD3z5b3IiNrZI2iFg3Bn9hlFRtUKzj97ZK3wmHHOeWeNpp113rn7MODDNcnsPZZFj5Z1CAm0ExLoICw4gKiaodht5udss5lLfztsNmw2Gw5zEI8cZz61w4OL9B/PiN6Zdr5hkJHjxG1AzdAg7Dawn+k/DpvN3D7z71UrLIjI8BDchkFUjVDs9gv/t/JAz+Y80LP5BY8BOM48rG5EKZcmd+XB/LGwZa7ZrlIT+j4L7e+GUiwo5u9UUFUAkyZNonnz5vz000+89NJLjBs3jpdeeolWrVpx7bXXFus53njjDe6++24+++wzWrZs6SmmAP72t78xduxY2rZti9vt5qmnnvLKioItWrRg9uzZjBo1iuzsbPLy8ujQoQMff/wxDz/8ML/97W+Jj4+nYcOGRfIOHTqUjz76iPj4eM+iFMV11113cezYMXr27InNZsPlcjFy5MjLFlR9+vTh1VdfpV27dnTu3FmLUoj4oIwcJxtTTxJgt1GtSiCNa1Ul1+WmSqCDKkGOyz9BKbndBpl5rjNTgcxpPNl5bg6eyibfbZCZ6+JIRi6rE4+y63AGHRvXxDgzfcg8H/Nv95kpRYbBniOnSTpqriLbqEYVz3Sr07kuTmU7aduoGnkuN4fTc4p8me/WojbRNUMJPnMh+OncfBpUD+H1ZbuIqRVK1+a1z3y5thX5Ymy3geF2k5Rs55eFO0g9kcOyXw/TPDKMxLRMAO7r2sTzOv9Zl8LpvJJNSXKcuYDeZoNAh73Ib+Ijw4MxKFrYnD3d7Ox9WSV83QuZu+HyF+pfTEHxmHeZ8+LqhRMc6CAs2MGxzDyua1qL07ku7u/RjGaRYZd5tH9wOp0sWLCAwYOvITAw0Oo4pXZvtyaXP6myCQgCZw5gg4T7oPfTEFrT6lSWsRnGufV+5ZSenk61atU4deqU555IADk5OSQlJdGkSZMi073Km9vtJj09nYiIiFItHS6Vy9n9JS8vz5I+K/6l8IvOYL/9opPncrPvRBY5znw2pp6kSpCDgVfXJzEtk7VJx/g59STJR0+z9cDlp9G0iAzjvm5NCAsJIMBuI8/lxm634cx343QZOPPdZOa62HEog4SYmgQH2Nl9JJODp3KoEx7sKZZcZ133cCrbScqxLLYfSsddif9P6zjrN+0BdhuOM78aP1nCUZsr0alJTQIddgIdNqoGm79Lvuqsa1sKBmZyXW7m/byf6JqhxNULJzPXRWy9cM95BaMq9jMjM56C0144MtSmYTWqBDnMkbF8N7kuN3uOnCYk0E7LuuFEXWRqVUVUEX7OyFl2r4TIVhBe12yfTIGs49Agvsxewpf6zMVqgwvRCJWIVGq5rnwyclxUrxJIwJmLbN1u46LTKM5mGAYnspyEBjkICSz9CIf7zG/lbTYbuc58ggLs2Gw2th1I54a/mXeNnzzkaqoEOjielUdokIMdhzJw5bsJDnQQYLcR4LCTeNj8st/7qkji6pk//F35bnJcbrLz8gkJtJOVl8/XO46QciyT2rmQeiKL7Yez2J2WicNu80zLsZ8ZlbDbbLgNg0VbD5GR7WLH4QyubVKTH89cF3BNTE3WJZvb17eoTUaOi42pJ6keGkizOmE4893kucwvlQUjKgX+1L8l18TUpGGNKp7368p3k3oiC7cbAhyFX1gLPg3HmX3TV+7mUPr5102Op3jTnM+1Ky2Tif8t3iIz834u3ZTiAjYbOGw2XG6DxrVCCQsOIDQogHXJx7HbYGzPZoQHBxaZQlQw/cd+ZipRdl4+73+XzIO9mtOyXuFKWUlHMwkNDCAwwEagw06A3UbysSzz76On2X8y2zPF58ek4zSpXRWHzcbcDftoXCuUoe0bFllc4OxFEFwuN7t376Zp06bsPprFiu1pNKhehQMns+navBbtGlUv8j7dBtzXrQl1woO5kFxnPs4zlebW/aeoGhxAgMNGjdAg8t0G2Xn5Zn8PdBSZIlZYyBT2jbOnk9ltUDU4gBpVg0r1+Tzcp0WpHncpBQsuiPilk6mw+En49X/Q7vcw9C1zf/Vo84+ooBKRyuXE6TzWJh3H6XbzXeJR/r029ZLnt24YwZb9lx/hKBASaGdQ6/os3HKQHKebuHrhbD+UAcDvr40mPCSArLx8vvzlQImuq5j8v63FOm/ljiO8snhHMZ81ADasLnaGAj+edZF1QTEF8O2uo57tk1lOftp74pLP85clO0v82hdSKyyIY5nnT7CqGuSga/PaLNl2mKdvuIp+reoRVdMs3gzD4HRePgdPZjP4jW9xnfliXzDtrEDX5rUIDnAQ6LCx9UA69aqFEBESiMtt8MOeYwQH2LmlQyOqBDqKjMTY7eZzFTzn0PaNqBEaeOZaiyu7PqjAiK7nT0OKj6p+3r7OzS7/XK/e2u6y55i/Od7F4AEty+Q3x8GBDgpKrWub1rrgOcWILiLlxZUL378J3/7FvKeqzQFVqoPbXSmvk7oUFVQiUqFl5bl4b3VSqb+8l6SYAshxuouMYBQUUwD/+jGlVBnO1TwyjFxXPi0iw0nPdnJNk5os2XqI3UdOX/7B56gSaMdht5OZ62Jo+4YE2G2ei6YLLpzelZZJriufPWee//ZO0UTXCqVmaBBVgwM4nJ5DZo6LRjWr8F3iUYIC7NQJC6ZhjSrUCQ8mwG4nyGFn34ksPv/lANWrBPLV5oOeURooLGSCAuwkNK5ByvEs4upFUDXY4Rkx+eKXAzStU5UqgQ5O5+XTJy6SR/q1JOzMFK7D6TlUOXOx+OVGGG02G2HBAbSoG86uaUWX9jUMg+On86gRGlSskUoRkQpn11Jz9b7jZ25/E90FBr8C9Vpbm8tHqaAqJl1qJv7C7XZf/iQ/ZhjGRX/DbxjmNSxpGbm8tTKRjy9RwNSLCKFpHfNWBaO7N6VqcADLfz1MoMNO7bBg/rthH01qV+X46bwiIy8Fnr+5NZm5Lm5sU5+1ycdZuT2NBZsP0qB6FfadyPacd22TmtSrFsJ3ice4rmlN6kaEsP9kNqdzXWzZf4p2UdUZ070ptcOCCXDY2XEonbVJxxnWoRFN64QVe7GECQPjAHOK35srEjl0KpuG1UMZltCIWlWDPFP3MnKcRIQEYsddLvPUb+nQ6BJHazE8IQqAN8/sKfjMitzT5CL+dvulF4cp9WpV57DZbNQKu/A0NRGRCu+nD+GLh83tsLrQ/wVo89uzbxIm51BBdRmBgYHYbDaOHDlCnTp1ymyqxuW43W7y8vLIycnRohRyWQX9JT09naNHj2K32wkKKt31A95QsGxyoMPOO9/s4ZfUk6zYnnbeeU/fcBVzN+zn14vcmyOmdlWSj5ZsVGbU9U0Y17P5Ba+vuCamcIWie7rEeLad+W4OnMwmumboBX8GNKoZyi0dGhUpEqB0Sz83qV2Vga3rl/hxBQIcdv6vX8uLHi+41svp9I3C22azEejQ/6RFRHxGq9/Aqpeg9S3Q43EI0TWAl6OC6jIcDgeNGjVi3759JCcne+11DcMgOzubKlWqeK2IE/91dn+pWrUq0dHRXivET2U5ych1kpaeizPfzS/7TlI1KIC4+uG0aVid7YfS2X4wg0PpOby2tGTT7l746tdLHi9OMRUWHMB3E3sTEVL6UZhAh53GtYp/4239NysiIn7BMGDHQnPBiZvfMkehqlSHP/wEQZVnRcorpYKqGMLCwmjRogVOp/eWeHU6nXzzzTd0797d8mUjxfcV9JfevXsTEhJSbl/of9p7nMVbD5OV67rkdLqyVHBtTeNaodx1XWO+333MM5oVERJAeo4LgK8e7ka9alU8N9MMDrB7Vu0TERGRcxzbDYuegF1LzHbLAXD1UHNbxVSJqKAqJofDgcNRfjd+vNDruVwuQkJCVFDJZRX0l4CAgDIvpvLdBnYbNJ24oEyf9++/78DRzFw6NalJw+pVqBocgN0GGTkuqgYHeFZIO9fI65uWaQ4REZFKJS/LXLnv+79Bfh7YA6HLQ9C8n9XJ/JYKKhHxOHE6jzH/XM/6yyx3XaBj4xp0b1GH2uHB3NiuPj/vPUF8VA2mfLGV+tVCiKldldphwXSMqVHsKXcRVfQLBBERkTJnGObUvsVPwakztwxp1hsGvQy1y/7+a5WJCioRASAz10WH55de8pyExjX4/XXRvLJoB+/cnUDrhtWKHO8RGwnAX38XX14xRUREpDTc+eZiE6dSoVo0DJwGcTdq9b4y4NWCateuXdxzzz0cPXqU6tWr88EHH9CqVavzzps1axYvvfQSbrebPn36MH36dAICAsjMzGTYsGH89NNPABw9WnQp4x9//JH777+frKwsoqKimD17NvXrl361LJGKKCPHyYyvdzN95e5iP6ZV/QjeurMj0bXMOdVD219qaWwRERHxCbmZ4AiEgGBwBMDgV2HPKuj2f7pOqgx5taC6//77GTNmDCNGjGDOnDmMHDmSNWvWFDknKSmJSZMm8fPPPxMZGclvfvMbZs2axf33309gYCATJkygVq1a9O3bt8jjDMPgjjvuYObMmfTs2ZNXX32V8ePH88knn3jzLYr4rI9/2MvT87dc9rwh7RrwxmXu9yMiIiI+zDBgy1xY8jR0Gg3X/8ncH9PV/CNlymtLYKWlpbFhwwbuvPNOAIYNG0ZSUtJ5S5HPmTOHoUOHUrduXWw2G2PHjvUURcHBwfTp04fq1auf9/zr168nODiYnj17AmbxNn/+fK+uzCfiawzDIPV4Fk2e+KpYxdSd1zVWMSUiIuLPDm+DD2+CuSMh4yBsnmtO95Ny47URqtTUVBo0aEBAgPmSNpuN6OhoUlJSiImJ8ZyXkpJC48aNPe2YmBhSUi6/PPO5jwsPDyc8PJyDBw8SHR193vm5ubnk5uZ62unp5o1DnU6nTxRhBRl8IYv4vnP7y7YD6dz81g8XPf8ft8fTr1XkJZ9LKjb9jJGSUp+RklKf8bKcdOzfvox93bvYjHyMgBDcXR7B3fkhyHebf3ycL/WZkmTw6pS/c5dzNgzjsudd7JwreX6AF198kSlTppy3f8mSJYSG+s6c0qVLL71IgEgBw4B//W8pqw/b+frghQefpya4qBoIzuT1LEj2bj7xTfoZIyWlPiMlpT5T/mpnbKVj8gwCXacAOFCtI1sa/p7sjDqwZIXF6UrOF/pMVlZWsc/1WkEVFRXFvn37PPfKMQyD1NTU80aPoqOji0wD3Lt37wVHmM517uMyMjLIyMi46KIUEydOZPz48Z52eno6UVFR9O/fn4iIiJK9uXLgdDpZunQp/fr1032o5JIOnsqhx6vfXPa8rx/tTv1qIV5IJP5AP2OkpNRnpKTUZ7zoeCwB77yGUbMZ+f1fpE6z3vSyOlMp+FKfKZi9VhxeK6giIyNp3749s2fPZsSIEcydO5eYmJgi0/3AvLaqW7duPPPMM0RGRjJjxgxuu+22yz5/x44dycnJYdWqVfTs2ZO3336bm2+++aIfRnBwMMHBweftDwwMtPwDPJuv5RHf8cOeY9z+zsWn9d3XtQmTbjp/FU2Rs+lnjJSU+oyUlPpMOcg+AYnLoc1ws103Du6aD40SCAg4//utv/GFPlOS1/fqlL+3336bESNGMG3aNCIiIvjwww8BGDVqFEOGDGHIkCE0bdqUKVOm0LVrV9xuN71792bkyJGe5+jQoQMHDx7kxIkTNGrUiF69evHRRx9ht9uZPXs2Y8eOJTs7m4YNGzJ79mxvvj2RMpPnchMUYMeV78Zht503DXb8p78w/+f95z2uQ3R1ggLsvPn7DtQO8/8fqCIiInIWtxs2fgzLJkPWMajZBBp2NI9p9T7LeLWgio2NPW+ZdICZM2cWaY8ePZrRo0df8Dk2bNhw0efv3Lkzv/zyy5WFFLHAqA/XsfzXtEue8+dhbXl87qYLHhvdLYZWrkRuuKGT5b/RERERkXJw4Gf46lHYv95s14kzCyyxnFcLKhEptGnfSX7z9++Kff6Fiqm37+pI/6vr4XQ6WbAgsSzjiYiIiC/IOg7Ln4OfPgAMCAqHnk/AtfebN+0Vy6mgErFArjO/RMXUhfz6/EBCAh1llEhERER8jtsNs/rDsV1mu82t0P95CK9nbS4pQgWViBct3XaYMf9cf8Fj30zoRVTNCy/Z3+SJrwDo3LQW/xpzXbnlExERER9it0PnB2HtuzD4FV0n5aNUUIl4ycuLtvPWqt1F9tWsGsRPk/pd9rFJL91QXrFERETEV2QeMRecaDkAWg0x93W4B9rfBQ59bfdV+mREytkPu4/x3Jfb+PVg0fsZ3N4pmmm3tLEolYiIiPiMfBesfw9WvgA5pyDpG4gdZF4jZbcDdqsTyiWooBIpJwXT9M51a0IjXhrWtshS6CIiIlJJ7V0DCx6Dw5vNdr22MPhVLTjhR1RQiZSDt7/efcH918TU5M/D23k5jYiIiPicjEOw9BnY9B+zHVId+kyCjveCXYtO+RMVVCJl7J1vdvPSwu3n7f9+Ym/qV6tiQSIRERHxOWnbzhRTNuhwN/R5FqrWsjqVlIIKKpEysmb3UX7/7o9F9i0d353mkeEWJRIRERGfknGocMnzZr2h+2PmtVINO1qbS66IrnATKQOfb9x/XjEFqJgSERERSD8Ac+6DNxPMoqpA76dVTFUAGqESuQIXGpUC2DJlAFWD9Z+XiIhIpebKgx+mw9cvg/M02OyweyXE3251MilD+sYnUkr7jmddsJja8+JgreAnIiJS2e1eAQsmwLFdZrtRJ7jhVaivxakqGhVUIsWU7zbo/vJKDpzMvuDx38Q34PXb2ns5lYiIiPgUwzCn9239r9muGgn9noO2vztzTympaFRQiVxGriufD79P5sUF56/cVyDppRu8mEhERER8ls0GEQ3A5oBOY6DXRAipZnUqKUcqqEQuYtuBU9zwt9WXPe/rx3p5IY2IiIj4rJ1LoFpDqHu12e75BMT/vrAtFZoKKpEL+DnlBLdM//6Cx5b9qQdNalXFbtd1UiIiIpXa8SRYNBF2LoSo6+C+ReYIVXC4iqlKRAWVyDlynPkXLKZ6x0Uya8Q1FiQSERERn+LMhtWvwerXIT8X7AEQdQ3k50FAsNXpxMtUUImc46pJi4q0tWqfiIiIAOaCEzsWwKIn4GSKua9JDxj8CtSJtTabWEYFlcgZX206SNLRzCL7VEyJiIiIx/av4D93mNsRDWHAVGh1sznNTyotFVRS6X26LpXH5246b/+cBzqrmBIREZFCLQdCw47mqFT3RyGoqtWJxAeooJJKrckTX130WMfGNb2YRERERHyKYcC2z2HdTLhjDgSGgCMARi4Fu8PqdOJDdHcxqbQuVUz9/Ew/LyYRERERn3JkB3x0M3x2DyR/C+veLTymYkrOoREqqZR+3HPsvH26Oa+IiEgll5sBX78MP0wHtwscwdDt/+CaUVYnEx+mgkoqHVe+m9ve+cHT/uC+TvRoWcfCRCIiImIpw4Atc2HJ05Bx0NzXchAMfBFqNrE2m/g8FVRSqXz4fTKT/7e1yD4VUyIiIpWczQabPzOLqRpNYNCfoeUAq1OJn1BBJRWeK9/NPe+t5fvdmuYnIiIiZ+ScAnc+hJ5ZhGrgi9AwAbr8wVyAQqSYtCiFVGiGYdDiqYUXLKZ2vDDIgkQiIiJiKcOAjZ/AmwmwdFLh/ppNocdjKqakxDRCJRXaa0t3XnC/RqZEREQqoYObYMFjkHrmWurUtZB3WveTkiuigkoqrHy3wZsrEj3tmFqhTBx8Ff2vrmdhKhEREfG67BOwYiqsnwWGGwKrmqNR1z0IAUFWpxM/p4JKKqzmTy4o0l75WC+LkoiIiIhlUn6Ef98OWWem/199C/R/Aao1tDaXVBgqqKRCmvjfTUXau6bqeikREZFKqU4sYIM6cTDoZWjaw+pEUsGooJIK538b9/Pvtame9gM9mxHg0PorIiIilcLpY/DLv6DzQ+Zy6FWqwz1fQO0W4Ai0Op1UQCqopEJx5bv54783Ftk3YWCcNWFERETEe9z58NMHsOJ585qpiIbQ+hbzWN1WlkaTik0FlVQYryzazvRVu4vs02p+IiIilUDqWljwKBz8xWzXbQ3VGlmbSSoNFVRSISzYfFDFlIiISGWTeQSWTYaNs812cDXo/RQkjASHvuaKd6inSYXw4McbirT/PeY6i5KIiIiI1/znzsJ7SrW/E/pMhrA6lkaSykdX6ovfSz56ukj7lg4NubZpLYvSiIiISLkyjMLt3k9B/XgYuQx+8w8VU2IJjVCJ3+v16irP9p4XB2Oz2awLIyIiIuUj4xAsmQR1r4Zuj5j7mnSHMavM1fxELKKCSvzW8dN5dHx+aZF9KqZEREQqmHwn/Pg2rHoJ8jJgRzgk3AchEeZx/b9fLKaCSvzKh98nM/l/Wy94TDfvFRERqWCSvoEFj8GR7Wa7YUcY/GphMSXiA1RQiV+5WDE1rEMj3bxXRESkokg/CIufhK3/NduhtaDvZIi/E+z6/734FhVU4jeaPPHVBfd/POpaujSv7eU0IiIiUm7yMuHXL8BmN5dA7/UkhNa0OpXIBamgEr+wYe+JIu3gADvfPdGbmlWDdN2UiIhIRXBkB9SJNbdrt4AbXoUGHaB+W2tziVyGxkzF52Xn5TPsre+L7Nv+wiBqhQWrmBIREfF3J1Pg33fAP66FAxsL93ccoWJK/IJGqMRnuN0GH/2wl0Oncrirc2MaVK8CQKtnFhU5b8+Lg62IJyIiImXJmQPfvwnf/gVc2WBzwL510CDe6mQiJaKCSnxCRo6TtpOXeNozvt59wfPeuTtBo1IiIiL+budiWPg4nEgy2427wuBXzHtMifgZFVRiKcMw+L//bOTzjQcue27diGD6tarrhVQiIiJSbuY9AL/8y9wOrw/9X4DWw3Q/KfFbKqjEUk0nLij2uT882bcck4iIiIhXNEqAzZ/CdeOgxwQIDrc6kcgV0aIUYpmMHOcF9ydOK7xG6taERgCM79fSK5lERESkDBkGth0LIHFZ4b6OI+DBtdD/eRVTUiFohEossWTrIe7/6Kci+/47rgvto2sAkPTSDRiGgc1m48/D21kRUURERK7EsUSu2/0XAjZugmrR8NBaCKwCdgfUamZ1OpEyo4JKvC7XlX9eMZX00g3nnafFJ0RERPxQ3mn45lUC1vyduvl5GI4gbG1/C4ZhdTKRcqGCSryu31+/tjqCiIiIlDXDgG3zYfFTkL4fG3A4oi0173iXwLpxVqcTKTcqqMTrUo9ne7Yf6NmMxwbEWphGREREysS+9fDZCHO7ejSuvi/wQ6LB4Jqa3icVmwoq8aq9x04XaU8YqN9YiYiI+C23G+xn1jiLugZaD4dazaHbIxgEwO7ir+Yr4q+0yp94Vc9XVnm2d04dZF0QERERKT3DgE2fwfRrIeNw4f5hM6HXRHPxCZFKQiNUUu4Mw6DVM4vIcbqL7A90qJ4XERHxO4e3woLHYO93Zvv7v8GAqea2FpSSSkgFlZSrHGc+V01adN7+nrF1LEgjIiIipZZ9Ela9CGvfBSMfAqpA90eh80NWJxOxlAoqKVcXKqYAZt1zjZeTiIiISKn98h9Y8hScPmK2rxoCA6ZB9Shrc4n4ABVU4lU3tK3P33/fweoYIiIiUhIHN5rFVK0WMPhlaNbb6kQiPkMFlZSbZz/fUqS98tGexNSualEaERERKbas45CbATUam+2eT0C1KLhmFAQEWZtNxMdoVQApF4Zh8M81ez3trc8NUDElIiLi69xu+OkDeLMjzBtrruYHEFINOo9TMSVyARqhkjJnGAZNJxa970RokLqaiIiIT9v/E3z1KBzYYLazT5jT/MIirc0l4uP0LVfK3LnF1Nqn+lqURERERC7r9DFYPgU2/BMwICjcvJdUpzHgCLQ6nYjPU0ElZSrfbZy3r054sAVJRERE5LIOb4X3B0POSbPd9jbo9xyE17U0log/UUElZeZweg7XTVteZN9XD3ezKI2IiIhcVu1YiGgA1RrB4FehcWerE4n4HRVUUmbOLaaSXrrBoiQiIiJyQZlHYM2b0PNJCAwBRwDcMQfC6prbIlJi+i9HysS2A+lWRxAREZGLyXfBupmwchrknoLgCOj+qHmsWkNrs4n4ORVUUiZu+Nu3nu3fXRPFn/rHWphGREREPJK/gwWPQdpWs12/HTTpYW0mkQpEBZWUuZeGtbU6goiIiKQfhKXPwOZPzXaVGtDnGehwD9gd1mYTqUBUUMkVOZ3rovWziz3thJgaFqYRERERj0WPw7bPARt0HGEWU6E1rU4lUuHYvfliu3btokuXLrRs2ZJOnTqxbdu2C543a9YsWrRoQbNmzRgzZgwul8tz7MsvvyQuLo7mzZszbNgwMjMzPcdmz55N27ZtiY+Pp3379ixcuLDc31Nld3YxBfDP+661KImIiIjgzi/c7vMsNO4Ko1fATa+rmBIpJ14tqO6//37GjBnDzp07mTBhAiNHjjzvnKSkJCZNmsTq1atJTEzk0KFDzJo1C4DMzExGjhzJ/PnzSUxMpH79+kydOhWA48ePM27cOBYvXszGjRt58803ueeee7z59iqdjBznefuqBGkKgYiIiNed2gefjYAvHyncV6sZ3LsAGnawKpVIpeC1KX9paWls2LCBJUuWADBs2DAeeughkpOTiYmJ8Zw3Z84chg4dSt265g3lxo4dy8svv8z999/PwoULSUhIIC4uDoBx48YxePBgXnzxRdxuN4ZheEasTp48SaNGjS6aJzc3l9zcXE87Pd1cpc7pdOJ0nl8oeFtBBl/IcjEPfbyhSLt7i9o+nbci84f+Ir5FfUZKSn3GR7lysa+dgX31X7A5szDsAbi6/gkirF+5T31GSsqX+kxJMnitoEpNTaVBgwYEBJgvabPZiI6OJiUlpUhBlZKSQuPGjT3tmJgYUlJSLnps//79uN1uateuzYwZM+jQoQM1a9YkOzubZcuWXTTPiy++yJQpU87bv2TJEkJDQ6/07ZaZpUuXWh3hor7ZVdh9XurkIsRxiAULFliYSHy5v4hvUp+RklKf8R110jfRdt9swnIPAXCsaks2Nbqb9NW/AL9YG+4s6jNSUr7QZ7Kysop9rlcXpbDZbEXahmFc9rxzzzn3OQqkp6czffp01q9fT2xsLF988QXDhw9n27ZtniLubBMnTmT8+PFFHh8VFUX//v2JiIgo9nsqL06nk6VLl9KvXz8CAwOtjnMeZ74b1hQWrLfcNNjCNOLr/UV8j/qMlJT6jA/JOIRj0QTsu81fYhpVI8nvM5mI1r+l20W+J1lBfUZKypf6TMHsteLwWkEVFRXFvn37cLlcBAQEYBgGqampREdHFzkvOjqa5ORkT3vv3r2ec6Kjo1mxYoXnWHJyMg0bNsRut7NkyRKqVatGbKx5/6ObbrqJ++67j9TUVJo0aXJenuDgYIKDg8/bHxgYaPkHeDZfy1Pgz4sLFxT59vFePpmxMvLV/iK+S31GSkp9xgeEVIXUH8DmgOsewNbjcQJCrP9l8MWoz0hJ+UKfKcnre21RisjISNq3b8/s2bMBmDt3LjExMUWm+4F5bdW8efM4fPgwhmEwY8YMbrvtNgAGDhzIunXr2L59OwDTp0/3HGvatCkbNmwgLS0NgDVr1uB2u2nY0Po5xBXRrNVJnu1GNXxniqSIiEiFlPIDFMzaCa0JQ9+GB76DAVPBh4spkcrAq1P+3n77bUaMGMG0adOIiIjgww8/BGDUqFEMGTKEIUOG0LRpU6ZMmULXrl1xu9307t3bsxpgeHg4M2fO5Oabb8blctGmTRvPc3To0IGJEyfSs2dPT1X76aefEhQU5M23KCIiIlJ2ju+BhU/ArsXw2w/h6pvN/S37WxpLRAp5taCKjY1lzZo15+2fOXNmkfbo0aMZPXr0BZ+joPC6kD/+8Y/88Y9/vPKgIiIiIlbKy4LVr8F3b0B+LtgD4WSK1alE5AK8WlBJxZDncnu2H+nbwsIkIiIiFYxhwK9fwOIn4VSqua9pLxj0MtRpaW02EbkgFVRSYrFPL/Rs39+jmYVJREREKpgFj8K6MzN3IhrBwGlw1RDwodX7RKQory1KIRVTSKDD6ggiIiIVR9wN4AiC6x+Fh9ZCq9+omBLxcRqhkhLJdeZ7tltEhlmYRERExM8ZBmz9L+Sdhg53m/ua9YZHNkN4PWuziUixqaCSEhn/aeGd1xc+0t3CJCIiIn4s7VdY8BgkfwtBYdC8H0TUN4+pmBLxKyqopNgMw2DB5oOetsOuKQgiIiIlkpMOX/8ZfpwBbhcEhECXP0CV6lYnE5FSUkElxTb8re+tjiAiIuKfDAM2fQpLJ0HmYXNf7A3mohM1YiyNJiJXRgWVFNuGlJOe7T0vDrYuiIiIiL85kQSfjzNHpWo2NZdBb9HP6lQiUgZUUEmxGIZRpG3TikMiIiKX5sqDgCBzu2ZT6DYeAoLNKX4BwdZmE5Eyo2XTpViaTlzg2X5sQKyFSURERHyc2w0/z4bX28DBTYX7ez8F3R9VMSVSwaigkhIb16u51RFERER804GN8F5/+PxByDwEP0y3OpGIlDNN+ZPL2nYg3bPdIbq6dUFERER8VdZxWPE8rH8fMMyl0Hs8DteOtTqZiJQzFVRyWf/3n58923Me6GJhEhERER/0y79h0UTIPm622/wW+j0HEQ2szSUiXqGCSi5r5+FMz7YWoxARETlHTrpZTEW2gsGvQEw3qxOJiBepoJJLysx1WR1BRETEt5w+Cqf2QYN4s51wHwRVhba3giPQ0mgi4n1alEIuqc2ziz3bv4nX1AUREanE3Pmw9l14swN8dg84c8z9jgBof4eKKZFKSiNUUmyv39be6ggiIiLWSPkBFjwKhzab7erR5ip+NWIsjSUi1lNBJSIiInIxGYdh2bPwyydmO6Qa9J5kTvOzO6zNJiI+QQWVXJRhGJ7tp264ysIkIiIiFjiZCm91gdx0wAYd7oI+z0LV2lYnExEfooJKLurPi7Z7tm+9JsrCJCIiIhaoHgWNu0DmYRj8F2jU0epEIuKDVFDJRb399R7PdkSILrQVEZEKLv0grJxqjkKF1TH3DX0bgiPArnW8ROTCVFCJiIhI5ebKgx/fgq9fhrxMwIDf/MM8VqW6lclExA+ooBIREZHKa/dKWDgBju402406wTWjrc0kIn5FBZVc0Edrkj3bwQGa5iAiIhXMqX2w+EnY9rnZDq0N/Z6Ddrdrep+IlIgKKrmgU9lOz/aWKQMsTCIiIlIO1vzDLKZsdug0BnpO1PQ+ESkVFVRyQX9ZstOzHeDQb+pERKQCyMuCoFBzu8fjcDLFLKTqtbY2l4j4NRVUIiIiUrGd2AuLJpr3k7rnC7DZzNGo2z62OpmIVAAqqOQ8mbkuz/ZDvZtbmEREROQKOLPhu7/B6r+CKwdsDji8Beq1sTqZiFQgKqjkPG2eXezZ/lP/WAuTiIiIlIJhwI6FsOgJOLnX3BdzPQx+BSKvsjabiFQ4KqhERESk4jh9DOaPhV1LzHZ4AxgwFa4eak71ExEpYyqopIgOzy2xOoKIiEjphUSYi03YA6HLQ3D9oxAcZnUqEanAVFBJESeyCpdL3zS5v4VJREREisEwYOdiaNYbAoLAEQhDZ0BQGNRuYXU6EakEtB62XFR4SKDVEURERC7u6C74aCh88jv44R+F+xu0VzElIl6jESrxyHXlWx1BRETk8nIz4ZuXYc10cDvBEQyG2+pUIlJJqaASjx92H/Ns//r8QAuTiIiIXIBhwJa5sGQSZBww97UYAINegppNrc0mIpWWCirxGPH+Os92SKDDwiQiIiIXsPw5855SADViYOCfIVa/ABQRa+kaKhEREfEP8b+H4GrQ6ykY96OKKRHxCRqhkvMMbF3P6ggiIlLZGQZs+hSO7YLeT5v7areA8VshONzabCIiZ1FBJec5ddbS6SIiIl53aDMseAxS1gA2uOomqN/OPKZiSkR8jAoqAeBIRo5ne+otbSxMIiIilVb2CVg5DdbNNFftCwyF7o9BnTirk4mIXJQKKgHgy18OerZjaoVamERERCodtxs2fgzLJkPWUXPf1UOh/wtQrZGl0URELkcFlQDw3JfbPNs2m83CJCIiUunknISlk8wRqtqxMPhlaNrT6lQiIsWigkpERES8L+cUBEeAzQahNaH/VMg6BteOhYAgq9OJiBSblk0XDMOwOoKIiFQW7nxY/x680Q62f1W4v/0d0PVhFVMi4nc0QiX8d8N+z/Z3T/S2MImIiFRo+9bDV3+CgxvN9s+z4aobLY0kInKlVFAJ834uLKgaVK9iYRIREamQTh+FZc+aBRSYU/16PQXXjLI2l4hIGVBBJXyXeNTqCCIiUlFtngNfjTevmQKIvwP6ToawSEtjiYiUFRVUlVy+W9dPiYhIOQqpbhZT9drC4Fch+lqrE4mIlCkVVJXc5P9t8Ww3rVPVwiQiIlIhZByGw5uheV+z3aIv/P5Ts213WJtNRKQcqKCq5Gb/kOLZ/mxsFwuTiIiIX8t3wtp3YdWLZvuh9RBe19xuOcC6XCIi5UwFVSXnsNs80/5qVtVStSIiUgpJ38LCCZB25ibxDTpAbnphQSUiUoGpoKrkCoqpsT2aWZxERET8TvoBWPI0bJlrtqvUNBecaH8X2HWrSxGpHFRQVWLHMnM9231babUlEREpgZxTMP0682+bHRLuM5dCD61pdTIREa9SQVWJJbywzLPdukE1C5OIiIjfCalmLoG+bz3c8CrUb2d1IhERS6igEgCCA7XykoiIXMKpfSQk/R0ON4ZG8ea+Ps+CI0jT+0SkUtNPQBEREbk4Vy588woBMzrT8ORaHMueLjwWGKJiSkQqPY1QVVKH03M8291a1LYwiYiI+KydS2DR43B8DzbgaFgs1fq+oN/GioicRQVVJXXdtOWe7X/e18nCJCIi4nOOJ8HiJ2HHArMdVg9Xn8l8t7cKg+tebW02EREfo4JKsNlsVkcQERFfsnOxWUzZA+C6B6D7BAxHFUhZYHUyERGfo4JKRESksjMMyDoOVWuZ7WtGwdEd0Ol+iIwz9zmd1uUTEfFhmgZdCRmG4dl+8/b2FiYRERHLHdsNH/8WZvU1F6AAcATAja8VFlMiInJRGqGqhI6dzvNsd22uBSlERCqlvNPw7V/g+zchPw/sgZD6IzTpbnUyERG/ooKqEprw2S+e7eqhgRYmERERrzMM+PV/sOhJSN9n7mvWBwa9DLWbW5tNRMQPqaCqhFbuOOLZ1oIUIiKVSN5p+PcdsGel2a4WDQNfhLgbQP8/EBEpFRVUIiIilUVgqLlynyMYuj0CXR+BoFCrU4mI+DUVVJVMRk7hKk2jrm9iYRIRESl3hgFb5kLTnlC1tjkKdcNfwMiHmk2tTiciUiFolb9KZs3uY57t/+vX0sIkIiJSrg5vgw9uhLkjYfmUwv01GquYEhEpQxqhqmSqBhd+5KFB+vhFRCqcnFOw6iX48W1zJCogBKpHm6NVuk5KRKTM6Rt1JbNoyyGrI4iISHkwDPjl37D0GTidZu6LuxEGTDNHpUREpFx4dcrfrl276NKlCy1btqRTp05s27btgufNmjWLFi1a0KxZM8aMGYPL5fIc+/LLL4mLi6N58+YMGzaMzMxMz7ETJ05wxx130KJFC6666iqeeOKJcn9P/mb2D3utjiAiIuXh+zdh/lizmKrVHO6cC7d9rGJKRKScebWguv/++xkzZgw7d+5kwoQJjBw58rxzkpKSmDRpEqtXryYxMZFDhw4xa9YsADIzMxk5ciTz588nMTGR+vXrM3XqVM9j77vvPtq3b8+uXbv49ddf+eMf/+i19yYiImKp9neay6D3nQwPfA/N+1qdSESkUvBaQZWWlsaGDRu48847ARg2bBhJSUkkJycXOW/OnDkMHTqUunXrYrPZGDt2LJ988gkACxcuJCEhgbi4OADGjRvnOZaYmMiGDRsYP36857nq16/vhXcmIiLiZW43bPgI5ow0p/oBhNaEhzdAt/+DgGBr84mIVCJeu4YqNTWVBg0aEBBgvqTNZiM6OpqUlBRiYmI856WkpNC4ceH0hJiYGFJSUi56bP/+/bjdbrZt20ZUVBRjx45l/fr11K5dmz//+c+0b9/+gnlyc3PJzc31tNPT0wFwOp04nc4LPsabCjKUZZa1Scc9208MbOkT71PKRnn0F6nY1Gf8l+3Az9gXP479wAYAXK2GYrQYUHiCu3w+U/UZKSn1GSkpX+ozJcng1UUpbOesLmQU/FbtEuede865z1HA6XSyZs0ann/+ed555x0WL17MTTfdRHJysqeIO9uLL77IlClTztu/ZMkSQkN95yaHS5cuLbPnemRN4b9DvVPbWLDgwtewif8qy/4ilYP6jP8IdGXQ6sAcGh9bhQ0Dpz2EHfWHsmdnLsauBV7LoT4jJaU+IyXlC30mKyur2Od6raCKiopi3759uFwuAgICMAyD1NRUoqOji5wXHR1dZBrg3r17PedER0ezYsUKz7Hk5GQaNmyI3W6ncePGNGzYkF69egEwYMAA8vLy2LdvX5ERsAITJ04sMj0wPT2dqKgo+vfvT0RERBm+89JxOp0sXbqUfv36ERgYWCbP+ciaJZ7twYMHl8lzim8oj/4iFZv6jB9x52P/+Z/Yv56GLfuEuav1b6H3s8SG1yPWSzHUZ6Sk1GekpHypzxTMXisOrxVUkZGRtG/fntmzZzNixAjmzp1LTEzMecXOsGHD6NatG8888wyRkZHMmDGD2267DYCBAwfy4IMPsn37duLi4pg+fbrnWMeOHYmIiGDTpk20bduW9evXA9CwYcML5gkODiY4+Pw55oGBgZZ/gGcrqzxnj/RF1aziU+9Ryo6v9V/xfeozfsBlwPp3IfsE1G0Ng1/B3riLd1eVOov6jJSU+oyUlC/0mZK8vlen/L399tuMGDGCadOmERERwYcffgjAqFGjGDJkCEOGDKFp06ZMmTKFrl274na76d27t2c1wPDwcGbOnMnNN9+My+WiTZs2nuew2Wx88MEHjBo1ipycHEJCQpg7d67lH4avOHY6z7O94I/dLUwiIiKXlXkEQqpBQJD554a/QNqvkDASHLqFpIiIL/HqT+XY2FjWrFlz3v6ZM2cWaY8ePZrRo0df8DkKCq8LSUhIYO3atVcetAKaNH+LZzssWP8zFhHxSfkuWD8LVkyF7o9C14fN/U26m39ERMTnWDVjQLxs/4lsqyOIiMil7P0e3ukBCydA7inYubhwSXQREfFZGqqoJOpVC2Hz/lMEOi68SqKIiFgk4xAsfQY2/cdsh1SHPs9AxxFwkZVtRUTEd6igqiSWbjsMgDNfv+0UEfEZv34B8x6AvAzABh3vgd7PQNVaVicTEZFiUkElIiJildotwZUNDTvC4FfMv0VExK+ooKpk/v77DlZHEBGpvE7thz2roP0dZrtOLNy3BBq0B7suaxYR8UcqqCqB9BynZ/v6FrUtTCIiUkm58uCHf8DXr5gjUvXaQP225rFGGpUSEfFnKqgqgQ17T3i2I6rovlwiIl6VuNxcue9YotmOuhYc+lksIlJRlHh+waZNm4q1T3zH/zYesDqCiEjlczIV/nMnzL7FLKaqRsLNM+C+xRB5ldXpRESkjJS4oBoxYkSx9onvWJd83OoIIiKVS74TZvU3V/GzOeC6cfCH9RB/u5ZCFxGpYIo95e/o0aOkpaWRk5PDr7/+inHmZoOnTp3i9OnT5RZQrtw+3dRXRMS7HIFw/XjYOs9cva/u1VYnEhGRclLsgurjjz/m9ddf58CBAwwePNizv1q1akyYMKFcwknZ6h0XaXUEEZGK6XgSLH4SOtwNsYPMfQkj4ZpRGpESEangil1QZWVlkZSUxPPPP8+kSZPKM5OUoVxnvmf73q5NLEwiIlIBObNh9Wuw+nXIz4Wju6DFAHMJdC2DLiJSKRT7p/1nn30GwLx588otjJS9uEmLPNsdGle3LoiISEViGLD9K/hHJ/j6z2Yx1aQH3PaxCikRkUqm2CNUVatW5aabbiI5OZlbb731vOOffvppmQaTshcapFXyRUSu2LHd5jLoicvMdkQjGDAVWv1G0/tERCqhYn/D/t///seSJUvYtGkTN9xwQ3lmEhER8V3HdpvFlCMIuvwBrv8TBFW1OpWIiFik2AVVjRo1+N3vfkft2rXp06dPeWYSERHxHYYBJ5Kh5pnrUFv2h15PQ+tboFYzS6OJiIj1il1Qfffdd3Tt2pXc3FwWLFhw3vGzV/4T3+DMd3u279OCFCIiJXdkByx4DA5sNO8jFXZmtdQej1kaS0REfEexC6oPPviArl278sorr5x3zGazqaDyQccy8zzbfxrQ0sIkIiJ+JjfDXGzih7fA7QJHMOxbB3Ga8i4iIkUVu6B69913AVi5cmW5hZGy9em6VM+2FqQQESkGw4DNc2DJ05B5yNwXOxgGTCuc8iciInKWYn/L3rZt2yWPt2rV6orDSNl6bdlOqyOIiPgPdz58NBSSvjbbNZrAoJfNa6ZEREQuotgF1aVW9rPZbOzZs6dMAomIiFjC7oDIqyB1LXT/E3T+AwSGWJ1KRER8XLELqqSkpPLMIeUotl641RFERHyP2w2b/g0N2puFFEDPidD5QagebW02ERHxG6W6sGbdunUsX74cm81G37596dixY1nnkiuU7zY82w/1bm5hEhERH3TwF3P1vtQfIeZ6uOcL86a8Vaqbf0RERIrJXtIHvPbaa/z2t7/l8OHDHDp0iOHDh/PGG2+URza5AsdP53q2e8dFWphERMSHZJ+Ar/4E7/Q0i6nAqtC8Lxjuyz5URETkQko8QvXWW2/x008/UatWLQAmTZrEddddxx//+McyDyell5ZeWFBphT8RqfTcbtg4G5ZNhqxj5r6rb4H+L0C1hpZGExER/1bib9r169f3FFMANWvWpF69emUaSq7coq2HrI4gIuI7Nn8K//uDuV0nDga/Ak26W5tJREQqhBIXVF27dmXUqFGMHDkSgPfff58BAwZ4llXX8um+YfvBDKsjiIhYyzDM66IAWg+DdbPg6puh0xhwBFoaTUREKo4SF1SffPIJAMuXLy+yf+bMmVo+3Ycs+/Ww1RFERKzhzoef3odf/g0jvoKAYLOAGrmksMASEREpIyUuqLR8un/RkukiUqmkrjUXnTi0yWxv/BgS7jO3VUyJiEg5KHZBVTClr4DNZiMyMrLI9VTiezJzXFZHEBEpf5lHYNmzZgEFEFwNej8N7e+2NpeIiFR4xS6obrjhhvP2HT16lNjYWP7zn//QrFmzMg0mZeO5m6+2OoKISPlxu2Hdu7BiKuSeMve1vxP6TIawOpZGExGRyqHYBdXFpvr985//5OGHH+arr74qs1ByZXJd+Z7tFpGa8iciFZjNBjsXm8VU/XYw+C8QdY3VqUREpBIp8Y19z3X33Xdz8ODBssgiZWTl9jTPdmREsIVJRETKQcYhyDpubttsMOhluOGvMHqliikREfG6Ky6oAPLz8y9/knhNenbhdVPBAQ4Lk4iIlKF8J3z/d3gzAZY/V7i/dnO4ZiTY9fNORES8r9hT/rKyss7bd/ToUd566y3atWtXpqHkyuw8rHtQiUgFs+drWPAYHN1htg9vBVceBARZm0tERCq9YhdUYWFh2Gw2DMMAzFX+6tSpw4ABA3j99dfLK5+IiFRmp/bBkqdh6zyzHVoL+k6B+DvAXiaTLERERK5IsQsqt9tdnjmkDK3ckXb5k0REfF3iMvjPXeDMApsdrhkFvZ6EKjWsTiYiIuJR4hv7iu87eDIHgMFt6lucRETkCjToAAHBUK8tDH4F6re1OpGIiMh5VFBVQNlOc5GQBZu1+qKI+JETe2Hzp3D9o+bqfaE1YdRyqNnUbIuIiPggFVQiImItZw589was/iu4cqBOHFx1k3mslm4aLyIivk0FVQXjdhue7ScGxVmYRESkGHYsgkWPw4lksx1zPdRqbmkkERGRklBBVcHMWp3k2b792mgLk4iIXMLxPbDwCdi12GyH14f+L0DrYZreJyIifkUFVQUzbcGvnu2IkEALk4iIXIRhmKv3Hd4C9kDoPA66T4DgMKuTiYiIlJgKKhERKX+GYf6x280RqH5T4Pu/w6CXoU5Lq9OJiIiUmgoqEREpX0d3wcIJ0LyfORoF0LwvNOuj6X0iIuL3dJv5CqZNw2oATB3a2uIkIlLp5WbC0mdhemfYvQK+/Yu5ol8BFVMiIlIBaISqgtm8/xQAu9NOW5xERCotw4Ct82DJ05C+39zXoj8MfAkCQ6zNJiIiUsZUUFVQRzJzrY4gIpXRsd3w5SOQ9I3Zrt4YBv0ZWg7UiJSIiFRIKqgqEMMovAfVvV1jrAsiIpWXKxeSv4OAEOj2f9D1jxBYxepUIiIi5UYFVQVyKtvp2W5ZN9zCJCJSaRgGHNwIDdqb7bqtYMibENMVasRYmUxERMQrtChFBbJp3ynPdtVg1coiUs4ObYH3B8O7feDw1sL97e9QMSUiIpWGvnVXIPe8t9bqCCJSGWSfhFUvwtp3wciHgCqQ9ivUvdrqZCIiIl6ngkpERIrH7YZfPoFlz8LpI+a+Vr+B/lOhepS12URERCyigkpERC7PMOBft0LiUrNdu6W5el+z3tbmEhERsZgKKhERuTybDZr3hZQ10ONxuHYsBARZnUpERMRyKqgqoG4talsdQUT8ndsNP/8TqkcXjkJdM8qc4hdR39psIiIiPkQFVQXUsLru+SIiV2DfT7DgUTiwAWo2hXE/QEAwOAJUTImIiJxDBVUFcfZNfds2qmZhEhHxW6ePwvIpsOEjwIDgCLhmNNgcVicTERHxWSqoKoj0bJdn+/oWdSxMIiJ+x50P69+DFc9Dzpn72bX7PfSdDOF1LY0mIiLi61RQVRA7Dmd4tqNqhlqYRET8TtI35hQ/gHptYPCrEH2dtZlERET8hAqqCuKZ+VusjiAi/sRdOKpN057Q5laI6gQJ94FdU/xERESKSwVVBREeoo9SRIoh3wXr3iVgzXSCoiaY+2w2GPautblERET8lL6FVxDr956wOoKI+Lrk1bDgMUjbhg2ICV0B3GZ1KhEREb+mgqqCiasXbnUEEfE16QdhydOwZY7ZrlITV6+n2XmgJs2tTSYiIuL3VFBVMNsPZVz+JBGpPL7/O6x6EfIyARsk3Au9J2EEhsPBBVanExER8XsqqCqYa5vUtDqCiPiSE0lmMdUwAW54FRq0N/c7ndbmEhERqSBUUFUAuc58z/abv29vYRIRsdypfZDvhJpNzHavp6BhR2h7G9jt1mYTERGpgPR/1wrgZLb5m2aH3UbtsGCL04iIJVy58M2r8Pdr4MtHwDDM/aE1If73KqZERETKiUaoKoC09FwA7Daw2WwWpxERr9u1DBZOgOO7zbYrF3LTIaSatblEREQqAa/+ynLXrl106dKFli1b0qlTJ7Zt23bB82bNmkWLFi1o1qwZY8aMweUqvAHll19+SVxcHM2bN2fYsGFkZmae9/j77rsPm812wWMV0frk4wA48w2Lk4iIV53YC5/8Hj4eZhZTYXVh6Dtw70IVUyIiIl7i1YLq/vvvZ8yYMezcuZMJEyYwcuTI885JSkpi0qRJrF69msTERA4dOsSsWbMAyMzMZOTIkcyfP5/ExETq16/P1KlTizz+iy++qHSjNEczc62OICLelroW/tEJdnwFNgd0fggeWg/tfmfeqFdERES8wmsFVVpaGhs2bODOO+8EYNiwYSQlJZGcnFzkvDlz5jB06FDq1q2LzWZj7NixfPLJJwAsXLiQhIQE4uLiABg3bpznGMCxY8eYMmUKf/3rX73zpnzE5v2nrI4gIt5WPx6qR0PM9fDAdzBgKoREWJ1KRESk0vHaNVSpqak0aNCAgADzJW02G9HR0aSkpBATE+M5LyUlhcaNG3vaMTExpKSkXPTY/v37cbvd2O12HnzwQSZPnky1apef6pKbm0tubuHITnp6OgBOpxOnDywnXJChOFm+3XUUgAC7zSeyi/eVpL+InzqRhP3Ht3D3ewEcQYAN7vwcQmubI1Il/OzVZ6Sk1GekpNRnpKR8qc+UJINXF6U4dyqeYVz4mp+zzzv3nItN5/vss88ICgrixhtvLFaWF198kSlTppy3f8mSJYSGhhbrObxh6dKlxTjL/Bh71s9nwQLdqLMyK15/EX/icOfS4tAXNE9bgMNw8evBLHbXHVxmz68+IyWlPiMlpT4jJeULfSYrK6vY53qtoIqKimLfvn24XC4CAgIwDIPU1FSio6OLnBcdHV1kGuDevXs950RHR7NixQrPseTkZBo2bIjdbmflypWsWLGiyGjX1VdfzZdffkmbNm3OyzNx4kTGjx/vaaenpxMVFUX//v2JiLB+2ozT6WTp0qX069ePwMDAS577yJolAJwIqMngwZ28EU98TEn6i/gJw8C24yscS6dgS98HgLtpL2L7/5HYWi2u+OnVZ6Sk1GekpNRnpKR8qc8UzF4rDq8VVJGRkbRv357Zs2czYsQI5s6dS0xMTJECCMxrq7p168YzzzxDZGQkM2bM4LbbbgNg4MCBPPjgg2zfvp24uDimT5/uOTZ9+nSmT5/ueR6bzcbWrVsJCwu7YJ7g4GCCg8+/Z1NgYKDlH+DZLpfHle/2bD85uJVPZRfv87X+K6V0dJe5DPruM79AqhYFA6Zhv+om7GW84IT6jJSU+oyUlPqMlJQv9JmSvL5Xp/y9/fbbjBgxgmnTphEREcGHH34IwKhRoxgyZAhDhgyhadOmTJkyha5du+J2u+ndu7dnNcDw8HBmzpzJzTffjMvlok2bNp7nqKyOZuZ5tq+qb/3ImoiUgUUTzWLKEQRd/wjdxkOQ70xFFhERkUJeLahiY2NZs2bNeftnzpxZpD169GhGjx59wecoKLwu52LXZ1U0BfegAqgS5LAwiYiUmmFAfh4EnBk1HzDVLKb6Pw+1mlmbTURERC7Jq/ehkrL37rd7rI4gIlci7Vf48CZY/FThvjqxcPu/VEyJiIj4ARVUfm7TPt2DSsQv5aTDoifhra6Q/C1s/BecPmZ1KhERESkhr075k7LXsHoV9p/MtjqGiBSXYcCmT2HpJMg8bO6LuxEGTIOqtazNJiIiIiWmgsrPJcTUYP/GbCYOjrM6iohczvEkmP8ApJy5lrRmMxj0MrToa20uERERKTUVVH5uweaDANQOO38JeBHxMcHhkLYNAkOh+2PQ+cHChShERETEL6mg8nPOfHM1w1PZTouTiMh53G5z+fPmfcBmg6q1Yfj75qIT1RpZnU5ERETKgBal8GNn39T3qnq6B5WITznwM7zXHz4eBjsXFe5v3kfFlIiISAWiESo/dux04U19OzWpaWESEfHIOg4rnof17wMGBIWZ+0RERKRCUkHlxxLTMj3bdrvNwiQigjsfNvwTlj8H2WcKqDa/hX7PQ0R9a7OJiIhIuVFB5cdWbk+zOoKIFJg7Crb+19yOvBoGvwIxXa3NJCIiIuVOBZUfm7U6yeoIIlKg3e2QuAx6PQnXjAaHfryKiIhUBvo/vohISeW7YP175pLnHe8x97XsD49sgio1rM0mIiIiXqWCSkSkJFJ+gK8ehcObITgCYgdDWB3zmIopERGRSkcFlR+7ukEEWw+kM3Voa6ujiFR8GYdh6TOw6d9mO6Qa9J4EoVphU0REpDJTQeXHth5IB6BOeIjFSUQqsHwnrH0XVr0IuemADTrcBX2eNW/UKyIiIpWaCio/dTKr8B5U2XkuC5OIVHDHEmHJ02DkQ4P2MPgv0Kij1alERETER6ig8lMffp/s2b6pXQPrgohURHmnIaiquR15FXR/zLyXVPu7wW63NpuIiIj4FH0z8FO7zrqpr82mm/qKlAlXHnz3Bvz1Kkj7tXB/r4nQcYSKKRERETmPvh34qa82HbQ6gkjFsnslzOhqLjyRcwp++tDqRCIiIuIHNOVPRCq3k6mw+En49X9mu2od6DvFvFGviIiIyGWooBKRymvNdFj+HLiyweaATmOg5xNQpbrVyURERMRPqKASkUrMMIup6C4w+BWop3u6iYiISMmooPJzY3s0szqCiP84kQxZx6DhmWXPO42B6tEQdyNocRcREREpBS1K4eeua1bL6ggivs+ZDStfhH9cC3NHgyvX3O8IhKtuUjElIiIipaYRKj/kzHd7tts2rGZhEhEfZxiwYyEsegJO7jX3VWsI2SchvK6l0URERKRiUEHlh05lOz3b4SH6CEUu6NhuWPg4JC412xENYcBUaHWzRqRERESkzOjbuB/6z7pUz3aAQ7M2Rc5zZKd5T6n8PLAHQpeH4PpHITjM6mQiIiJSwaig8kOvLt5hdQQR31a7BcRcDxgw6GWzLSIiIlIONLzhhxpWr2J1BBHfcmQnfHo3ZB032zYb3PpPuPO/KqZERESkXGmEyg+dyMoDwK7LQKSyy82Ar1+GH6aD2wWhteHGv5rHNL1PREREvEAFlR/KyssH4NZroixOImIRw4Atc2HJ05Bx0NzXcpB5rZSIiIiIF6mg8mMrt6dZHUHE+w5vg4UTIPlbs10jxrxOquUAS2OJiIhI5aSCys+4zroH1cTBV1mYRMQi6941i6mAELj+T9DlYQgMsTqViIiIVFIqqPxM8rHTnu0b2tS3MImIlxgG5KZDyJmbWPeeBM5s6DkRajS2NpuIiIhUeiqo/My/fkzxbOseVFLhHdoMXz0KQaHmin02G4TWhKEzrE4mIiIiAqig8jvN6mjlMqkEsk/AymmwbiYYbggMheN7oFYzq5OJiIiIFKGCys8s3XbY6ggi5cftho0fw7LJkHXU3Hf1UOj/AlRrZGk0ERERkQtRQeVnTue5rI4gUj5O7Tdvzrt/vdmuHQuDX4amPS2NJSIiInIpKqj8zOH0HABiaoVanESkjFWtAzknISgMej4B144FR6DVqUREREQuSQWVn0k9ng3AqWynxUlErpA7HzbPgda3mIVTQBAMfw/C6kJ4PavTiYiIiBSLCio/dSJLBZX4sdR1sOBROLjRvFaq84Pm/vrtLI0lIiIiUlIqqPzU//VraXUEkZLLPALLJ8PPs812cAQEVbU0koiIiMiVUEHlp6JqVrE6gkjx5btg/Xuw8gXIOWXui78D+k6GsEhLo4mIiIhcCRVUfuqq+hFWRxApvoWPmQUVmNP6Br8KUZ2szSQiIiJSBlRQ+RHDMDzb1asEWZhEpIQ6jYFfv4CeE6HjCLA7rE4kIiIiUiZUUPmRjNzCe1CFheijEx+V74S170DWMejzjLkv8ip4ZAsEhlibTURERKSM6Vu5H0k6ctqzXTVIv+EXH5T0LSx4DI78CjY7tP0d1Ik1j6mYEhERkQpIBZUfWbP7mGfbZrNZmETkHOkHYMnTsGWu2a5S01xwolYLS2OJiIiIlDcVVH7Ebrc6gcg5XHnww3T4+mVwnjZHpRLug15PQWhNq9OJiIiIlDsVVH7kdG4+ANWqBFqcROSMnFPw7V/NYirqWhj8im7OKyIiIpWKCio/Mn1lIgCnsp0WJ5FKLfMIhNUxt8PqwMAXwWaDtrdpGFVEREQqHX378SPXt6gNQN+r6lqcRColZw58/Qq83gZ2Li7c3/4OiP+9iikRERGplDRC5UdOnhmZSoipYXESqXR2LoGFE+BEktneOh9aDrA0koiIiIgvUEHlR35OOQnAwVM51gaRyuN4EiyaCDsXmu2wetD/BWgz3NpcIiIiIj5CBZUfqheh+/mIF6x9FxY/Bfm5YA+A6x6AHo9DcLjVyURERER8hgoqP9RRU/7EG6o1MoupJj3M1fsKbtArIiIiIh4qqPxQaKDD6ghSER3bDccSC6+Nih0EI76Cxl3NVfxERERE5DwqqPxEek7hUumNaoRamEQqnLzT8O1f4Ps3IbAK/GEDVDVXlCSmm7XZRERERHycCio/kX7WvaciquhjkzJgGLDtc/M6qfR95r6Y68GlRU9EREREikvfzP1E8tEsz7ZN06/kSh3ZYS6DvmeV2a4eDQNfgtjBmt4nIiIiUgIqqPxEtjPf6ghSUWQegbe7myNRjmDo9n/Q7RFzup+IiIiIlIgKKj9x/HQuAN1a1LY4ifi9sDrQ/i44tQ8Gvgg1m1idSERERMRvqaDyE0/M3QzA6l1HLU4ifufwNlg8EQb+GSLjzH0DXwKH/vMXERERuVJ2qwOISDnJOQULn4AZ3cxrpZY+U3hMxZSIiIhImdC3Kj/TOy7S6gji69xu2PRvs4A6fcTcd9VNMGCatblEREREKiAVVH7Gme+2OoL4skObYckTkPqj2a7VHAa9DM37WJtLREREpIJSQeVn+lxV1+oI4sPsyV+bxVRgVejxGFz3IAQEWR1LREREpMJSQeVnOjauYXUE8SVuN2Qegip1zOY19+PIOgrXjYNqDS0OJyIiIlLxqaDyM9WrBFodQXzF/g2w4FHISYfRX5v7HIEwYKq1uUREREQqERVUfiDp6GnPdoQKKjl9DJZPgQ3/BAwICsd2eKvVqUREREQqJa8um75r1y66dOlCy5Yt6dSpE9u2bbvgebNmzaJFixY0a9aMMWPG4HK5PMe+/PJL4uLiaN68OcOGDSMzMxOAAwcOMGDAAGJjY2nbti233norx48f98r7Km/zNuzzbKugqsTc+bBuJrzZATZ8CBjQ9nfwh/UYDdpbnU5ERESkUvJqQXX//fczZswYdu7cyYQJExg5cuR55yQlJTFp0iRWr15NYmIihw4dYtasWQBkZmYycuRI5s+fT2JiIvXr12fqVHN6k8PhYNKkSezYsYNNmzbRuHFjnnjiCW++vXKzZvcxqyOI1bKOw7u94Ks/Qc5JqNsa7l0It7wD4fWsTiciIiJSaXltyl9aWhobNmxgyZIlAAwbNoyHHnqI5ORkYmJiPOfNmTOHoUOHUreuuZrd2LFjefnll7n//vtZuHAhCQkJxMXFATBu3DgGDx7Miy++SN26dT2PAbj22muZMWPGRfPk5uaSm5vraaenpwPgdDpxOp1l9r5LqyCD0+nkqvrhrN97osh+qWQCwnAEV8MWHIG7x5O4O44AewCc1U/O/lvkctRnpKTUZ6Sk1GekpHypz5Qkg9cKqtTUVBo0aEBAgPmSNpuN6OhoUlJSihRUKSkpNG7c2NOOiYkhJSXlosf279+P2+3Gbi8cbMvPz+cf//gHN99880XzvPjii0yZMuW8/UuWLCE0NLS0b7PMLV26lI077BQMJi5YsMDaQOIVNiOfxkdXsr/GdTgDwgCoUvVm8lvcSt6RCFi05IKPW7p0qTdjSgWgPiMlpT4jJaU+IyXlC30mKyur2Od6dVEKm81WpG0YxmXPO/ecc5/jXIZhMG7cOKpXr84f/vCHi543ceJExo8f72mnp6cTFRVF//79iYiIuORreIPT6WTp0qX069ePZzd+C5jXkQ0ePNjaYFLubClrcCx+AlvaVlrXDcA98OXLPubs/hIYqOvs5PLUZ6Sk1GekpNRnpKR8qc8UzF4rDq8VVFFRUezbtw+Xy0VAQACGYZCamkp0dHSR86Kjo0lOTva09+7d6zknOjqaFStWeI4lJyfTsGHDIqNTDz/8MKmpqcyfP7/I/nMFBwcTHBx83v7AwEDLP8CzBQYGcirbLKZi64X7VDYpYxmHYMkk2Pyp2Q6pjqN+Wxwl+Mx9rf+K71OfkZJSn5GSUp+RkvKFPlOS1/faohSRkZG0b9+e2bNnAzB37lxiYmKKTPcD89qqefPmcfjwYQzDYMaMGdx2220ADBw4kHXr1rF9+3YApk+f7jkGZjGVmJjIvHnzCAoK8s4b86KmdapaHUHKQ74Tvn8T3ux4ppiyQccR8IcNkHCv1elERERE5BK8OuXv7bffZsSIEUybNo2IiAg+/PBDAEaNGsWQIUMYMmQITZs2ZcqUKXTt2hW3203v3r09qwGGh4czc+ZMbr75ZlwuF23atPE8x3fffcebb75JXFwc1157LQBNmjRh3rx53nyL5aJ1wwi27E/n5viGVkeR8vD1n+GbV8zthgkw+BVo2MHaTCIiIiJSLF4tqGJjY1mzZs15+2fOnFmkPXr0aEaPHn3B5ygovM7VtWvXi16T5e+y8vIB3YOqQjEMKLge8NqxsHUedH0E4u+AS0xVFRERERHf4tWCSkonM8e8hioiRB+X33Plwpp/wKFNMPx9s6iqWhseXKdCSkRERMQP6Ru6H8g4U1CFBWuEyq8lLoOFj8OxRLOdMBKaXG9uq5gSERER8UsqqHxcnstNttOc8ldNU/7804m9sPhJ2P6l2a4aCf2fh5hu1uYSERERkSumgsrHHc3M9WyHa8qff3HlwndvwLd/AVcO2Bzm9VI9H4eQalanExEREZEyoG/oPu5QemFBZbdf+qbG4mMMAzb+yyymYq43V++LvMrqVCIiIiJShlRQ+bixH/9sdQQpiRPJENEIHAEQGAI3vQ6nj0LrYYWr+omIiIhIhaEr4X3cySyn1RGkOPKyYMVU+HsnWHfWbQCa9oQ2w1VMiYiIiFRQKqh8XKv64VZHkEsxDPj1C/jHtfDNy5CfCynfW51KRERERLxEU/58XFiwPiKfdTQRFj4Gu1eY7YhGMHAaXHX+jadFREREpGLSt3UfFx9VnbXJJ7ivaxOro8jZNnwEX/4fuJ3gCIIuD8P14yGoqtXJRERERMSLVFD5uKw886a+ocEOi5NIEQ3ag+GG5v1g0J+hVjOrE4mIiIiIBVRQ+bisPPOmvqFBKqgslbYdUn+AjiPMdr3WMHa1uQy6FpwQERERqbRUUPm4jBxzhCo8JNDiJJVUTjp8/Wf4cYa5AEXUtYX3kqrbytpsIiIiImI5FVQ+LiO3oKDSR+VVhgGbP4MlkyDzkLkvdrCukRIRERGRIvQt3cedPlNQVQ3SR+U1h7fCgsdg73dmu2ZTGPhnaNnf2lwiIiIi4nP0Ld3HZWqEyrvysuD9wZBzEgKqQPdHofNDEBhidTIRERER8UH6lu7jko5mARCqEaryYxiFC0sEhUKPCZD6I/SfCtWjrM0mIiIiIj7NbnUAKZ4qQfqoysWBjTCrPyQuK9x33Ti49Z8qpkRERETksvQt3Ye53IXbtcOCrQtSEWUdhy/Hwzs9Yd9aWP6cOVIFWgZdRERERIpN88h8WE5+4XZYsD6qMuF2w8//hGVTIPu4ua/1cOj/vAopERERESkxfUv3YblnCqqQQDsBDg0mXrEDP5ujUgc2mO06V8HgV6DJ9dbmEhERERG/pYLKhxUUVDlO96VPlOI5tc8spoIjoOdE6DQaHLphsoiIiIiUngoqH3YoW1PQrog7H47ugsg4sx13I/R7Htr+DsLrWptNRERERCoEzSPzYcEOqxP4sZQf4Z0e8P5AcwEKMK+R6vqwiikRERERKTMqqHxY/pmZfh0b17A2iD/JTIN5D8B7/eHQZjDccHiL1alEREREpILSlD8flnemoAoJ1FDVZeW7YN27sHIa5Kab+9rfBX0nQ9XalkYTERERkYpLBZUPKyioQoNUUF2SKxfe7QOHN5vtBu1h8F+gUUdrc4mIiIhIhaeCyocVrPJXRSNUlxYQDI0SIH0f9HkWOtwNdv2biYiIiEj50zVUPizbZa7yFxaiureIfCd89zdzBb8CfZ+FP2yAhHtVTImIiIiI16ig8mFL9psfz79+TLE4iQ/Zswre6gpLJ8HCCWAY5v4qNSC0pqXRRERERKTy0dCHH6hfLcTqCNY7tQ8WPwXb5pvt0NrQepilkUREREREVFD5sA613Gw4ZmfU9U2tjmIdVy6s+Tt88yo4s8Bmh2tGQ68noUp1q9OJiIiISCWngsqHadl0YMM/Yflz5nZ0Zxj8CtRrY20mEREREZEzVFD5sC0nzGuocp35FifxMrcb7Gcu7+twD2ydb67c1/ZWsNksjSYiIiIicjYtSuEHZv+41+oI3uHMgVV/hpm9zZX8AAKC4N6voN3vVEyJiIiIiM/RCJUfiAgJtDpC+duxCBY9DieSzfa2z6HNcEsjiYiIiIhcjgoqP9CkdlWrI5Sf43tg4ROwa7HZDm8AA16Aq2+xNpeIiIiISDGooPIDwzo2sjpC2ct3wtd/hu/egPw8sAdC5weh+2MQHGZ1OhERERGRYlFB5aOc+W7PdnTNUAuTlBN7AKT8YBZTTXuZq/fVbmF1KhERERGRElFB5aMyclye7cjwYAuTlKGju6BqHfP+UTYbDH4Vju6Eq27SghMiIiIi4pe0yp+POn46D4DgADvB/n4fqtxMWPosTO8MK6cV7o+Mg1ZDVEyJiIiIiN/SCJWPynGaU/5yXe7LnOnDDAO2/hcWPw0ZB8x9p1KL3mdKRERERMSPqaDyUT8kHbM6wpVJ+xUWPAbJ35rt6o1h0J8hdpC1uUREREREypAKKh/VqHoVqyOU3pb/wn9Hg9sFASHQbTx0fRgC/fg9iYiIiIhcgAoqH5VvmH9f26SGtUFKI+Z6CKwKTa6HAdOgRmOrE4mIiIiIlAsVVD4qx5kPQBV/WJDi0BbYNh96P222w+rAgz9ARANLY4mIiIiIlDcVVD5q/d4TAKzaedTiJJeQfdJctW/du2C4odE10HKAeUzFlIiIiIhUAiqofFTK8WyrI1yc2w2//MtcCj3rTMHX6maoe7WlsUREREREvE0FlY+KCDE/mmtifOwaqgM/m6v37Vtntmu3hEEvQ7Ne1uYSEREREbGACioftXz7EQDWJZ+wOMlZ3Pkw5z44vgeCwqDH43DtWAgIsjqZiIiIiIgldHdVH9Unrg4AY7s3sTaIO9/8A2B3mKv2tR4OD60zl0JXMSUiIiIilZgKKh/XoHqIdS++bz3M7APr3yvcFzsIhs/SohMiIiIiImjKn886nWeOClUNsuAjOn0Ulk2Gnz860z4GHe8Fh7qLiIiIiMjZ9A3ZR2WfKaiqBHnxPlTufHM0asXzkHPK3Nfu99BvioopEREREZEL0LdkH1VQUIV6q6A6+At8/iAc2my267WBwX+B6Gu98/oiIiIiIn5IBZWPynJ6uaDCBoe3Qkg16D0JEu4zF6EQEREREZGLUkHlo3LOFFQhgeVU1OS7YN9aaNzFbNdvC7e8C017QtXa5fOaIiIiIiIVjFb581FHM/MAqFIeBVXyanj7evjwJkjbXri/zXAVUyIiIiIiJaARKh9kGIZn233W9hVLPwhLnoYtc8x2lZpwMgUi48ruNUREREREKhEVVD4oz+X2bNcJC77yJ3TlwY9vwdcvQ14mYDOvker9NITWvPLnFxERERGppFRQ+aCCe1BBGSybbhjw/kDY/5PZbnQNDH4VGsRf2fOKiIiIiIgKKl904nSeZ9tht13Zk9ls0OpmOLEX+j0H7W4Huy6dExEREREpCyqofFC2M//yJ12MKxe+fxMaJZgr9gFc9wB0uBuqVC+LeCIiIiIicoYKKh+U7zYXoqgRVMIFKXYthYUT4PgeqNUCHvgeAoLAEahiSkRERESkHKig8kHZZ66hKvblUyeSYdGTsOMrsx1WF3pMMAspEREREREpNyqofFDBlL+gy13q5MyG796A1a+BKwfsAXDtWOjxOIRElH9QEREREZFKTgWVD8o6M0IVeLmCavdKWPWiud2kOwx6RfeUEhERERHxIhVUPijXVVBQXeAaKmcOBIaY27GDIP4OaN4Xrh5qrugnIiIiIiJeo4LKB+U4zRv7FhmhysuCb/8CG/8FD3xn3pDXZoObp1sTUkRERERE8OoNiXbt2kWXLl1o2bIlnTp1Ytu2bRc8b9asWbRo0YJmzZoxZswYXC6X59iXX35JXFwczZs3Z9iwYWRmZnqO/fjjj8THx9OyZUv69OnDwYMHy/09lYdc51lT/gwDtn0O/+gE374KGQdg82fWBhQREREREcDLBdX999/PmDFj2LlzJxMmTGDkyJHnnZOUlMSkSZNYvXo1iYmJHDp0iFmzZgGQmZnJyJEjmT9/PomJidSvX5+pU6cCYBgGd9xxB6+//jo7d+5k0KBBjB8/3ptvr8zkuswRqmgO4Pjkt/Dp3XAqFapFw+9mQ6cxFicUERERERHwYkGVlpbGhg0buPPOOwEYNmwYSUlJJCcnFzlvzpw5DB06lLp162Kz2Rg7diyffPIJAAsXLiQhIYG4OHPhhXHjxnmOrV+/nuDgYHr27AmYxdv8+fNxOp3eeYNlKCfPxWMB/+bV0xOxJ60CRzB0nwAP/ghX3aRrpUREREREfITXrqFKTU2lQYMGBASYL2mz2YiOjiYlJYWYmBjPeSkpKTRu3NjTjomJISUl5aLH9u/fj9vtPu9YeHg44eHhHDx4kOjo6PPy5Obmkpub62mnp6cD4HQ6LS/CsvJc1CODAPLJb9oX98AXoUYTzgS0NJv4poI+a3XfFf+hPiMlpT4jJaU+IyXlS32mJBm8uiiF7ZyRFcO4wCp255x37jnnPkdpnh/gxRdfZMqUKeftX7JkCaGhoRd9nDekH7YxL/RWcsPiqV6tPaz5FfjV0kziH5YuXWp1BPEz6jNSUuozUlLqM1JSvtBnsrKyin2u1wqqqKgo9u3bh8vlIiAgAMMwSE1NPW/0KDo6usg0wL1793rOiY6OZsWKFZ5jycnJNGzYELvdft7jMjIyyMjIoH79+hfMM3HixCLXWKWnpxMVFUX//v2JiLD2priDMavipUuX0q9fPwIDAy3NI75P/UVKSn1GSkp9RkpKfUZKypf6TMHsteLwWkEVGRlJ+/btmT17NiNGjGDu3LnExMQUme4H5rVV3bp145lnniEyMpIZM2Zw2223ATBw4EAefPBBtm/fTlxcHNOnT/cc69ixIzk5OaxatYqePXvy9ttvc/PNN1/0wwgODiY4OPi8/YGBgZZ/gGfztTzi29RfpKTUZ6Sk1GekpNRnpKR8oc+U5PW9OuXv7bffZsSIEUybNo2IiAg+/PBDAEaNGsWQIUMYMmQITZs2ZcqUKXTt2hW3203v3r09qwGGh4czc+ZMbr75ZlwuF23atPE8h91uZ/bs2YwdO5bs7GwaNmzI7Nmzvfn2RERERESkkvFqQRUbG8uaNWvO2z9z5swi7dGjRzN69OgLPkdB4XUhnTt35pdffrnyoCIiIiIiIsXg1ftQiYiIiIiIVCQqqEREREREREpJBZWIiIiIiEgpqaASEREREREpJRVUIiIiIiIipaSCSkREREREpJRUUImIiIiIiJSSCioREREREZFSUkElIiIiIiJSSiqoRERERERESkkFlYiIiIiISCmpoBIRERERESklFVQiIiIiIiKlpIJKRERERESklFRQiYiIiIiIlFKA1QF8hWEYAKSnp1ucxOR0OsnKyiI9PZ3AwECr44iPU3+RklKfkZJSn5GSUp+RkvKlPlNQExTUCJeiguqMjIwMAKKioixOIiIiIiIiviAjI4Nq1apd8hybUZyyqxJwu90cOHCA8PBwbDab1XFIT08nKiqK1NRUIiIirI4jPk79RUpKfUZKSn1GSkp9RkrKl/qMYRhkZGTQoEED7PZLXyWlEaoz7HY7jRo1sjrGeSIiIizvUOI/1F+kpNRnpKTUZ6Sk1GekpHylz1xuZKqAFqUQEREREREpJRVUIiIiIiIipaSCykcFBwfz7LPPEhwcbHUU8QPqL1JS6jNSUuozUlLqM1JS/tpntCiFiIiIiIhIKWmESkREREREpJRUUImIiIiIiJSSCioREREREZFSUkFloV27dtGlSxdatmxJp06d2LZt2wXPmzVrFi1atKBZs2aMGTMGl8vl5aTiK4rTZ1asWMG1115Lq1ataN26NU899RS6VLLyKu7PGYCcnBxatWpFQkKCFxOKrylun9m8eTM9e/bkqquuIjY2lv/+979eTiq+ojh9xjAMHnvsMa6++mratm1Lr169SExMtCCtWO3hhx8mJiYGm83Gli1bLnqeX33/NcQyvXr1Mt5//33DMAzjs88+M6677rrzztmzZ49Rv35949ChQ4bb7TZuuukmY8aMGV5OKr6iOH1mw4YNxu7duw3DMIzs7Gyja9euxscff+zNmOJDitNnCowfP9647777jI4dO3opnfii4vSZ06dPG02bNjW+/fZbwzAMw+l0Gmlpad6MKT6kOH1m/vz5RqdOnYy8vDzDMAzj+eefN3772996M6b4iK+//tpITU01GjdubGzevPmC5/jb91+NUFkkLS2NDRs2cOeddwIwbNgwkpKSSE5OLnLenDlzGDp0KHXr1sVmszF27Fg++eQTCxKL1YrbZ9q3b0/Tpk0BCAkJIT4+nj179ng7rviA4vYZgG+//ZZdu3Zx1113eTml+JLi9pl//etfdO7cmW7dugEQEBBAnTp1vB1XfEBJfs7k5uaSk5ODYRikp6fTqFEjL6cVX9C9e/fLfvb+9v1XBZVFUlNTadCgAQEBAQDYbDaio6NJSUkpcl5KSgqNGzf2tGNiYs47RyqH4vaZsx06dIg5c+YwePBgb8UUH1LcPnP69GkeeeQR3nrrLStiig8pbp/Ztm0bISEh3HjjjcTHx3P33Xdz5MgRKyKLxYrbZ2666SZ69epFvXr1qF+/PsuXL+e5556zIrL4AX/7/quCykI2m61I27jIdS5nn3exc6RyKG6fAUhPT+emm25iwoQJdOjQobyjiY8qTp957LHHePDBB2nYsKG3YokPK06fcTqdLF68mLfffpuff/6ZqKgoHnzwQW9FFB9TnD6zYcMGtm/fzv79+zlw4AB9+vThoYce8lZE8UP+9P1XBZVFoqKi2Ldvn+cCO8MwSE1NJTo6ush50dHRRYbN9+7de945UjkUt88AZGRkMHDgQIYMGcL48eO9HVV8RHH7zOrVq3nuueeIiYnhtttuY/PmzVx99dVWRBaLFbfPNG7cmF69etGwYUNsNht33HEHa9eutSKyWKy4feaDDz6gV69eVK9eHbvdzj333MPKlSutiCx+wN++/6qgskhkZCTt27dn9uzZAMydO5eYmBhiYmKKnDds2DDmzZvH4cOHMQyDGTNmcNttt1mQWKxW3D6TmZnJwIEDGTBgAJMmTbIgqfiK4vaZTZs2kZycTHJyMv/+979p06YNW7dutSCxWK24febWW29l3bp1pKenA7Bo0SLatWvn7bjiA4rbZ5o2bcry5ctxOp0AfPHFF7Ru3drbccVP+N33XwsWwpAztm/fblx33XVGixYtjI4dOxpbtmwxDMMwRo4caXz++eee89555x2jWbNmRpMmTYyRI0d6VsiRyqc4feaFF14wAgICjHbt2nn+vPDCC1bGFgsV9+dMgZUrV2qVv0quuH3mww8/NFq1amW0bdvWGDRokJGammpVZLFYcfpMTk6OMWrUKCM2NtZo06aN0b9/fyMpKcnC1GKVcePGGQ0bNjQcDodRt25do1mzZoZh+Pf3X5th+PikRBERERERER+lKX8iIiIiIiKlpIJKRERERESklFRQiYiIiIiIlJIKKhERERERkVJSQSUiIiIiIlJKKqhERERERERKSQWViIiUWkxMDHFxccTHx9OqVSv+8Y9/WJrn9ddfJy0tzdOeMWMGr732Wpk9f3x8vOe9BgQEeNq/+93vyuw1LuTkyZO8/PLLpX58TEwMW7ZsKcNEIiJSQPehEhGRUouJieHLL7+kdevWpKam0qZNG7755hvatm1brMe7XC4CAgKuOIfb7QagadOmnjzlKTk5mYSEBI4ePVqix+Xn5+NwOLz2egXO/pxERKRsaYRKRETKRFRUFC1btmTnzp0cPnyYoUOH0qZNG1q3bs0777zjOS8mJoapU6fSq1cv7rnnHlatWkW7du2499576dixIwkJCfzyyy+e819++WWuvvpq2rRpwx133MGpU6cAmDx5MnfddRe33HIL8fHxPP/88xw4cIDhw4cTHx/Pxo0bmTx5Mo8++ihgFjOPPvoorVu3pnXr1vzhD38gLy8PgBEjRjBu3Dj69u1Ly5YtueWWWzzHLsflcjFgwAASEhK4+uqrueOOO8jKygLggw8+YODAgdx9990kJCSwdu1a5s6dS1xcHO3bt+eFF17AZrORmZkJwLp16+jduzcJCQl06NCBuXPnAjB27FhOnjxJfHw8CQkJABw6dIhbb72VTp060bZtW5555hlPpm+//ZY2bdrQqVMnHnroIfS7UxGRcmSIiIiUUuPGjY3NmzcbhmEYmzZtMsLDw42dO3cat956q/HEE08YhmEYhw8fNho1amT8+OOPnseMGTPGcLvdhmEYxsqVKw3AWLlypWEYhvGf//zHaNWqlWEYhrFgwQIjLi7OOHHihGEYhjF69Ghj3LhxhmEYxrPPPms0bNjQOHz48AXzFJzzpz/9yTAMw5g+fbrRs2dPIycnx3A6ncagQYOMl19+2TAMw7jnnnuMzp07G1lZWYbL5TK6dOli/Otf/7ro+05KSjJq1aplGIZhuN1u4+jRo57tsWPHGq+88ophGIbx/vvvG1WrVjV27tzp+beoWbOmp/3aa68ZgJGRkWGcOHHCaN++vXHgwAHDMAzjyJEjRnR0tHHw4MEir1egf//+xtdff20YhmE4nU5jwIABxn//+18jJyfHaNCgQZF/T6DIv4uIiJSdK59nISIildrw4cMJCQkhNDSU9957jxYtWrBs2TLPKFNkZCS33HILy5cvp9P/t3M/r7B/cRzHn3M1YRiJIo1YoKRm0iyEjCaNhbCxkFJqEmGhiJV/wE7NQhNNxsJQFJKdhCSFUjbkRxYoWUjjRzGZ7+Lb/cS93Ktx77V5PVZzPud83nM+Z/fqfM6npAQAr9eLyWQyauTn5+N2uwFobGykvb2di4sLlpaWaG5uJjU1FYDOzk6ampqM++rq6sjIyPjQPJeWlmhtbSU+Ph6AtrY2/H4//f39ADQ0NJCYmAhASUkJx8fHH6objUYZGhpicXGRSCTCzc0NlZWVRn9FRQUFBQUAbG5u4nQ6jbbX66WnpweAjY0NTk5OqKmpeVX74OCA3NzcV/95d3fH8vIyl5eXxrXb21v29/fJy8vDYrH8tJ4iIvJ3KFCJiMinzMzMvHk252Vg+rGdnJz827omk4loNPrpOt/9rlZCQoLxOy4ujkgk8qG6oVCI1dVV1tbWsFqt+Hw+1tbW3pzjW3N42edwOF7d+93p6emr9vPzMyaTia2tLcxm86u+l69LiojI36czVCIi8sd5PB7j3NTV1RWzs7NUVVW9O/7o6MgIEjMzM9hsNrKysqiurmZqaopwOAzAyMgIHo/n3TopKSnGGasfVVdXEwwGeXx8JBKJEAgEflnro66vr0lPT8dqtRIOhwkGg++OLS0tZWdnh6OjIwDGx8eNvvLycg4PD1leXjau7e7u8vj4SEpKCvf390bIs1qtuFwuBgcHjbEXFxecnZ1RWFjIw8PDq/V8b01EROTztEMlIiJ/nM/no6OjA4fDwfPzMwMDA8brfm8pLi5mamqK3t5eotEooVAIgJqaGvb29igrK8NkMuFwOBgeHn63Tnd3N16vF4vF8lOwaW9v5/j4GKfTCYDb7aa7u/vTz9rS0sL8/DxFRUXYbDZcLhfn5+dvjs3MzMTv91NbW0t6ejr19fWYzWYsFgvfvn1jYWGB/v5+enp6eHp6Iicnh7m5OdLS0mhubsZut5OUlMT29jYTExP09vZit9uB/3fC/H4/2dnZTE5O0tXVRWJiIm63m5ycnE8/p4iIvE2fTRcRkS+1srJCX18f29vbXz2VfyIcDmO1WgEYGxsjEAiwvr7+xbMSEZFYaYdKRETkH/L5fExPTxOJREhLS2N0dPSrpyQiIp+gHSoREREREZEY6aMUIiIiIiIiMVKgEhERERERiZEClYiIiIiISIwUqERERERERGKkQCUiIiIiIhIjBSoREREREZEYKVCJiIiIiIjESIFKREREREQkRgpUIiIiIiIiMfoPuiv+fu28A9gAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def uplift_plot(uplift_df):\n",
    "    \"\"\"\n",
    "    Plot the uplift curve\n",
    "    \"\"\"\n",
    "    gain_x = uplift_df.percent_rank\n",
    "    gain_y = uplift_df.group_uplift\n",
    "    # Plot the data\n",
    "    fig = plt.figure(figsize=(10, 6))\n",
    "    mpl.rcParams[\"font.size\"] = 8\n",
    "\n",
    "    ax = plt.plot(gain_x, gain_y, color=\"#2077B4\", label=\"Normalized Uplift Model\")\n",
    "\n",
    "    plt.plot(\n",
    "        [0, gain_x.max()],\n",
    "        [0, gain_y.max()],\n",
    "        \"--\",\n",
    "        color=\"tab:orange\",\n",
    "        label=\"Random Treatment\",\n",
    "    )\n",
    "    plt.legend()\n",
    "    plt.xlabel(\"Porportion Targeted\")\n",
    "    plt.ylabel(\"Uplift\")\n",
    "    plt.grid()\n",
    "\n",
    "    return fig, ax\n",
    "\n",
    "\n",
    "test_ranked_pd_df = test_ranked_df.select([\"pred_uplift\", \"percent_rank\", \"group_uplift\"]).toPandas()\n",
    "fig, ax = uplift_plot(test_ranked_pd_df)\n",
    "\n",
    "mlflow.log_figure(fig, \"UpliftCurve.png\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a37345c8-bc29-4ccd-a6d6-5d71d26cdc70",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "This analysis and the uplift curve both show that the top 20% population, as ranked by the prediction, would have a large gain if they received the treatment. This means that the top 20% of the population represents the **persuadables** group. Therefore, you can then set the cutoff score for the desired size of treatment group at 20%, to identify the target selection customers for the greatest impact."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "044c8efc-6d02-42c1-914e-407118c34939",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:23:55.4667139Z",
       "execution_start_time": "2023-09-08T23:23:52.884446Z",
       "livy_statement_state": "available",
       "parent_msg_id": "041f52a7-fc5f-44b1-a305-0cf2b48db8bd",
       "queued_time": "2023-09-08T23:19:41.8712797Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 0,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 22
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 22, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uplift scores that exceed 0.0039 map to Persuadables.\n"
     ]
    }
   ],
   "source": [
    "cutoff_percentage = 0.2\n",
    "cutoff_score = test_ranked_pd_df.iloc[int(len(test_ranked_pd_df) * cutoff_percentage)][\n",
    "    \"pred_uplift\"\n",
    "]\n",
    "\n",
    "print(\"Uplift scores that exceed {:.4f} map to Persuadables.\".format(cutoff_score))\n",
    "mlflow.log_metrics(\n",
    "    {\"cutoff_score\": cutoff_score, \"cutoff_percentage\": cutoff_percentage}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adcc9678-b7fe-4cd8-b559-76b5606eb1a1",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## Step 4: Register the final ML Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a363089c-5691-426d-a9dd-e30b87c0abc5",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "You'll use MLflow to track and log all experiments for both treatment and control groups. This tracking and logging includes the corresponding parameters, metrics, and the models. Note that this information is logged under the experiment name, in the workspace, for later use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71a0f059-0f53-42e7-a80f-5db480644fb1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:24:04.1742773Z",
       "execution_start_time": "2023-09-08T23:23:55.7318985Z",
       "livy_statement_state": "available",
       "parent_msg_id": "6dc62ca4-0a3d-474b-a616-396e6371d679",
       "queued_time": "2023-09-08T23:19:41.8719946Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 0,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 23
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 23, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Successfully registered model 'aisample-upliftmodelling-treatmentmodel'.\n",
      "2023/09/08 23:24:00 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: aisample-upliftmodelling-treatmentmodel, version 7\n",
      "Created version '7' of model 'aisample-upliftmodelling-treatmentmodel'.\n",
      "Successfully registered model 'aisample-upliftmodelling-controlmodel'.\n",
      "2023/09/08 23:24:03 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: aisample-upliftmodelling-controlmodel, version 7\n",
      "Created version '7' of model 'aisample-upliftmodelling-controlmodel'.\n"
     ]
    }
   ],
   "source": [
    "# Register the model\n",
    "treatment_model_uri = \"runs:/{}/model\".format(treatment_run_id)\n",
    "mlflow.register_model(treatment_model_uri, f\"{EXPERIMENT_NAME}-treatmentmodel\")\n",
    "\n",
    "control_model_uri = \"runs:/{}/model\".format(control_run_id)\n",
    "mlflow.register_model(control_model_uri, f\"{EXPERIMENT_NAME}-controlmodel\")\n",
    "\n",
    "mlflow.end_run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1f73622-39fb-417c-a1f6-e243aec57ae4",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "To view your experiments:\n",
    "1. On the left panel, select your workspace.\n",
    "1. Find and select the experiment name, in this case _aisample-upliftmodelling_."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd88fb36-8929-4976-8c3f-93d61d8a47e7",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "\n",
    "<img style=\"float: left;\" src=\"https://synapseaisolutionsa.blob.core.windows.net/public/Uplift-experiment.png\"  width=\"75%\" height=\"30%\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a987e7f-db47-4ef6-9bec-60b50c448eba",
   "metadata": {},
   "source": [
    "## Step 5: Save the prediction results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29d3cc36-e280-45ba-9166-4cc9384e4092",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Microsoft Fabric offers PREDICT - a scalable function that supports batch scoring in any compute engine. It enables customers to operationalize machine learning models. Users can create batch predictions straight from a notebook or the item page for a specific model. Visit [this](https://aka.ms/fabric-predict) resource to learn more about PREDICT, and to learn how to use PREDICT in Microsoft Fabric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ae39e12-f39e-4cfb-bfab-4b25e3f7b53f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:24:12.9208265Z",
       "execution_start_time": "2023-09-08T23:24:04.4464071Z",
       "livy_statement_state": "available",
       "parent_msg_id": "7f6654c1-79b6-47b4-9f28-c8626a60cb47",
       "queued_time": "2023-09-08T23:19:41.8731428Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [
         {
          "completionTime": "2023-09-08T23:24:11.291GMT",
          "dataRead": 56187832,
          "dataWritten": 0,
          "description": "Job group for statement 24:\n# Load the model back\nloaded_treatmentmodel = mlflow.spark.load_model(treatment_model_uri, dfs_tmpdir=\"Files/spark\")\nloaded_controlmodel = mlflow.spark.load_model(control_model_uri, dfs_tmpdir=\"Files/spark\")\n\n# Make predictions\nbatch_predictions_treatment = loaded_treatmentmodel.transform(test_df)\nbatch_predictions_control = loaded_controlmodel.transform(test_df)\nbatch_predictions_treatment.show(5)\n",
          "displayName": "showString at NativeMethodAccessorImpl.java:0",
          "jobGroup": "24",
          "jobId": 52,
          "killedTasksSummary": {},
          "name": "showString at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 58,
          "stageIds": [
           70
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:09.847GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:09.279GMT",
          "dataRead": 817,
          "dataWritten": 0,
          "description": "Job group for statement 24:\n# Load the model back\nloaded_treatmentmodel = mlflow.spark.load_model(treatment_model_uri, dfs_tmpdir=\"Files/spark\")\nloaded_controlmodel = mlflow.spark.load_model(control_model_uri, dfs_tmpdir=\"Files/spark\")\n\n# Make predictions\nbatch_predictions_treatment = loaded_treatmentmodel.transform(test_df)\nbatch_predictions_control = loaded_controlmodel.transform(test_df)\nbatch_predictions_treatment.show(5)\n",
          "displayName": "first at ReadWrite.scala:587",
          "jobGroup": "24",
          "jobId": 51,
          "killedTasksSummary": {},
          "name": "first at ReadWrite.scala:587",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 1,
          "stageIds": [
           69
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:09.242GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:09.186GMT",
          "dataRead": 817,
          "dataWritten": 0,
          "description": "Job group for statement 24:\n# Load the model back\nloaded_treatmentmodel = mlflow.spark.load_model(treatment_model_uri, dfs_tmpdir=\"Files/spark\")\nloaded_controlmodel = mlflow.spark.load_model(control_model_uri, dfs_tmpdir=\"Files/spark\")\n\n# Make predictions\nbatch_predictions_treatment = loaded_treatmentmodel.transform(test_df)\nbatch_predictions_control = loaded_controlmodel.transform(test_df)\nbatch_predictions_treatment.show(5)\n",
          "displayName": "first at ReadWrite.scala:587",
          "jobGroup": "24",
          "jobId": 50,
          "killedTasksSummary": {},
          "name": "first at ReadWrite.scala:587",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 1,
          "stageIds": [
           68
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:09.148GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:09.094GMT",
          "dataRead": 226,
          "dataWritten": 0,
          "description": "Job group for statement 24:\n# Load the model back\nloaded_treatmentmodel = mlflow.spark.load_model(treatment_model_uri, dfs_tmpdir=\"Files/spark\")\nloaded_controlmodel = mlflow.spark.load_model(control_model_uri, dfs_tmpdir=\"Files/spark\")\n\n# Make predictions\nbatch_predictions_treatment = loaded_treatmentmodel.transform(test_df)\nbatch_predictions_control = loaded_controlmodel.transform(test_df)\nbatch_predictions_treatment.show(5)\n",
          "displayName": "first at ReadWrite.scala:587",
          "jobGroup": "24",
          "jobId": 49,
          "killedTasksSummary": {},
          "name": "first at ReadWrite.scala:587",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 1,
          "stageIds": [
           67
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:09.058GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:08.743GMT",
          "dataRead": 226,
          "dataWritten": 0,
          "description": "Job group for statement 24:\n# Load the model back\nloaded_treatmentmodel = mlflow.spark.load_model(treatment_model_uri, dfs_tmpdir=\"Files/spark\")\nloaded_controlmodel = mlflow.spark.load_model(control_model_uri, dfs_tmpdir=\"Files/spark\")\n\n# Make predictions\nbatch_predictions_treatment = loaded_treatmentmodel.transform(test_df)\nbatch_predictions_control = loaded_controlmodel.transform(test_df)\nbatch_predictions_treatment.show(5)\n",
          "displayName": "runJob at PythonRDD.scala:172",
          "jobGroup": "24",
          "jobId": 48,
          "killedTasksSummary": {},
          "name": "runJob at PythonRDD.scala:172",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 1,
          "stageIds": [
           66
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:08.642GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:07.285GMT",
          "dataRead": 817,
          "dataWritten": 0,
          "description": "Job group for statement 24:\n# Load the model back\nloaded_treatmentmodel = mlflow.spark.load_model(treatment_model_uri, dfs_tmpdir=\"Files/spark\")\nloaded_controlmodel = mlflow.spark.load_model(control_model_uri, dfs_tmpdir=\"Files/spark\")\n\n# Make predictions\nbatch_predictions_treatment = loaded_treatmentmodel.transform(test_df)\nbatch_predictions_control = loaded_controlmodel.transform(test_df)\nbatch_predictions_treatment.show(5)\n",
          "displayName": "first at ReadWrite.scala:587",
          "jobGroup": "24",
          "jobId": 47,
          "killedTasksSummary": {},
          "name": "first at ReadWrite.scala:587",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 1,
          "stageIds": [
           65
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:07.247GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:07.192GMT",
          "dataRead": 817,
          "dataWritten": 0,
          "description": "Job group for statement 24:\n# Load the model back\nloaded_treatmentmodel = mlflow.spark.load_model(treatment_model_uri, dfs_tmpdir=\"Files/spark\")\nloaded_controlmodel = mlflow.spark.load_model(control_model_uri, dfs_tmpdir=\"Files/spark\")\n\n# Make predictions\nbatch_predictions_treatment = loaded_treatmentmodel.transform(test_df)\nbatch_predictions_control = loaded_controlmodel.transform(test_df)\nbatch_predictions_treatment.show(5)\n",
          "displayName": "first at ReadWrite.scala:587",
          "jobGroup": "24",
          "jobId": 46,
          "killedTasksSummary": {},
          "name": "first at ReadWrite.scala:587",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 1,
          "stageIds": [
           64
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:07.149GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:07.038GMT",
          "dataRead": 226,
          "dataWritten": 0,
          "description": "Job group for statement 24:\n# Load the model back\nloaded_treatmentmodel = mlflow.spark.load_model(treatment_model_uri, dfs_tmpdir=\"Files/spark\")\nloaded_controlmodel = mlflow.spark.load_model(control_model_uri, dfs_tmpdir=\"Files/spark\")\n\n# Make predictions\nbatch_predictions_treatment = loaded_treatmentmodel.transform(test_df)\nbatch_predictions_control = loaded_controlmodel.transform(test_df)\nbatch_predictions_treatment.show(5)\n",
          "displayName": "first at ReadWrite.scala:587",
          "jobGroup": "24",
          "jobId": 45,
          "killedTasksSummary": {},
          "name": "first at ReadWrite.scala:587",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 1,
          "stageIds": [
           63
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:07.003GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:06.914GMT",
          "dataRead": 226,
          "dataWritten": 0,
          "description": "Job group for statement 24:\n# Load the model back\nloaded_treatmentmodel = mlflow.spark.load_model(treatment_model_uri, dfs_tmpdir=\"Files/spark\")\nloaded_controlmodel = mlflow.spark.load_model(control_model_uri, dfs_tmpdir=\"Files/spark\")\n\n# Make predictions\nbatch_predictions_treatment = loaded_treatmentmodel.transform(test_df)\nbatch_predictions_control = loaded_controlmodel.transform(test_df)\nbatch_predictions_treatment.show(5)\n",
          "displayName": "runJob at PythonRDD.scala:172",
          "jobGroup": "24",
          "jobId": 44,
          "killedTasksSummary": {},
          "name": "runJob at PythonRDD.scala:172",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 1,
          "rowCount": 1,
          "stageIds": [
           62
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:06.290GMT",
          "usageDescription": ""
         }
        ],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 9,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 24
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 24, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023/09/08 23:24:04 INFO mlflow.spark: 'runs:/0f031ba3-ab05-4f08-8607-78cbd5e05767/model' resolved as 'sds://lake.trident.com/bf0c4a18-5045-40be-a890-7e904b6be261/6a40b6fe-f756-4db8-ba35-11494ad4abab/0f031ba3-ab05-4f08-8607-78cbd5e05767/artifacts/model'\n",
      "2023/09/08 23:24:05 INFO mlflow.spark: File 'sds://lake.trident.com/bf0c4a18-5045-40be-a890-7e904b6be261/6a40b6fe-f756-4db8-ba35-11494ad4abab/0f031ba3-ab05-4f08-8607-78cbd5e05767/artifacts/model/sparkml' not found on DFS. Will attempt to upload the file.\n",
      "2023/09/08 23:24:05 INFO mlflow.spark: Copied SparkML model to Files/spark/3930a08b-e99d-43c0-aa9e-a413d9c4c0c7\n",
      "2023/09/08 23:24:07 INFO mlflow.spark: 'runs:/7f6bd071-b8f3-4d38-bf98-c4bac3199e2e/model' resolved as 'sds://lake.trident.com/bf0c4a18-5045-40be-a890-7e904b6be261/6a40b6fe-f756-4db8-ba35-11494ad4abab/7f6bd071-b8f3-4d38-bf98-c4bac3199e2e/artifacts/model'\n",
      "2023/09/08 23:24:08 INFO mlflow.spark: File 'sds://lake.trident.com/bf0c4a18-5045-40be-a890-7e904b6be261/6a40b6fe-f756-4db8-ba35-11494ad4abab/7f6bd071-b8f3-4d38-bf98-c4bac3199e2e/artifacts/model/sparkml' not found on DFS. Will attempt to upload the file.\n",
      "2023/09/08 23:24:08 INFO mlflow.spark: Copied SparkML model to Files/spark/b0a62136-6f62-4a64-a4c2-f04e3d3ef56d\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+------------------+-----------------+-----------------+------------------+------------------+-------------------+------------------+-----------------+------------------+-----------------+--------------------+---------+----------+-----+--------+--------------------+--------------------+--------------------+----------+\n",
      "|                f0|                f1|               f2|               f3|                f4|                f5|                 f6|                f7|               f8|                f9|              f10|                 f11|treatment|conversion|visit|exposure|            features|       rawPrediction|         probability|prediction|\n",
      "+------------------+------------------+-----------------+-----------------+------------------+------------------+-------------------+------------------+-----------------+------------------+-----------------+--------------------+---------+----------+-----+--------+--------------------+--------------------+--------------------+----------+\n",
      "|12.616364906986497|10.059654474774547|8.214382844395335|4.679881620097284|10.280525225748212|0.3463602638335841|0.29444271125560606| 10.65526759211423|3.971857989350875|13.190055934673358|5.300374864042156|-0.16867922100056124|        1|         0|    0|       0|[-0.1686792210005...|[6.57403697245921...|[0.99860579671475...|       0.0|\n",
      "|12.616364906986497|10.059654474774547|8.214382844395335|4.679881620097284|10.280525225748212|1.1285177631409207|0.29444271125560606| 9.703094148526326|3.971857989350875|13.190055934673358|5.300374864042156|-0.16867922100056124|        1|         0|    0|       0|[-0.1686792210005...|[6.64070365171077...|[0.9986955958172,...|       0.0|\n",
      "|12.616364906986497|10.059654474774547|8.214382844395335|4.679881620097284|10.280525225748212|1.1285177631409207|0.29444271125560606|11.151218312552132|3.971857989350875|13.190055934673358|5.300374864042156|-0.16867922100056124|        1|         0|    0|       0|[-0.1686792210005...|[6.57403697245921...|[0.99860579671475...|       0.0|\n",
      "|12.616364906986497|10.059654474774547|8.214382844395335|4.679881620097284|10.280525225748212|1.6242183527412077|0.29444271125560606|10.705732371502856|3.971857989350875|13.190055934673358|5.300374864042156|-0.16867922100056124|        1|         0|    0|       0|[-0.1686792210005...|[6.57403697245921...|[0.99860579671475...|       0.0|\n",
      "|12.616364906986497|10.059654474774547|8.214382844395335|4.679881620097284|10.280525225748212|2.2309068425557226|0.29444271125560606|  9.32342987847462|3.971857989350875|13.190055934673358|5.300374864042156|-0.16867922100056124|        1|         0|    0|       0|[-0.1686792210005...|[6.64070365171077...|[0.9986955958172,...|       0.0|\n",
      "+------------------+------------------+-----------------+-----------------+------------------+------------------+-------------------+------------------+-----------------+------------------+-----------------+--------------------+---------+----------+-----+--------+--------------------+--------------------+--------------------+----------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Load the model back\n",
    "loaded_treatmentmodel = mlflow.spark.load_model(treatment_model_uri, dfs_tmpdir=\"Files/spark\")\n",
    "loaded_controlmodel = mlflow.spark.load_model(control_model_uri, dfs_tmpdir=\"Files/spark\")\n",
    "\n",
    "# Make predictions\n",
    "batch_predictions_treatment = loaded_treatmentmodel.transform(test_df)\n",
    "batch_predictions_control = loaded_controlmodel.transform(test_df)\n",
    "batch_predictions_treatment.show(5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92cfe364-aa31-44e8-ad74-28036cc83e0b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:25:24.3753058Z",
       "execution_start_time": "2023-09-08T23:24:13.1872222Z",
       "livy_statement_state": "available",
       "parent_msg_id": "aefe4c6d-5d85-44f2-9287-a57785dbef19",
       "queued_time": "2023-09-08T23:19:44.9011583Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [
         {
          "completionTime": "2023-09-08T23:25:22.920GMT",
          "dataRead": 4896,
          "dataWritten": 0,
          "description": "Delta: Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n: Compute snapshot for version: 6",
          "displayName": "toString at String.java:2994",
          "jobGroup": "25",
          "jobId": 70,
          "killedTasksSummary": {},
          "name": "toString at String.java:2994",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 2,
          "numSkippedTasks": 57,
          "numTasks": 58,
          "rowCount": 50,
          "stageIds": [
           102,
           103,
           104
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:25:22.880GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:25:22.854GMT",
          "dataRead": 16283,
          "dataWritten": 4896,
          "description": "Delta: Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n: Compute snapshot for version: 6",
          "displayName": "toString at String.java:2994",
          "jobGroup": "25",
          "jobId": 69,
          "killedTasksSummary": {},
          "name": "toString at String.java:2994",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 50,
          "numCompletedStages": 1,
          "numCompletedTasks": 50,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 7,
          "numTasks": 57,
          "rowCount": 72,
          "stageIds": [
           100,
           101
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:25:22.375GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:25:22.038GMT",
          "dataRead": 17281,
          "dataWritten": 16283,
          "description": "Delta: Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n: Compute snapshot for version: 6",
          "displayName": "toString at String.java:2994",
          "jobGroup": "25",
          "jobId": 68,
          "killedTasksSummary": {},
          "name": "toString at String.java:2994",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 7,
          "numCompletedStages": 1,
          "numCompletedTasks": 7,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 7,
          "rowCount": 44,
          "stageIds": [
           99
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:25:21.925GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:25:21.472GMT",
          "dataRead": 3775,
          "dataWritten": 0,
          "description": "Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n",
          "displayName": "$anonfun$recordDeltaOperationInternal$1 at SynapseLoggingShim.scala:95",
          "jobGroup": "25",
          "jobId": 67,
          "killedTasksSummary": {},
          "name": "$anonfun$recordDeltaOperationInternal$1 at SynapseLoggingShim.scala:95",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 50,
          "numCompletedStages": 1,
          "numCompletedTasks": 50,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 6,
          "numTasks": 56,
          "rowCount": 8,
          "stageIds": [
           97,
           98
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:25:21.350GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:25:21.244GMT",
          "dataRead": 158653838,
          "dataWritten": 111522222,
          "description": "Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n",
          "displayName": "save at NativeMethodAccessorImpl.java:0",
          "jobGroup": "25",
          "jobId": 66,
          "killedTasksSummary": {},
          "name": "save at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 5593254,
          "stageIds": [
           96,
           95
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:25:11.451GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:25:11.366GMT",
          "dataRead": 1361805488,
          "dataWritten": 158653838,
          "description": "Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n",
          "displayName": "save at NativeMethodAccessorImpl.java:0",
          "jobGroup": "25",
          "jobId": 65,
          "killedTasksSummary": {},
          "name": "save at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 2798031,
          "stageIds": [
           94
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:50.706GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:50.565GMT",
          "dataRead": 4891,
          "dataWritten": 0,
          "description": "Delta: Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n: Compute snapshot for version: 5",
          "displayName": "toString at String.java:2994",
          "jobGroup": "25",
          "jobId": 64,
          "killedTasksSummary": {},
          "name": "toString at String.java:2994",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 2,
          "numSkippedTasks": 56,
          "numTasks": 57,
          "rowCount": 50,
          "stageIds": [
           93,
           91,
           92
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:50.528GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:50.508GMT",
          "dataRead": 14015,
          "dataWritten": 4891,
          "description": "Delta: Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n: Compute snapshot for version: 5",
          "displayName": "toString at String.java:2994",
          "jobGroup": "25",
          "jobId": 63,
          "killedTasksSummary": {},
          "name": "toString at String.java:2994",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 50,
          "numCompletedStages": 1,
          "numCompletedTasks": 50,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 6,
          "numTasks": 56,
          "rowCount": 69,
          "stageIds": [
           89,
           90
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:49.971GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:49.672GMT",
          "dataRead": 15379,
          "dataWritten": 14015,
          "description": "Delta: Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n: Compute snapshot for version: 5",
          "displayName": "toString at String.java:2994",
          "jobGroup": "25",
          "jobId": 62,
          "killedTasksSummary": {},
          "name": "toString at String.java:2994",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 6,
          "numCompletedStages": 1,
          "numCompletedTasks": 6,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 6,
          "rowCount": 38,
          "stageIds": [
           88
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:49.579GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:49.189GMT",
          "dataRead": 4896,
          "dataWritten": 0,
          "description": "Delta: Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n: Compute snapshot for version: 6",
          "displayName": "toString at String.java:2994",
          "jobGroup": "25",
          "jobId": 61,
          "killedTasksSummary": {},
          "name": "toString at String.java:2994",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 2,
          "numSkippedTasks": 57,
          "numTasks": 58,
          "rowCount": 50,
          "stageIds": [
           85,
           86,
           87
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:49.155GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:49.132GMT",
          "dataRead": 16282,
          "dataWritten": 4896,
          "description": "Delta: Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n: Compute snapshot for version: 6",
          "displayName": "toString at String.java:2994",
          "jobGroup": "25",
          "jobId": 60,
          "killedTasksSummary": {},
          "name": "toString at String.java:2994",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 50,
          "numCompletedStages": 1,
          "numCompletedTasks": 50,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 7,
          "numTasks": 57,
          "rowCount": 72,
          "stageIds": [
           84,
           83
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:48.623GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:48.260GMT",
          "dataRead": 17281,
          "dataWritten": 16282,
          "description": "Delta: Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n: Compute snapshot for version: 6",
          "displayName": "toString at String.java:2994",
          "jobGroup": "25",
          "jobId": 59,
          "killedTasksSummary": {},
          "name": "toString at String.java:2994",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 7,
          "numCompletedStages": 1,
          "numCompletedTasks": 7,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 7,
          "rowCount": 44,
          "stageIds": [
           82
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:48.133GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:47.696GMT",
          "dataRead": 3775,
          "dataWritten": 0,
          "description": "Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n",
          "displayName": "$anonfun$recordDeltaOperationInternal$1 at SynapseLoggingShim.scala:95",
          "jobGroup": "25",
          "jobId": 58,
          "killedTasksSummary": {},
          "name": "$anonfun$recordDeltaOperationInternal$1 at SynapseLoggingShim.scala:95",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 50,
          "numCompletedStages": 1,
          "numCompletedTasks": 50,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 6,
          "numTasks": 56,
          "rowCount": 8,
          "stageIds": [
           81,
           80
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:47.506GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:47.368GMT",
          "dataRead": 162000111,
          "dataWritten": 113578701,
          "description": "Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n",
          "displayName": "save at NativeMethodAccessorImpl.java:0",
          "jobGroup": "25",
          "jobId": 57,
          "killedTasksSummary": {},
          "name": "save at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 25,
          "numTasks": 26,
          "rowCount": 5593254,
          "stageIds": [
           78,
           79
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:37.216GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:37.166GMT",
          "dataRead": 1361805488,
          "dataWritten": 162000111,
          "description": "Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n",
          "displayName": "save at NativeMethodAccessorImpl.java:0",
          "jobGroup": "25",
          "jobId": 56,
          "killedTasksSummary": {},
          "name": "save at NativeMethodAccessorImpl.java:0",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 25,
          "numCompletedStages": 1,
          "numCompletedTasks": 25,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 25,
          "rowCount": 2798031,
          "stageIds": [
           77
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:15.385GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:15.210GMT",
          "dataRead": 4891,
          "dataWritten": 0,
          "description": "Delta: Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n: Compute snapshot for version: 5",
          "displayName": "toString at String.java:2994",
          "jobGroup": "25",
          "jobId": 55,
          "killedTasksSummary": {},
          "name": "toString at String.java:2994",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 1,
          "numCompletedStages": 1,
          "numCompletedTasks": 1,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 2,
          "numSkippedTasks": 56,
          "numTasks": 57,
          "rowCount": 50,
          "stageIds": [
           74,
           75,
           76
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:15.117GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:15.076GMT",
          "dataRead": 14012,
          "dataWritten": 4891,
          "description": "Delta: Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n: Compute snapshot for version: 5",
          "displayName": "toString at String.java:2994",
          "jobGroup": "25",
          "jobId": 54,
          "killedTasksSummary": {},
          "name": "toString at String.java:2994",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 50,
          "numCompletedStages": 1,
          "numCompletedTasks": 50,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 1,
          "numSkippedTasks": 6,
          "numTasks": 56,
          "rowCount": 69,
          "stageIds": [
           72,
           73
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:14.256GMT",
          "usageDescription": ""
         },
         {
          "completionTime": "2023-09-08T23:24:13.863GMT",
          "dataRead": 15379,
          "dataWritten": 14012,
          "description": "Delta: Job group for statement 25:\n# Save the predictions in the lakehouse\nbatch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n)\nbatch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n)\n: Compute snapshot for version: 5",
          "displayName": "toString at String.java:2994",
          "jobGroup": "25",
          "jobId": 53,
          "killedTasksSummary": {},
          "name": "toString at String.java:2994",
          "numActiveStages": 0,
          "numActiveTasks": 0,
          "numCompletedIndices": 6,
          "numCompletedStages": 1,
          "numCompletedTasks": 6,
          "numFailedStages": 0,
          "numFailedTasks": 0,
          "numKilledTasks": 0,
          "numSkippedStages": 0,
          "numSkippedTasks": 0,
          "numTasks": 6,
          "rowCount": 38,
          "stageIds": [
           71
          ],
          "status": "SUCCEEDED",
          "submissionTime": "2023-09-08T23:24:13.484GMT",
          "usageDescription": ""
         }
        ],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 18,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 25
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 25, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Save the predictions in the lakehouse\n",
    "batch_predictions_treatment.write.format(\"delta\").mode(\"overwrite\").save(\n",
    "    f\"{DATA_FOLDER}/predictions/batch_predictions_treatment\"\n",
    ")\n",
    "batch_predictions_control.write.format(\"delta\").mode(\"overwrite\").save(\n",
    "    f\"{DATA_FOLDER}/predictions/batch_predictions_control\"\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e31c67e-c44a-407a-b365-bf07cc6363ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2023-09-08T23:25:24.9776327Z",
       "execution_start_time": "2023-09-08T23:25:24.6414455Z",
       "livy_statement_state": "available",
       "parent_msg_id": "64f1bcf9-91ba-4b41-a3c4-59ca1d9d10dc",
       "queued_time": "2023-09-08T23:19:45.7920195Z",
       "session_id": "bb877a9c-c8cf-471c-98da-25e85626530f",
       "session_start_time": null,
       "spark_jobs": {
        "jobs": [],
        "limit": 20,
        "numbers": {
         "FAILED": 0,
         "RUNNING": 0,
         "SUCCEEDED": 0,
         "UNKNOWN": 0
        },
        "rule": "ALL_DESC"
       },
       "spark_pool": null,
       "state": "finished",
       "statement_id": 26
      },
      "text/plain": [
       "StatementMeta(, bb877a9c-c8cf-471c-98da-25e85626530f, 26, Finished, Available)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Full run cost 320 seconds.\n"
     ]
    }
   ],
   "source": [
    "# Determine the entire runtime\n",
    "print(f\"Full run cost {int(time.time() - ts)} seconds.\")"
   ]
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "synapse_pyspark"
  },
  "kernelspec": {
   "display_name": "Synapse PySpark",
   "language": "Python",
   "name": "synapse_pyspark"
  },
  "language_info": {
   "name": "python"
  },
  "notebook_environment": {},
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  },
  "save_output": true,
  "spark_compute": {
   "compute_id": "/trident/default",
   "session_options": {
    "conf": {},
    "enableDebugMode": false
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}