examples/models/mlflow_server_ab_test_ambassador/mlflow_server_ab_test_ambassador.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MLFlow Pre-packaged Model Server AB Test Deployment \n",
    "In this example we will build two models with MLFlow and we will deploy them as an A/B test deployment. The reason this is powerful is because it allows you to deploy a new model next to the old one, distributing a percentage of traffic. These deployment strategies are quite simple using Seldon, and can be extended to shadow deployments, multi-armed-bandits, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tutorial Overview\n",
    "\n",
    "This tutorial will follow closely break down in the following sections:\n",
    "\n",
    "1. Train the MLFlow elastic net wine example\n",
    "\n",
    "2. Deploy your trained model leveraging our pre-packaged MLFlow model server\n",
    "\n",
    "3. Test the deployed MLFlow model by sending requests\n",
    "\n",
    "4. Deploy your second model as an A/B test\n",
    "\n",
    "5. Visualise and monitor the performance of your models using Seldon Analytics\n",
    "\n",
    "It will follow closely our talk at the [Spark + AI Summit 2019 on Seldon and MLflow](https://www.youtube.com/watch?v=D6eSfd9w9eA)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies\n",
    "\n",
    "For this example to work you must be running Seldon 0.3.2 or above - you can follow our [getting started guide for this](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/install.html).\n",
    "\n",
    "In regards to other dependencies, make sure you have installed:\n",
    "\n",
    "* Helm v3.0.0+\n",
    "* kubectl v1.14+\n",
    "* Python 3.6+\n",
    "* MLFlow 1.1.0\n",
    "* [pygmentize](https://pygments.org/docs/cmdline/)\n",
    "* [tree](http://mama.indstate.edu/users/ice/tree/)\n",
    "\n",
    "We will also take this chance to load the Python dependencies we will use through the tutorial:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from seldon_core.seldon_client import SeldonClient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get started! 🚀🔥"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Train the first MLFlow Elastic Net Wine example\n",
    "\n",
    "For our example, we will use the elastic net wine example from [MLflow's tutorial](https://github.com/mlflow/mlflow/tree/master/examples/sklearn_elasticnet_wine)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MLproject\n",
    "\n",
    "As any other MLflow project, it is defined by its `MLproject` file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize -l yaml MLproject"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that this project uses Conda for the environment and that it's defined in the `conda.yaml` file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize conda.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we can also see that the training will be performed by the `train.py` file, which receives two parameters `alpha` and `l1_ratio`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize train.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset\n",
    "\n",
    "We will use the wine quality dataset.\n",
    "Let's load it to see what's inside:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"wine-quality.csv\")\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "We've set up our MLflow project and our dataset is ready, so we are now good to start training.\n",
    "MLflow allows us to train our model with the following command:\n",
    "\n",
    "``` bash\n",
    "$ mlflow run . -P alpha=... -P l1_ratio=...\n",
    "```\n",
    "\n",
    "On each run, `mlflow` will set up the Conda environment defined by the `conda.yaml` file and will run the training commands defined in the `MLproject` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mlflow run . -P alpha=0.5 -P l1_ratio=0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of these commands will create a new run which can be visualised through the MLFlow dashboard as per the screenshot below.\n",
    "\n",
    "![mlflow-dashboard](images/mlflow-dashboard.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of these models can actually be found on the `mlruns` folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tree -L 1 mlruns/0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MLmodel\n",
    "\n",
    "Inside each of these folders, MLflow stores the parameters we used to train our model, any metric we logged during training, and a snapshot of our model.\n",
    "If we look into one of them, we can see the following structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tree mlruns/0/$(ls mlruns/0 | head -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In particular, we are interested in the `MLmodel` file stored under `artifacts/model`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize -l yaml mlruns/0/$(ls mlruns/0 | head -1)/artifacts/model/MLmodel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This file stores the details of how the model was stored.\n",
    "With this information (plus the other files in the folder), we are able to load the model back.\n",
    "Seldon's MLflow server will use this information to serve this model.\n",
    "\n",
    "Now we should upload our newly trained model into a public Google Bucket or S3 bucket.\n",
    "We have already done this to make it simpler, which you will be able to find at `gs://seldon-models/mlflow/model-a`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Deploy your model using the Pre-packaged Moldel Server for MLFlow\n",
    "\n",
    "Now we can deploy our trained MLFlow model.\n",
    "\n",
    "For this we have to create a Seldon definition of the model server definition, which we will break down further below.\n",
    "\n",
    "We will be using the model we updated to our google bucket (gs://seldon-models/mlflow/elasticnet_wine), but you can use your model if you uploaded it to a public bucket."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup Seldon Core\n",
    "\n",
    "Use the setup notebook to [Setup Cluster](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html#Setup-Cluster) with [Ambassador Ingress](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html#Ambassador) and [Install Seldon Core](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html#Install-Seldon-Core). Instructions [also online](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize mlflow-model-server-seldon-config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we write our configuration file, we are able to deploy it to our cluster by running it with our command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f mlflow-model-server-seldon-config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's created we just wait until it's deployed. \n",
    "\n",
    "It will basically download the image for the pre-packaged MLFlow model server, and initialise it with the model we specified above.\n",
    "\n",
    "You can check the status of the deployment with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl rollout status deployment.apps/mlflow-deployment-mlflow-deployment-dag-0-wines-classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's deployed, we should see a \"succcessfully rolled out\" message above. We can now test it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Test the deployed MLFlow model by sending requests\n",
    "Now that our model is deployed in Kubernetes, we are able to send any requests."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will first need the URL that is currently available through Ambassador. \n",
    "\n",
    "If you are running this locally, you should be able to reach it through localhost, in this case we can use port 80."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get svc | grep ambassador"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will select the first datapoint in our dataset to send to the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_0 = data.drop([\"quality\"], axis=1).values[:1]\n",
    "print(list(x_0[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can try sending a request first using curl:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -X POST -H 'Content-Type: application/json' \\\n",
    "    -d '{\"data\": {\"names\": [], \"ndarray\": [[7.0, 0.27, 0.36, 20.7, 0.045, 45.0, 170.0, 1.001, 3.0, 0.45, 8.8]]}}' \\\n",
    "    http://localhost:8003/seldon/seldon/mlflow-deployment/api/v0.1/predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also send the request by using our python client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "import math\n",
    "import numpy as np\n",
    "import subprocess\n",
    "\n",
    "HOST = \"localhost\" # Add the URL you found above\n",
    "port = \"80\" # Make sure you use the port above\n",
    "batch = x_0\n",
    "payload_type = \"ndarray\"\n",
    "\n",
    "sc = SeldonClient(\n",
    "    gateway=\"ambassador\", \n",
    "    namespace=\"seldon\",\n",
    "    gateway_endpoint=HOST + \":\" + port)\n",
    "\n",
    "client_prediction = sc.predict(\n",
    "    data=batch, \n",
    "    deployment_name=\"mlflow-deployment\",\n",
    "    names=[],\n",
    "    payload_type=payload_type)\n",
    "\n",
    "print(client_prediction.response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Deploy your second model as an A/B test\n",
    "\n",
    "Now that we have a model in production, it's possible to deploy a second model as an A/B test.\n",
    "Our model will also be an Elastic Net model but using a different set of parameters.\n",
    "We can easily train it by leveraging MLflow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mlflow run . -P alpha=0.75 -P l1_ratio=0.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we did before, we will now need to upload our model to a cloud bucket.\n",
    "To speed things up, we already have done so and the second model is now accessible in `gs://seldon-models/mlflow/model-b`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A/B test\n",
    "\n",
    "We will deploy our second model as an A/B test.\n",
    "In particular, we will redirect 20% of the traffic to the new model.\n",
    "\n",
    "This can be done by simply adding a `traffic` attribute on our `SeldonDeployment` spec:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize ab-test-mlflow-model-server-seldon-config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And similar to the model above, we only need to run the following to deploy it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f ab-test-mlflow-model-server-seldon-config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check that the models have been deployed and are running with the following command.\n",
    "\n",
    "We should now see the \"a-\" model and the \"b-\" models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get pods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Visualise and monitor the performance of your models using Seldon Analytics\n",
    "\n",
    "This section is optional, but by following the instructions you will be able to visualise the performance of both models as per the chart below.\n",
    "\n",
    "In order for this example to work you need to install and run the [Grafana Analytics package for Seldon Core](https://docs.seldon.io/projects/seldon-core/en/latest/analytics/analytics.html#helm-analytics-chart).\n",
    "\n",
    "For this we can access the URL with the command below, it will request an admin and password which by default are set to the following:\n",
    "* Username: admin\n",
    "* Password: password\n",
    "\n",
    "You can access the grafana dashboard through the port provided below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get svc grafana-prom -o jsonpath='{.spec.ports[0].nodePort}'  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have both models running in our Kubernetes cluster, we can analyse their performance using Seldon Core's integration with Prometheus and Grafana.\n",
    "To do so, we will iterate over the training set (which can be found in `wine-quality.csv`), making a request and sending the feedback of the prediction.\n",
    "\n",
    "Since the `/feedback` endpoint requires a `reward` signal (i.e. the higher the better), we will simulate one as:\n",
    "\n",
    "$$\n",
    "  R(x_{n})\n",
    "    = \\begin{cases}\n",
    "        \\frac{1}{(y_{n} - f(x_{n}))^{2}} &, y_{n} \\neq f(x_{n}) \\\\\n",
    "        500 &, y_{n} = f(x_{n})\n",
    "      \\end{cases}\n",
    "$$\n",
    "\n",
    ", where $R(x_{n})$ is the reward for input point $x_{n}$, $f(x_{n})$ is our trained model and $y_{n}$ is the actual value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc = SeldonClient(\n",
    "    gateway=\"ambassador\", \n",
    "    namespace=\"seldon\",\n",
    "    deployment_name='wines-classifier'\n",
    ")\n",
    "\n",
    "def _get_reward(y, y_pred):\n",
    "    if y == y_pred:\n",
    "        return 500    \n",
    "    \n",
    "    return 1 / np.square(y - y_pred)\n",
    "\n",
    "def _test_row(row):\n",
    "    input_features = row[:-1]\n",
    "    feature_names = input_features.index.to_list()\n",
    "    X = input_features.values.reshape(1, -1)\n",
    "    y = row[-1].reshape(1, -1)\n",
    "    \n",
    "    # Note that we are re-using the SeldonClient defined previously\n",
    "    r = sc.predict(\n",
    "        deployment_name=\"mlflow-deployment\",\n",
    "        data=X,\n",
    "        names=feature_names)\n",
    "    \n",
    "    y_pred = r.response['data']['tensor']['values']\n",
    "    reward = _get_reward(y, y_pred)\n",
    "    sc.feedback(\n",
    "        deployment_name=\"mlflow-deployment\",\n",
    "        prediction_request=r.request,\n",
    "        prediction_response=r.response,\n",
    "        reward=reward)\n",
    "    \n",
    "    return reward[0]\n",
    "\n",
    "data.apply(_test_row, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should now be able to see Seldon's pre-built Grafana dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![grafana-mlflow](images/grafana-mlflow.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In bottom of the dashboard you can see the following charts: \n",
    "\n",
    "- On the left: the requests per second, which shows the different traffic breakdown we specified.\n",
    "- On the center: the reward, where you can see how model `a` outperforms model `b` by a large margin.\n",
    "- On the right, the latency for each one of them.\n",
    "\n",
    "You are able to add your own custom metrics, and try out other more complex deployments by following further guides at https://docs.seldon.io/projects/seldon-core/en/latest/examples/mlflow_server_ab_test_ambassador.html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}