examples/metrics/Metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom Metrics Sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requirements\n",
    "\n",
    "- Authenticated to gcloud (```gcloud auth application-default login```)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade -r requirements.txt > /dev/null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%env GOOGLE_CLOUD_PROJECT=your-gcp-project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import merlin\n",
    "import warnings\n",
    "import os\n",
    "from merlin.model import ModelType, PyFuncModel\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Initialize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Set Server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merlin.set_url(\"http://localhost:8080\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Set Active Project\n",
    "\n",
    "`project` represent a project in real life. You may have multiple model within a project.\n",
    "\n",
    "`merlin.set_project(<project_name>)` will set the active project into the name matched by argument. You can only set it to an existing project. If you would like to create a new project, please do so from the MLP console at http://localhost:8080/projects/create."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merlin.set_project(\"sample\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Set Active Model\n",
    "\n",
    "`model` represents an abstract ML model. Conceptually, `model` in Merlin is similar to a class in programming language. To instantiate a `model` you'll have to create a `model_version`.\n",
    "\n",
    "Each `model` has a type, currently model type supported by Merlin are: sklearn, xgboost, tensorflow, pytorch, and user defined model (i.e. pyfunc model).\n",
    "\n",
    "`model_version` represents a snapshot of particular `model` iteration. You'll be able to attach information such as metrics and tag to a given `model_version` as well as deploy it as a model service.\n",
    "\n",
    "`merlin.set_model(<model_name>, <model_type>)` will set the active model to the name given by parameter, if the model with given name is not found, a new model will be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merlin.set_model(\"pyfunc-metric\", ModelType.PYFUNC)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Create Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step we are going to create an echo model which count the number of incoming request as a metrics called `my_counter` and return the incoming request as response."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Define PyFunc Model Class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a PyFunc model you'll have to extend `merlin.PyFuncModel` class and implement its `initialize` and `infer` method.\n",
    "\n",
    "`initialize` will be called once during model initialization. The argument to `initialize` is a dictionary containing a key value pair of artifact name and its URL. The artifact's keys are the same value as received by `log_pyfunc_model`.\n",
    "\n",
    "`infer` method is the prediction method that is need to be implemented. It accept a dictionary type argument which represent incoming request body. `infer` should return a dictionary object which correspond to response body of prediction result.\n",
    "\n",
    "In following example we are creating PyFunc model called `EchoModel`. In its `initialize` method we create a prometheus counter with `my_counter` as metrics name. The `infer` method itself is just increment the counter and simply return incoming request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from prometheus_client import Counter\n",
    "\n",
    "class EchoModel(PyFuncModel):\n",
    "    def initialize(self, artifacts):\n",
    "        self.counter = Counter(\"my_counter\", 'My custom counter')\n",
    "    \n",
    "    def infer(self, request):\n",
    "        self.counter.inc()\n",
    "        return request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test it locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = EchoModel()\n",
    "m.initialize({})\n",
    "m.infer({\"instances\": [[1,2,3,4], [2,1,2,4]] })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Deploy Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To deploy the model, we will have to create an iteration of the model (by create a `model_version`), upload the serialized model to Merlin, and then deploy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Create Model Version and Upload"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`merlin.new_model_version()` is a convenient method to create a model version and start its development process. It is equal to following codes:\n",
    "\n",
    "```\n",
    "v = model.new_model_version()\n",
    "v.start()\n",
    "v.log_pyfunc_model(model_instance=EchoModel(), \n",
    "                conda_env=\"env.yaml\")\n",
    "v.finish()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To upload PyFunc model you have to provide following arguments:\n",
    "1. `model_instance` is the instance of PyFunc model, the model has to extend `merlin.PyFuncModel`\n",
    "2. `conda_env` is path to conda environment yaml file. The environment yaml file must contain all dependency required by the PyFunc model.\n",
    "3. (Optional) `artifacts` is additional artifact that you want to include in the model\n",
    "4. (Optional) `code_dir` is a list of directory containing python code that will be loaded during model initialization, this is required when `model_instance` depend on local python package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with merlin.new_model_version() as v:    \n",
    "    merlin.log_pyfunc_model(model_instance=EchoModel(), \n",
    "                conda_env=\"env.yaml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Deploy Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of a deployed model version will have its own generated url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint = merlin.deploy(v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Send Test Request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$endpoint.url\"\n",
    "curl -v -X POST $1 -d '{\n",
    "  \"instances\": [\n",
    "    [2.8,  1.0,  6.8,  0.4],\n",
    "    [3.1,  1.4,  4.5,  1.6]\n",
    "  ]\n",
    "}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Delete Deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merlin.undeploy(v)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Merlin SDK",
   "language": "python",
   "name": "Merlin-sdk"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}