notebooks/server_examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example Model Servers with Seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Seldon Core\n",
    "\n",
    "Use the setup notebook to [Setup Cluster](seldon_core_setup.ipynb#Setup-Cluster) with [Ambassador Ingress](seldon_core_setup.ipynb#Ambassador) and [Seldon Core](seldon_core_setup.ipynb#Install-Seldon-Core). Instructions [also online](./seldon_core_setup.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serve SKlearn Iris Model\n",
    "We can deploy a sklearn model uploaded to an object store by using the sklearn model server implementation as the config below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ../servers/sklearnserver/samples/iris.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ../servers/sklearnserver/samples/iris.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: sklearn\n",
    "spec:\n",
    "  name: iris\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: SKLEARN_SERVER\n",
    "      modelUri: gs://seldon-models/sklearn/iris\n",
    "      name: classifier\n",
    "    name: default\n",
    "    replicas: 1\n",
    "    svcOrchSpec: \n",
    "      env: \n",
    "      - name: SELDON_LOG_LEVEL\n",
    "        value: DEBUG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then we apply it to deploy it to our kubernetes cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!kubectl apply -f ../servers/sklearnserver/samples/iris.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=sklearn -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's deployed we can send our sklearn model requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "sc = SeldonClient(deployment_name=\"sklearn\",namespace=\"seldon\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "r = sc.predict(gateway=\"ambassador\",transport=\"rest\",shape=(1,4))\n",
    "print(r)\n",
    "assert(r.success==True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And delete the model we deployed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!kubectl delete -f ../servers/sklearnserver/samples/iris.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serve XGBoost Iris Model\n",
    "We can deploy a xgboost model uploaded to an object store by using the xgboost model server implementation as the config below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%writefile ../servers/xgboostserver/samples/iris.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: xgboost\n",
    "spec:\n",
    "  name: iris\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: XGBOOST_SERVER\n",
    "      modelUri: gs://seldon-models/xgboost/iris\n",
    "      name: classifier\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then we apply it to deploy it to our kubernetes cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!kubectl apply -f ../servers/xgboostserver/samples/iris.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=xgboost -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's deployed we can send our xgboost model requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "sc = SeldonClient(deployment_name=\"xgboost\",namespace=\"seldon\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "r = sc.predict(gateway=\"ambassador\",transport=\"rest\",shape=(1,4))\n",
    "print(r)\n",
    "assert(r.success==True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And delete the model we deployed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!kubectl delete -f ../servers/xgboostserver/samples/iris.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serve Tensorflow MNIST Model\n",
    "We can deploy a tensorflow model uploaded to an object store by using the tensorflow model server implementation as the config below.\n",
    "\n",
    "This notebook contains two examples, one which shows how you can use the TFServing prepackaged serve with the Seldon Protocol, and a second one which shows how you can deploy it using the tensorlfow protocol (so you can send requests of the exact format as you would to a tfserving server).\n",
    "\n",
    "### Serve Tensorflow MNIST Model with Seldon Protocol\n",
    "\n",
    "The config file below shows how you can deploy your Tensorflow model which exposes the Seldon protocol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ../servers/tfserving/samples/mnist_rest.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ../servers/tfserving/samples/mnist_rest.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: tfserving\n",
    "spec:\n",
    "  name: mnist\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: TENSORFLOW_SERVER\n",
    "      modelUri: gs://seldon-models/tfserving/mnist-model\n",
    "      name: mnist-model\n",
    "      parameters:\n",
    "        - name: signature_name\n",
    "          type: STRING\n",
    "          value: predict_images\n",
    "        - name: model_name\n",
    "          type: STRING\n",
    "          value: mnist-model\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's deployed we can send our sklearn model requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!kubectl apply -f ../servers/tfserving/samples/mnist_rest.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=tfserving -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's deployed we can send our sklearn model requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "sc = SeldonClient(deployment_name=\"tfserving\",namespace=\"seldon\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "r = sc.predict(gateway=\"ambassador\",transport=\"rest\",shape=(1,784))\n",
    "print(r)\n",
    "assert(r.success==True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And delete the model we deployed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!kubectl delete -f ../servers/tfserving/samples/mnist_rest.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serve Tensorflow MNIST Model with Tensorflow protocol\n",
    "\n",
    "The config file below shows how you can deploy your Tensorflow model which exposes the Tensorflow protocol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ../servers/tfserving/samples/halfplustwo_rest.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ../servers/tfserving/samples/halfplustwo_rest.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: hpt\n",
    "spec:\n",
    "  name: hpt\n",
    "  protocol: tensorflow\n",
    "  transport: rest\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: TENSORFLOW_SERVER\n",
    "      modelUri: gs://seldon-models/tfserving/half_plus_two\n",
    "      name:  halfplustwo\n",
    "      parameters:\n",
    "        - name: model_name\n",
    "          type: STRING\n",
    "          value: halfplustwo\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!kubectl apply -f ../servers/tfserving/samples/halfplustwo_rest.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=hpt -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "X=!curl -s -d '{\"instances\": [1.0, 2.0, 5.0]}' \\\n",
    "   -X POST http://localhost:8003/seldon/seldon/hpt/v1/models/halfplustwo/:predict \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "d=json.loads(\"\".join(X))\n",
    "print(d)\n",
    "assert(d[\"predictions\"][0] == 2.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl delete -f ../servers/tfserving/samples/halfplustwo_rest.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serve MLFlow Elasticnet Wines Model\n",
    "We can deploy an MLFlow model uploaded to an object store by using the MLFlow model server implementation as the config below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ../servers/mlflowserver/samples/elasticnet_wine.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ../servers/mlflowserver/samples/elasticnet_wine.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: mlflow\n",
    "spec:\n",
    "  name: wines\n",
    "  predictors:\n",
    "  - componentSpecs:\n",
    "    - spec:\n",
    "        # We are setting high failureThreshold as installing conda dependencies\n",
    "        # can take long time and we want to avoid k8s killing the container prematurely\n",
    "        containers:\n",
    "        - name: classifier\n",
    "          livenessProbe:\n",
    "            initialDelaySeconds: 80\n",
    "            failureThreshold: 200\n",
    "            periodSeconds: 5\n",
    "            successThreshold: 1\n",
    "            httpGet:\n",
    "              path: /health/ping\n",
    "              port: http\n",
    "              scheme: HTTP\n",
    "          readinessProbe:\n",
    "            initialDelaySeconds: 80\n",
    "            failureThreshold: 200\n",
    "            periodSeconds: 5\n",
    "            successThreshold: 1\n",
    "            httpGet:\n",
    "              path: /health/ping\n",
    "              port: http\n",
    "              scheme: HTTP\n",
    "    graph:\n",
    "      children: []\n",
    "      implementation: MLFLOW_SERVER\n",
    "      modelUri: gs://seldon-models/mlflow/elasticnet_wine\n",
    "      name: classifier\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!kubectl apply -f ../servers/mlflowserver/samples/elasticnet_wine.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=mlflow -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "sc = SeldonClient(deployment_name=\"mlflow\",namespace=\"seldon\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "r = sc.predict(gateway=\"ambassador\",transport=\"rest\",shape=(1,11))\n",
    "print(r)\n",
    "assert(r.success==True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!kubectl delete -f ../servers/mlflowserver/samples/elasticnet_wine.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}