notebooks/server_examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example Model Servers with Seldon\n",
    "\n",
    "Follow [docs](https://docs.seldon.io/projects/seldon-core/en/latest/) to install Seldon Core."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "a7884322-4705-454e-8bfb-ae64c013a611"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error from server (AlreadyExists): namespaces \"seldon\" already exists\n",
      "Already exists\n"
     ]
    }
   ],
   "source": [
    "!kubectl create namespace seldon || echo \"Already exists\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "a5f19d76-cb9d-4c43-b3aa-1d22f57796fb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context \"kind-kind\" modified.\n"
     ]
    }
   ],
   "source": [
    "!kubectl config set-context $(kubectl config current-context) --namespace=seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "ca2e1cd5-e856-4131-9f2a-4f8aff4dc36e"
   },
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serve SKLearn Iris Model\n",
    "\n",
    "**Note**: Seldon has adopted the industry-standard Open Inference Protocol (OIP) and is no longer maintaining the Seldon and TensorFlow protocols. This transition allows for greater interoperability among various model serving runtimes, such as MLServer. To learn more about implementing OIP for model serving in Seldon Core 1, see [MLServer](https://docs.seldon.ai/mlserver).\n",
    "\n",
    "We strongly encourage you to adopt the OIP, which provides seamless integration across diverse model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n",
    "\n",
    "In order to deploy SKLearn artifacts, we can leverage the [pre-packaged SKLearn inference server](https://docs.seldon.io/projects/seldon-core/en/latest/servers/sklearn.html).\n",
    "The exposed API can follow either:\n",
    "- Open Inference Protocol or V2 Protocol.\n",
    "- Seldon protocol. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Default Seldon protocol\n",
    "\n",
    "To deploy and start serving an SKLearn artifact using Seldon's default protocol, we can use a config like the one below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "e11682b2-164a-4071-a124-f0dc2efcace0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ../servers/sklearnserver/samples/iris.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ../servers/sklearnserver/samples/iris.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: sklearn\n",
    "spec:\n",
    "  predictors:\n",
    "  - graph:\n",
    "      name: classifier\n",
    "      implementation: SKLEARN_SERVER\n",
    "      modelUri: gs://seldon-models/v1.19.0-dev/sklearn/iris\n",
    "    name: default\n",
    "    replicas: 1\n",
    "    svcOrchSpec:\n",
    "      env:\n",
    "      - name: SELDON_LOG_LEVEL\n",
    "        value: DEBUG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then apply it to deploy it to our Kubernetes cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "5422c1b1-79ee-4dbd-a26c-abe620fa0fb1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/sklearn created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f ../servers/sklearnserver/samples/iris.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "9fb9fb1b-cd8a-48f1-b110-21c479212ef6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"sklearn-default-0-classifier\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=sklearn -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's deployed we can send our sklearn model requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### REST Requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "063556c9-c737-4364-983a-91a2fafe1947"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': {'names': ['t:0', 't:1', 't:2'], 'ndarray': [[9.912315378486718e-07, 0.0007015931307743852, 0.9992974156376878]]}, 'meta': {'requestPath': {'classifier': 'seldonio/sklearnserver:1.18.0'}}}\n"
     ]
    }
   ],
   "source": [
    "X=!curl -s -d '{\"data\": {\"ndarray\":[[1.0, 2.0, 5.0, 6.0]]}}' \\\n",
    "   -X POST http://localhost:8004/seldon/seldon/sklearn/api/v1.0/predictions \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "d=json.loads(X[0])\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "83141dc8-7ff5-4885-a9e6-ee74bcf3e9de"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-01-10 16:09:43.411334: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n",
      "2024-01-10 16:09:43.448748: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.\n",
      "2024-01-10 16:09:43.527779: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.\n",
      "2024-01-10 16:09:43.528677: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2024-01-10 16:09:44.193744: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"
     ]
    }
   ],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "\n",
    "sc = SeldonClient(deployment_name=\"sklearn\", namespace=\"seldon\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "8f88db7e-a0bd-41e1-98b6-ac8f6a7b3da5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "meta {\n",
      "}\n",
      "data {\n",
      "  tensor {\n",
      "    shape: 1\n",
      "    shape: 4\n",
      "    values: 0.17141173176497349\n",
      "    values: 0.5937726115406986\n",
      "    values: 0.3305595820782128\n",
      "    values: 0.2046623631664879\n",
      "  }\n",
      "}\n",
      "\n",
      "Response:\n",
      "{'data': {'names': ['t:0', 't:1', 't:2'], 'tensor': {'shape': [1, 3], 'values': [0.41804496555456905, 0.3930727629239069, 0.1888822715215241]}}, 'meta': {'requestPath': {'classifier': 'seldonio/sklearnserver:1.18.0'}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"rest\", shape=(1, 4))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### gRPC Requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "2ba77986-01f1-4ead-b4ca-d11399ca8912"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "{'meta': {}, 'data': {'tensor': {'shape': [1, 4], 'values': [0.39156951217983293, 0.9277883706101252, 0.965137780400803, 0.7889296899083068]}}}\n",
      "Response:\n",
      "{'meta': {'requestPath': {'classifier': 'seldonio/sklearnserver:1.18.0'}}, 'data': {'names': ['t:0', 't:1', 't:2'], 'tensor': {'shape': [1, 3], 'values': [0.17859883395119414, 0.2418372547184021, 0.5795639113304039]}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"grpc\", shape=(1, 4))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "f6801e58-3d3c-4594-9136-91921132f2ba",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'meta': {'requestPath': {'classifier': 'seldonio/sklearnserver:1.18.0'}}, 'data': {'names': ['t:0', 't:1', 't:2'], 'ndarray': [[9.912315378486718e-07, 0.0007015931307743852, 0.9992974156376878]]}}\n"
     ]
    }
   ],
   "source": [
    "X=!cd ../executor/proto && grpcurl -d '{\"data\":{\"ndarray\":[[1.0,2.0,5.0,6.0]]}}' \\\n",
    "         -rpc-header seldon:sklearn -rpc-header namespace:seldon \\\n",
    "         -plaintext \\\n",
    "         -proto ./prediction.proto  0.0.0.0:8004 seldon.protos.Seldon/Predict\n",
    "d=json.loads(\"\".join(X))\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And delete the model we deployed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "72f981e3-338d-4bdf-8350-be81d410c184"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"sklearn\" deleted\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f ../servers/sklearnserver/samples/iris.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Open Inference Protocol or V2 Protocol\n",
    "\n",
    "For example, we can consider the config below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "5c9eb23c-78a8-49cc-8285-14b41392e043"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./resources/iris-sklearn-v2.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./resources/iris-sklearn-v2.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: sklearn\n",
    "spec:\n",
    "  name: iris\n",
    "  protocol: v2\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: SKLEARN_SERVER\n",
    "      modelUri: gs://seldon-models/sklearn/iris-0.23.2/lr_model\n",
    "      name: classifier\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then apply it to deploy our model to our Kubernetes cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "2a4c2438-2953-42f8-a6c3-ab8e80b02e9b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/sklearn created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f resources/iris-sklearn-v2.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "27a1d48e-4438-4dec-afff-6f7091f002e3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deployment \"sklearn-default-0-classifier\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=sklearn -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's deployed, we can send inference requests to our model.\n",
    "Note that, since it's using the V2 Protocol, these requests will be different to the ones using the default Seldon Protocol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "2aa5f67b-eb03-4055-84f8-fb92164b08b4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"model_name\": \"classifier\",\n",
      "  \"model_version\": \"v1\",\n",
      "  \"id\": \"99c6d14e-4e8b-4458-b41c-6d2922c544a4\",\n",
      "  \"parameters\": {},\n",
      "  \"outputs\": [\n",
      "    {\n",
      "      \"name\": \"predict\",\n",
      "      \"shape\": [\n",
      "        1,\n",
      "        1\n",
      "      ],\n",
      "      \"datatype\": \"INT64\",\n",
      "      \"parameters\": {\n",
      "        \"content_type\": \"np\"\n",
      "      },\n",
      "      \"data\": [\n",
      "        2\n",
      "      ]\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "import requests\n",
    "\n",
    "inference_request = {\n",
    "    \"inputs\": [\n",
    "        {\"name\": \"predict\", \"shape\": [1, 4], \"datatype\": \"FP32\", \"data\": [[1, 2, 3, 4]]}\n",
    "    ]\n",
    "}\n",
    "\n",
    "endpoint = \"http://localhost:8004/seldon/seldon/sklearn/v2/models/infer\"\n",
    "response = requests.post(endpoint, json=inference_request)\n",
    "\n",
    "print(json.dumps(response.json(), indent=2))\n",
    "assert response.ok"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can delete the model we deployed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "e1c09164-2180-4deb-9bb5-a1459daa192c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"sklearn\" deleted\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f resources/iris-sklearn-v2.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serve XGBoost Iris Model\n",
    "\n",
    "In order to deploy XGBoost models, we can leverage the [pre-packaged XGBoost inference server](https://docs.seldon.io/projects/seldon-core/en/latest/servers/xgboost.html).\n",
    "The exposed API can follow either:\n",
    "\n",
    "- The default Seldon protocol. \n",
    "- The V2 protocol"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Default Seldon protocol\n",
    "\n",
    "We can deploy a XGBoost model uploaded to an object store by using the XGBoost model server implementation as shown in the config below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "d8033a2f-d6c9-4cdf-9b61-12c20fab00fc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting resources/iris.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile resources/iris.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: xgboost\n",
    "spec:\n",
    "  name: iris\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: XGBOOST_SERVER\n",
    "      modelUri: gs://seldon-models/xgboost/iris\n",
    "      name: classifier\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then we apply it to deploy it to our kubernetes cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "5f660da1-fe82-4fab-a617-66302680f26b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/xgboost created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f resources/iris.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "b85f5334-8428-4a39-aef4-752fc76b2094"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"xgboost-default-0-classifier\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"xgboost-default-0-classifier\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=xgboost -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Rest Requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "ab8bc83c-bd4e-41ea-8d84-78fe2510ae7f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': {'names': [], 'ndarray': [2.0]}, 'meta': {'requestPath': {'classifier': 'seldonio/xgboostserver:1.18.0'}}}\n"
     ]
    }
   ],
   "source": [
    "X=!curl -s -d '{\"data\": {\"ndarray\":[[1.0, 2.0, 5.0, 6.0]]}}' \\\n",
    "   -X POST http://localhost:8004/seldon/seldon/xgboost/api/v1.0/predictions \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "d=json.loads(X[0])\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "f1f12c00-6d9e-4c85-ac7f-bbf5bafa9068"
   },
   "outputs": [],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "\n",
    "sc = SeldonClient(deployment_name=\"xgboost\", namespace=\"seldon\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "3cad82b0-c041-4f4e-8cc1-a6bd8dd39889"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "meta {\n",
      "}\n",
      "data {\n",
      "  tensor {\n",
      "    shape: 1\n",
      "    shape: 4\n",
      "    values: 0.5059863560669698\n",
      "    values: 0.6819976066125761\n",
      "    values: 0.8325416286359293\n",
      "    values: 0.2531200764967788\n",
      "  }\n",
      "}\n",
      "\n",
      "Response:\n",
      "{'data': {'names': [], 'tensor': {'shape': [1], 'values': [0.0]}}, 'meta': {'requestPath': {'classifier': 'seldonio/xgboostserver:1.18.0'}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"rest\", shape=(1, 4))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### gRPC Requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "09839102-1c4f-4f4a-929c-3e48f139723a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "{'meta': {}, 'data': {'tensor': {'shape': [1, 4], 'values': [0.3402484582438542, 0.7075322398328269, 0.2491942933114213, 0.08837167380879685]}}}\n",
      "Response:\n",
      "{'meta': {'requestPath': {'classifier': 'seldonio/xgboostserver:1.18.0'}}, 'data': {'tensor': {'shape': [1], 'values': [0.0]}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"grpc\", shape=(1, 4))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "fab2ec4e-6d7f-4219-a83e-fb6157bcb970",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'meta': {'requestPath': {'classifier': 'seldonio/xgboostserver:1.18.0'}}, 'data': {'ndarray': [2]}}\n"
     ]
    }
   ],
   "source": [
    "X=!cd ../executor/proto && grpcurl -d '{\"data\":{\"ndarray\":[[1.0,2.0,5.0,6.0]]}}' \\\n",
    "         -rpc-header seldon:xgboost -rpc-header namespace:seldon \\\n",
    "         -plaintext \\\n",
    "         -proto ./prediction.proto  0.0.0.0:8004 seldon.protos.Seldon/Predict\n",
    "d=json.loads(\"\".join(X))\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And delete the model we deployed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "57e4a707-ad6b-487e-a12b-09f8b1142e8e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"xgboost\" deleted\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f resources/iris.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### V2 protocol\n",
    "\n",
    "We can deploy a XGBoost model, exposing an API compatible with v2 protocol by specifying the `protocol` of our `SeldonDeployment` as `v2`.\n",
    "For example, we can consider the config below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "62fd4633-b5d7-43f0-9017-a79cf3fd2e9f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./resources/iris-xgboost-v2.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./resources/iris-xgboost-v2.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: xgboost\n",
    "spec:\n",
    "  name: iris\n",
    "  protocol: v2\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: XGBOOST_SERVER\n",
    "      modelUri: gs://seldon-models/xgboost/iris\n",
    "      name: iris\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then apply it to deploy our model to our Kubernetes cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "ca0523bc-e32b-4533-8988-46a7480cf321"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/xgboost created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f ./resources/iris-xgboost-v2.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "59f853ed-6215-47e1-b4bb-07c202230c64"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"xgboost-default-0-iris\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"xgboost-default-0-iris\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=xgboost -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's deployed, we can send inference requests to our model.\n",
    "Note that, since it's using the V2 Protocol, these requests will be different to the ones using the default Seldon Protocol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "d36513d9-57a2-4b64-8c4d-8bade289344e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"model_name\": \"iris\",\n",
      "  \"model_version\": \"v0.1.0\",\n",
      "  \"id\": \"3cab5b54-2ab7-4fe4-b9a2-7dd0f5346e2f\",\n",
      "  \"parameters\": {},\n",
      "  \"outputs\": [\n",
      "    {\n",
      "      \"name\": \"predict\",\n",
      "      \"shape\": [\n",
      "        1,\n",
      "        1\n",
      "      ],\n",
      "      \"datatype\": \"FP32\",\n",
      "      \"parameters\": {\n",
      "        \"content_type\": \"np\"\n",
      "      },\n",
      "      \"data\": [\n",
      "        2.0\n",
      "      ]\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "import requests\n",
    "\n",
    "inference_request = {\n",
    "    \"inputs\": [\n",
    "        {\"name\": \"predict\", \"shape\": [1, 4], \"datatype\": \"FP32\", \"data\": [[1, 2, 3, 4]]}\n",
    "    ]\n",
    "}\n",
    "\n",
    "endpoint = \"http://localhost:8004/seldon/seldon/xgboost/v2/models/infer\"\n",
    "response = requests.post(endpoint, json=inference_request)\n",
    "\n",
    "print(json.dumps(response.json(), indent=2))\n",
    "assert response.ok"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can delete the model we deployed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "c39b0340-2218-4bb5-9bde-75fc52e3bb8f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"xgboost\" deleted\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f ./resources/iris-xgboost-v2.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serve Tensorflow MNIST Model\n",
    "We can deploy a tensorflow model uploaded to an object store by using the\n",
    "tensorflow model server implementation as the config below.\n",
    "\n",
    "This notebook contains two examples, one which shows how you can use the\n",
    "TFServing prepackaged serve with the Seldon Protocol, and a second one which\n",
    "shows how you can deploy it using the tensorlfow protocol (so you can send\n",
    "requests of the exact format as you would to a tfserving server)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Serve Tensorflow MNIST Model with Seldon Protocol\n",
    "\n",
    "The config file below shows how you can deploy your Tensorflow model which\n",
    "exposes the Seldon protocol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "3abf7dc8-3c74-4a53-902a-2917caca3b02"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./resources/mnist_rest.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./resources/mnist_rest.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: tfserving\n",
    "spec:\n",
    "  name: mnist\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: TENSORFLOW_SERVER\n",
    "      modelUri: gs://seldon-models/tfserving/mnist-model\n",
    "      name: mnist-model\n",
    "      parameters:\n",
    "        - name: signature_name\n",
    "          type: STRING\n",
    "          value: predict_images\n",
    "        - name: model_name\n",
    "          type: STRING\n",
    "          value: mnist-model\n",
    "        - name: model_input\n",
    "          type: STRING\n",
    "          value: images\n",
    "        - name: model_output\n",
    "          type: STRING\n",
    "          value: scores\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "id": "790f123a-9d45-48b8-b4b1-ec424d73f33f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/tfserving created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f ./resources/mnist_rest.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "0a9ba950-f33a-40a8-8d0e-beba7ec46da1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"tfserving-default-0-mnist-model\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"tfserving-default-0-mnist-model\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=tfserving -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "id": "8c75c9fe-8f0a-48d3-a610-45b7fbeaef83"
   },
   "outputs": [],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "\n",
    "sc = SeldonClient(deployment_name=\"tfserving\", namespace=\"seldon\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### REST Request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "id": "922a4ed5-b939-45bf-85d7-6b9bbf2ea344"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "meta {\n",
      "}\n",
      "data {\n",
      "  tensor {\n",
      "    shape: 1\n",
      "    shape: 784\n",
      "    values: 0.40042030856645994\n",
      "    values: 0.12706617370273143\n",
      "    values: 0.5995010949374973\n",
      "    values: 0.20673616604277123\n",
      "    values: 0.267965079749103\n",
      "    values: 0.28732441282638177\n",
      "    values: 0.05431953974251236\n",
      "    values: 0.9386931014283844\n",
      "    values: 0.7783730270524005\n",
      "    values: 0.33817086214875713\n",
      "    values: 0.29145765680083213\n",
      "    values: 0.44285879591906574\n",
      "    values: 0.6551587968022324\n",
      "    values: 0.31238417218286163\n",
      "    values: 0.4256316305209036\n",
      "    values: 0.6511456989870854\n",
      "    values: 0.5006871383565527\n",
      "    values: 0.4686249869286161\n",
      "    values: 0.12155327610393096\n",
      "    values: 0.37477288407329035\n",
      "    values: 0.8231317995094778\n",
      "    values: 0.48279144040735\n",
      "    values: 0.6486814039080503\n",
      "    values: 0.4618496078306792\n",
      "    values: 0.849852223594817\n",
      "    values: 0.5694862005802025\n",
      "    values: 0.6927834049569376\n",
      "    values: 0.7441208124168749\n",
      "    values: 0.06791460371138147\n",
      "    values: 0.9385293032052238\n",
      "    values: 0.603400825306626\n",
      "    values: 0.43896138616652847\n",
      "    values: 0.9672913419920942\n",
      "    values: 0.21631413220109796\n",
      "    values: 0.5624466080944204\n",
      "    values: 0.6750025201131201\n",
      "    values: 0.6854662639423265\n",
      "    values: 0.7436185407965809\n",
      "    values: 0.6677067769398716\n",
      "    values: 0.6598140034790021\n",
      "    values: 0.13240513166054046\n",
      "    values: 0.633350182215756\n",
      "    values: 0.07309764459984713\n",
      "    values: 0.989834783786183\n",
      "    values: 0.7767938966676395\n",
      "    values: 0.16704509222933184\n",
      "    values: 0.9165170740033651\n",
      "    values: 0.7743588637498594\n",
      "    values: 0.7277921690932967\n",
      "    values: 0.22823755344717733\n",
      "    values: 0.5927319172617582\n",
      "    values: 0.5937748370912145\n",
      "    values: 0.3219925151589419\n",
      "    values: 0.9579033278660997\n",
      "    values: 0.28320148661035516\n",
      "    values: 0.872925428322407\n",
      "    values: 0.865093114698652\n",
      "    values: 0.5671200938859204\n",
      "    values: 0.12342215796631395\n",
      "    values: 0.06168027942989884\n",
      "    values: 0.6780360733848022\n",
      "    values: 0.8079706943064379\n",
      "    values: 0.6648363401411947\n",
      "    values: 0.8578632362252419\n",
      "    values: 0.45327664580190163\n",
      "    values: 0.27039311457857096\n",
      "    values: 0.850734695860167\n",
      "    values: 0.6400356495942985\n",
      "    values: 0.44086734994340404\n",
      "    values: 0.9085392096493083\n",
      "    values: 0.9387785408710769\n",
      "    values: 0.05571670867151002\n",
      "    values: 0.9714553267086583\n",
      "    values: 0.21347772294930212\n",
      "    values: 0.8413313599741823\n",
      "    values: 0.8776074201494235\n",
      "    values: 0.20205946696815646\n",
      "    values: 0.8707582071093695\n",
      "    values: 0.8333971708656888\n",
      "    values: 0.440933290602714\n",
      "    values: 0.6980336488502751\n",
      "    values: 0.0219863543248483\n",
      "    values: 0.3479315642981834\n",
      "    values: 0.3100423418670978\n",
      "    values: 0.2968803467810235\n",
      "    values: 0.30026491909707265\n",
      "    values: 0.6277191155910714\n",
      "    values: 0.9536906397948098\n",
      "    values: 0.611456796041938\n",
      "    values: 0.4616559053755851\n",
      "    values: 0.19575826825285014\n",
      "    values: 0.591124872245425\n",
      "    values: 0.7067071306936854\n",
      "    values: 0.8245028009034754\n",
      "    values: 0.05822402473383448\n",
      "    values: 0.6157630413853799\n",
      "    values: 0.5331841496412985\n",
      "    values: 0.07554345777138294\n",
      "    values: 0.8388045326190345\n",
      "    values: 0.08870066374900332\n",
      "    values: 0.30818530735029104\n",
      "    values: 0.7158552028314802\n",
      "    values: 0.1318903095824845\n",
      "    values: 0.8922383206055012\n",
      "    values: 0.33285494815574446\n",
      "    values: 0.6021481820386826\n",
      "    values: 0.07130092973637814\n",
      "    values: 0.6161669869892523\n",
      "    values: 0.2030726363106533\n",
      "    values: 0.974045411471201\n",
      "    values: 0.8680698351956628\n",
      "    values: 0.10232485528932944\n",
      "    values: 0.8194611272236905\n",
      "    values: 0.5234414990860082\n",
      "    values: 0.17170186082177075\n",
      "    values: 0.7994065593233911\n",
      "    values: 0.6289850123162115\n",
      "    values: 0.019747368723527003\n",
      "    values: 0.26680186552632046\n",
      "    values: 0.8145561398671172\n",
      "    values: 0.9300140417400974\n",
      "    values: 0.21942326201647644\n",
      "    values: 0.19556710163344515\n",
      "    values: 0.5569141899721448\n",
      "    values: 0.026953087074983118\n",
      "    values: 0.9770301735642452\n",
      "    values: 0.35943429019066697\n",
      "    values: 0.9909260086099312\n",
      "    values: 0.8308169708412381\n",
      "    values: 0.2794303020430591\n",
      "    values: 0.08152887055711289\n",
      "    values: 0.18519346178942941\n",
      "    values: 0.14762334726604898\n",
      "    values: 0.08971954159526685\n",
      "    values: 0.6551884850738449\n",
      "    values: 0.2364049751575481\n",
      "    values: 0.37617151093229784\n",
      "    values: 0.10473742722250079\n",
      "    values: 0.5215420182273054\n",
      "    values: 0.8254766317620303\n",
      "    values: 0.7745397016285644\n",
      "    values: 0.7846155305314919\n",
      "    values: 0.23070499971530345\n",
      "    values: 0.5844807187531911\n",
      "    values: 0.21151741168966098\n",
      "    values: 0.15653637092911743\n",
      "    values: 0.21468271332623778\n",
      "    values: 0.8547827116293403\n",
      "    values: 0.5981178793325721\n",
      "    values: 0.8262320780914909\n",
      "    values: 0.6703655289437207\n",
      "    values: 0.524453183814789\n",
      "    values: 0.3934225651989858\n",
      "    values: 0.9416966381299064\n",
      "    values: 0.5811030870398775\n",
      "    values: 0.4765457781687924\n",
      "    values: 0.7054038843311344\n",
      "    values: 0.4912701885425007\n",
      "    values: 0.7293071941114737\n",
      "    values: 0.3336296925586394\n",
      "    values: 0.12441832852067647\n",
      "    values: 0.9517614095473921\n",
      "    values: 0.030040326613105406\n",
      "    values: 0.250959234636025\n",
      "    values: 0.1999091442701898\n",
      "    values: 0.3852968632350193\n",
      "    values: 0.12592039081304396\n",
      "    values: 0.6252057420138347\n",
      "    values: 0.7925361422448846\n",
      "    values: 0.9165487379883502\n",
      "    values: 0.8308962518891186\n",
      "    values: 0.039817331823081226\n",
      "    values: 0.21380368006304507\n",
      "    values: 0.1583080982136792\n",
      "    values: 0.6305214240852576\n",
      "    values: 0.6189030994237334\n",
      "    values: 0.4976717437268079\n",
      "    values: 0.8313710648022805\n",
      "    values: 0.5915351636227794\n",
      "    values: 0.28915631455191615\n",
      "    values: 0.7845509839705243\n",
      "    values: 0.09746071957552371\n",
      "    values: 0.097863413711336\n",
      "    values: 0.15189928864630997\n",
      "    values: 0.6580121350821213\n",
      "    values: 0.24183766340877955\n",
      "    values: 0.8503592591471376\n",
      "    values: 0.03272873691446754\n",
      "    values: 0.9848374389529514\n",
      "    values: 0.6439869945183643\n",
      "    values: 0.8701438291344454\n",
      "    values: 0.5762291996842376\n",
      "    values: 0.8895312481482751\n",
      "    values: 0.5628070673664918\n",
      "    values: 0.16564273205496827\n",
      "    values: 0.7549801969543055\n",
      "    values: 0.813412353612258\n",
      "    values: 0.42662655466993804\n",
      "    values: 0.42237054956524245\n",
      "    values: 0.07719457062109736\n",
      "    values: 0.27319734760625247\n",
      "    values: 0.42582519057388113\n",
      "    values: 0.34300376573350466\n",
      "    values: 0.8000311972930885\n",
      "    values: 0.9400938719667916\n",
      "    values: 0.7962891022280469\n",
      "    values: 0.7371664001593162\n",
      "    values: 0.42157502284408\n",
      "    values: 0.4360178324706838\n",
      "    values: 0.9929158828870517\n",
      "    values: 0.11561084469848448\n",
      "    values: 0.8666571576358613\n",
      "    values: 0.062377416289506216\n",
      "    values: 0.33129251474184795\n",
      "    values: 0.13614016463516265\n",
      "    values: 0.576546060689628\n",
      "    values: 0.12011448851657724\n",
      "    values: 0.3912762818209131\n",
      "    values: 0.464614728780658\n",
      "    values: 0.971788414970606\n",
      "    values: 0.7448830274363585\n",
      "    values: 0.9958691017579306\n",
      "    values: 0.7081347089809419\n",
      "    values: 0.35384179612923816\n",
      "    values: 0.8739494130262287\n",
      "    values: 0.83618837517398\n",
      "    values: 0.16542630625280363\n",
      "    values: 0.30224596671593085\n",
      "    values: 0.2695858275529318\n",
      "    values: 0.32593719735560167\n",
      "    values: 0.15829297968282874\n",
      "    values: 0.8521750132452464\n",
      "    values: 0.3422193832053866\n",
      "    values: 0.5662959173092238\n",
      "    values: 0.08196185839519565\n",
      "    values: 0.49417103206253976\n",
      "    values: 0.1537211214582349\n",
      "    values: 0.5596101344163842\n",
      "    values: 0.1657085526327683\n",
      "    values: 0.5007841816263026\n",
      "    values: 0.3666794128116245\n",
      "    values: 0.6846960934156174\n",
      "    values: 0.1552404955922062\n",
      "    values: 0.7343942390191002\n",
      "    values: 0.5765799840287049\n",
      "    values: 0.5113249777365446\n",
      "    values: 0.87034044190804\n",
      "    values: 0.5823765643031866\n",
      "    values: 0.04627214068516983\n",
      "    values: 0.7810548133139147\n",
      "    values: 0.01753533408912078\n",
      "    values: 0.096927926658813\n",
      "    values: 0.41016903120551196\n",
      "    values: 0.06078575911314077\n",
      "    values: 0.11969686783063993\n",
      "    values: 0.586455785644092\n",
      "    values: 0.9255547652186291\n",
      "    values: 0.19064087541557428\n",
      "    values: 0.4875379565750708\n",
      "    values: 0.11213834814526191\n",
      "    values: 0.9580928814650108\n",
      "    values: 0.7085325708556545\n",
      "    values: 0.7412298471193591\n",
      "    values: 0.3644935040828432\n",
      "    values: 0.0006192185756462854\n",
      "    values: 0.4400432866151328\n",
      "    values: 0.6889195554796107\n",
      "    values: 0.8945746096174412\n",
      "    values: 0.4798242522575198\n",
      "    values: 0.09958267919381858\n",
      "    values: 0.012857704052786967\n",
      "    values: 0.14174375988805166\n",
      "    values: 0.6742597812587166\n",
      "    values: 0.6244963503821598\n",
      "    values: 0.3642739089096455\n",
      "    values: 0.14961956462302473\n",
      "    values: 0.046813425733917446\n",
      "    values: 0.49765787099527614\n",
      "    values: 0.7242949536815119\n",
      "    values: 0.8717630353327681\n",
      "    values: 0.8683544918378224\n",
      "    values: 0.8667543075744667\n",
      "    values: 0.680130965126705\n",
      "    values: 0.1039627814042754\n",
      "    values: 0.7878586530898906\n",
      "    values: 0.8192856191382673\n",
      "    values: 0.2942029135953579\n",
      "    values: 0.9335051265027072\n",
      "    values: 0.6686956925368511\n",
      "    values: 0.021721340093141528\n",
      "    values: 0.6225903238193273\n",
      "    values: 0.9828618558042794\n",
      "    values: 0.5010703445721822\n",
      "    values: 0.9669009644872706\n",
      "    values: 0.48196995549163213\n",
      "    values: 0.37447900865291195\n",
      "    values: 0.8307960524375816\n",
      "    values: 0.8722264568762788\n",
      "    values: 0.9884095215757212\n",
      "    values: 0.25201667739800815\n",
      "    values: 0.24150074373440544\n",
      "    values: 0.6492639780755063\n",
      "    values: 0.8863131440263702\n",
      "    values: 0.09080820531570566\n",
      "    values: 0.6817779800308623\n",
      "    values: 0.7769255218398078\n",
      "    values: 0.31776966919778626\n",
      "    values: 0.43797574005306017\n",
      "    values: 0.466931435896639\n",
      "    values: 0.8447017214272248\n",
      "    values: 0.406730452752176\n",
      "    values: 0.0758613008415393\n",
      "    values: 0.6337477952983234\n",
      "    values: 0.31600857835067997\n",
      "    values: 0.5449147364425556\n",
      "    values: 0.4490493198475377\n",
      "    values: 0.6352541079951081\n",
      "    values: 0.8849812114375933\n",
      "    values: 0.9564827102983281\n",
      "    values: 0.8161793027818625\n",
      "    values: 0.32515277036591994\n",
      "    values: 0.0954601597284962\n",
      "    values: 0.58659084978464\n",
      "    values: 0.0998137217340983\n",
      "    values: 0.599328695816969\n",
      "    values: 0.2628118434190768\n",
      "    values: 0.5030803654424614\n",
      "    values: 0.23296009075014634\n",
      "    values: 0.9582794437676396\n",
      "    values: 0.31431922227524534\n",
      "    values: 0.5788766567455906\n",
      "    values: 0.583391371636573\n",
      "    values: 0.23621334023155238\n",
      "    values: 0.40988872058756876\n",
      "    values: 0.8463561176995359\n",
      "    values: 0.33045724436369694\n",
      "    values: 0.9899427774621905\n",
      "    values: 0.08520500290284372\n",
      "    values: 0.8001442839886551\n",
      "    values: 0.09509287457915572\n",
      "    values: 0.05622248664454588\n",
      "    values: 0.9401051446887039\n",
      "    values: 0.2574454445948382\n",
      "    values: 0.07243860137123292\n",
      "    values: 0.7724268969500797\n",
      "    values: 0.5296648446226829\n",
      "    values: 0.7332147769459911\n",
      "    values: 0.991774755432883\n",
      "    values: 0.13832957481754038\n",
      "    values: 0.0880271062731195\n",
      "    values: 0.09541082399953604\n",
      "    values: 0.684777629491355\n",
      "    values: 0.6693717010313193\n",
      "    values: 0.026147757712143616\n",
      "    values: 0.7363846410068133\n",
      "    values: 0.2548413381588627\n",
      "    values: 0.022995870370375204\n",
      "    values: 0.1971132229371908\n",
      "    values: 0.9431898781162932\n",
      "    values: 0.7582372875741431\n",
      "    values: 0.7753855104196299\n",
      "    values: 0.7439775612720075\n",
      "    values: 0.7604726634740231\n",
      "    values: 0.4363790478025168\n",
      "    values: 0.8479855184668654\n",
      "    values: 0.29366645266407854\n",
      "    values: 0.43561080136327746\n",
      "    values: 0.846565613238752\n",
      "    values: 0.2925152851851528\n",
      "    values: 0.7697038455704354\n",
      "    values: 0.6747583813708982\n",
      "    values: 0.8768384575177233\n",
      "    values: 0.3585415296208958\n",
      "    values: 0.1043492132207845\n",
      "    values: 0.3766833472407186\n",
      "    values: 0.6144382338578266\n",
      "    values: 0.9348049944593937\n",
      "    values: 0.6475214364707088\n",
      "    values: 0.029183268830357934\n",
      "    values: 0.6015693697898158\n",
      "    values: 0.716428498148026\n",
      "    values: 0.12190166677705205\n",
      "    values: 0.4556608109559179\n",
      "    values: 0.5373331840653155\n",
      "    values: 0.728373754268816\n",
      "    values: 0.6804231255370042\n",
      "    values: 0.9451388300116323\n",
      "    values: 0.1722216140695435\n",
      "    values: 0.009948777272247744\n",
      "    values: 0.8329147017989859\n",
      "    values: 0.5516677998853604\n",
      "    values: 0.8056310918709919\n",
      "    values: 0.44811919584521076\n",
      "    values: 0.30122143891103514\n",
      "    values: 0.9080281488159193\n",
      "    values: 0.03877458570730219\n",
      "    values: 0.3787702372788333\n",
      "    values: 0.28071869929325477\n",
      "    values: 0.25620167512864345\n",
      "    values: 0.12745880794607234\n",
      "    values: 0.5969820993473214\n",
      "    values: 0.5970567710753318\n",
      "    values: 0.8435352184614504\n",
      "    values: 0.29558886901624215\n",
      "    values: 0.9359074938305227\n",
      "    values: 0.9158761550854784\n",
      "    values: 0.5873288706157358\n",
      "    values: 0.38295350417552676\n",
      "    values: 0.9628229593385594\n",
      "    values: 0.8076658079676605\n",
      "    values: 0.23158532103092366\n",
      "    values: 0.6570531458076208\n",
      "    values: 0.45518886677894377\n",
      "    values: 0.3264674398850339\n",
      "    values: 0.2444242889378342\n",
      "    values: 0.5141989266437995\n",
      "    values: 0.16148265153881247\n",
      "    values: 0.4440348675381278\n",
      "    values: 0.6211048138794073\n",
      "    values: 0.6565708139642651\n",
      "    values: 0.5468574198276173\n",
      "    values: 0.8854563372675569\n",
      "    values: 0.4663424781448757\n",
      "    values: 0.6734371980179013\n",
      "    values: 0.11061509039002304\n",
      "    values: 0.07284361752105195\n",
      "    values: 0.06394653698165287\n",
      "    values: 0.3890494124994379\n",
      "    values: 0.9509399277743682\n",
      "    values: 0.45574396979553167\n",
      "    values: 0.11373269661891205\n",
      "    values: 0.25471418971507687\n",
      "    values: 0.4755528167131128\n",
      "    values: 0.4697597332586574\n",
      "    values: 0.8889197090048873\n",
      "    values: 0.024286269446541997\n",
      "    values: 0.43244843729521665\n",
      "    values: 0.33208076770733486\n",
      "    values: 0.9619102796551526\n",
      "    values: 0.49476933194730943\n",
      "    values: 0.012578191369302716\n",
      "    values: 0.39279810223257927\n",
      "    values: 0.7743688455507176\n",
      "    values: 0.8833985502307768\n",
      "    values: 0.7756056605865619\n",
      "    values: 0.5271751328383691\n",
      "    values: 0.9565587994623225\n",
      "    values: 0.6647802275362388\n",
      "    values: 0.14060161766328294\n",
      "    values: 0.2771374645480985\n",
      "    values: 0.5972852791735968\n",
      "    values: 0.9376840538009811\n",
      "    values: 0.3055840162492778\n",
      "    values: 0.3959083568318503\n",
      "    values: 0.669627397826262\n",
      "    values: 0.05781009027215489\n",
      "    values: 0.31535580182422995\n",
      "    values: 0.42197425353467255\n",
      "    values: 0.663765465086562\n",
      "    values: 0.7223381716766016\n",
      "    values: 0.06320160540769781\n",
      "    values: 0.7545260821673904\n",
      "    values: 0.31775665796123564\n",
      "    values: 0.7078367802893542\n",
      "    values: 0.7233528843338944\n",
      "    values: 0.8134566569046943\n",
      "    values: 0.26488541570716795\n",
      "    values: 0.72530738594891\n",
      "    values: 0.9443419064241455\n",
      "    values: 0.9951241215956158\n",
      "    values: 0.5866703643748258\n",
      "    values: 0.7256735501815762\n",
      "    values: 0.8458151422236926\n",
      "    values: 0.14784247861699173\n",
      "    values: 0.7380432572551165\n",
      "    values: 0.5189494534340429\n",
      "    values: 0.8004776979944521\n",
      "    values: 0.8945870932776523\n",
      "    values: 0.21547189197242256\n",
      "    values: 0.2813587750840635\n",
      "    values: 0.4925798797603491\n",
      "    values: 0.14361987603236204\n",
      "    values: 0.5372751973846441\n",
      "    values: 0.9710768835296164\n",
      "    values: 0.0698767841490362\n",
      "    values: 0.8641970396514891\n",
      "    values: 0.7545606755618388\n",
      "    values: 0.7122742803184413\n",
      "    values: 0.6974227222790047\n",
      "    values: 0.8117569670978347\n",
      "    values: 0.4365486455652098\n",
      "    values: 0.3706828596936296\n",
      "    values: 0.170562442510385\n",
      "    values: 0.39394701264730725\n",
      "    values: 0.09814626442739594\n",
      "    values: 0.5933512547188575\n",
      "    values: 0.45904182275620686\n",
      "    values: 0.7796854524068374\n",
      "    values: 0.978042319358437\n",
      "    values: 0.1274520660270938\n",
      "    values: 0.0447901631855645\n",
      "    values: 0.2721351352746172\n",
      "    values: 0.12799151739606762\n",
      "    values: 0.7276434357376849\n",
      "    values: 0.8686010094768765\n",
      "    values: 0.07982826675884558\n",
      "    values: 0.26720682458350153\n",
      "    values: 0.5989971850838636\n",
      "    values: 0.705423213958317\n",
      "    values: 0.35343281130473214\n",
      "    values: 0.5199886746609936\n",
      "    values: 0.6466286512669338\n",
      "    values: 0.15525782500507268\n",
      "    values: 0.47166639635795893\n",
      "    values: 0.4927522360522425\n",
      "    values: 0.3156108503958961\n",
      "    values: 0.46750432907825445\n",
      "    values: 0.9491679107701392\n",
      "    values: 0.20300753497501778\n",
      "    values: 0.7798330625168591\n",
      "    values: 0.820715957062062\n",
      "    values: 0.009981683551279708\n",
      "    values: 0.9374812680302831\n",
      "    values: 0.13273317075711244\n",
      "    values: 0.9459882355565733\n",
      "    values: 0.047738570805289204\n",
      "    values: 0.5167745459619151\n",
      "    values: 0.12565758436390384\n",
      "    values: 0.8135916518934602\n",
      "    values: 0.5012603580691297\n",
      "    values: 0.4915913294537142\n",
      "    values: 0.710329101399897\n",
      "    values: 0.40945250200715355\n",
      "    values: 0.6817239348940404\n",
      "    values: 0.08476502798614072\n",
      "    values: 0.33636624644226154\n",
      "    values: 0.04086718325461136\n",
      "    values: 0.910801758631036\n",
      "    values: 0.35942972877316826\n",
      "    values: 0.3764657856174899\n",
      "    values: 0.5278287765032823\n",
      "    values: 0.20988497161959407\n",
      "    values: 0.32440097426495984\n",
      "    values: 0.7059028842919635\n",
      "    values: 0.21058259271403112\n",
      "    values: 0.6727717137661621\n",
      "    values: 0.9932716967109171\n",
      "    values: 0.8347804003948003\n",
      "    values: 0.6556301815398304\n",
      "    values: 0.541823131159416\n",
      "    values: 0.5432000492956476\n",
      "    values: 0.3210311578369238\n",
      "    values: 0.39282104415011143\n",
      "    values: 0.33811366056539816\n",
      "    values: 0.10943333907008523\n",
      "    values: 0.1209935952662976\n",
      "    values: 0.4101752168225993\n",
      "    values: 0.8826172035440437\n",
      "    values: 0.9530863224899422\n",
      "    values: 0.7203977643458354\n",
      "    values: 0.4795741461023928\n",
      "    values: 0.9180525045549714\n",
      "    values: 0.9625028527805679\n",
      "    values: 0.5070737933349118\n",
      "    values: 0.5731291303930872\n",
      "    values: 0.995373966865749\n",
      "    values: 0.4046061419827941\n",
      "    values: 0.9078546357489615\n",
      "    values: 0.013296642888238286\n",
      "    values: 0.500087607855155\n",
      "    values: 0.31556275807261225\n",
      "    values: 0.1988433638004271\n",
      "    values: 0.22870241811735004\n",
      "    values: 0.8675085091037273\n",
      "    values: 0.20275911658992218\n",
      "    values: 0.9334679932088872\n",
      "    values: 0.4558630748292568\n",
      "    values: 0.4462061377296065\n",
      "    values: 0.35666451042389147\n",
      "    values: 0.16519787759102966\n",
      "    values: 0.8975334632727425\n",
      "    values: 0.5406181104552787\n",
      "    values: 0.13689204292573076\n",
      "    values: 0.5803976227188142\n",
      "    values: 0.4314477329157014\n",
      "    values: 0.7090077328153647\n",
      "    values: 0.0929633021566667\n",
      "    values: 0.4143102452547204\n",
      "    values: 0.816350787077863\n",
      "    values: 0.48432072460668263\n",
      "    values: 0.3450769576076126\n",
      "    values: 0.7381857507825426\n",
      "    values: 0.3993682606110922\n",
      "    values: 0.3113664002859645\n",
      "    values: 0.3245649678699538\n",
      "    values: 0.7482049331994516\n",
      "    values: 0.2117747302479822\n",
      "    values: 0.4785987872509587\n",
      "    values: 0.40294256496075465\n",
      "    values: 0.6405647206604268\n",
      "    values: 0.9968356701086415\n",
      "    values: 0.10750823416422406\n",
      "    values: 0.28771832194270064\n",
      "    values: 0.4019864765054858\n",
      "    values: 0.9704554475794165\n",
      "    values: 0.721617832668024\n",
      "    values: 0.22873917662678578\n",
      "    values: 0.37123672093798676\n",
      "    values: 0.29969322937540854\n",
      "    values: 0.30663706771354615\n",
      "    values: 0.269049342183323\n",
      "    values: 0.37355995538054443\n",
      "    values: 0.01171334470141705\n",
      "    values: 0.6524901244388978\n",
      "    values: 0.15108862629738595\n",
      "    values: 0.5360323137137989\n",
      "    values: 0.020722910001765182\n",
      "    values: 0.13188897850804793\n",
      "    values: 0.011385425264361149\n",
      "    values: 0.4348010057735693\n",
      "    values: 0.7078134320342155\n",
      "    values: 0.5277804102626256\n",
      "    values: 0.23244564596315664\n",
      "    values: 0.6465738203945826\n",
      "    values: 0.9488869176731716\n",
      "    values: 0.036640974606355625\n",
      "    values: 0.3693353554791605\n",
      "    values: 0.31849653017043567\n",
      "    values: 0.21905578724045116\n",
      "    values: 0.24623451075073377\n",
      "    values: 0.426516155002016\n",
      "    values: 0.6885901036295276\n",
      "    values: 0.21500659973134795\n",
      "    values: 0.4370332703931602\n",
      "    values: 0.08976216746923815\n",
      "    values: 0.525063444445776\n",
      "    values: 0.07221779687128316\n",
      "    values: 0.33799206277956517\n",
      "    values: 0.9190849110792108\n",
      "    values: 0.05706172228560902\n",
      "    values: 0.43947948503272083\n",
      "    values: 0.3960577730730783\n",
      "    values: 0.7100532042935843\n",
      "    values: 0.6924040228382837\n",
      "    values: 0.9210204279400606\n",
      "    values: 0.6466877588004382\n",
      "    values: 0.41030249829164545\n",
      "    values: 0.9058056698082396\n",
      "    values: 0.48866762542147035\n",
      "    values: 0.9699685226185767\n",
      "    values: 0.9020196592748602\n",
      "    values: 0.48510992802272224\n",
      "    values: 0.08441380054487113\n",
      "    values: 0.8421920177027633\n",
      "    values: 0.11709678342034013\n",
      "    values: 0.8532720011649287\n",
      "    values: 0.2451545444542087\n",
      "    values: 0.5884486154426892\n",
      "    values: 0.11607911199705223\n",
      "    values: 0.6774676070696881\n",
      "    values: 0.23818013788862913\n",
      "    values: 0.6294261725298481\n",
      "    values: 0.3591605315335321\n",
      "    values: 0.9366991479498308\n",
      "    values: 0.13718669443343423\n",
      "    values: 0.41669146459690953\n",
      "    values: 0.20560875856222216\n",
      "    values: 0.16223396577912308\n",
      "    values: 0.9874286213575769\n",
      "    values: 0.38684933752025596\n",
      "    values: 0.6692220301420121\n",
      "    values: 0.49817019657495654\n",
      "    values: 0.20319257377366617\n",
      "    values: 0.8301939734561454\n",
      "    values: 0.289242072648237\n",
      "    values: 0.9152224408390173\n",
      "    values: 0.6396685092150689\n",
      "    values: 0.36148432082360715\n",
      "    values: 0.29099997778126896\n",
      "    values: 0.9488195614060585\n",
      "    values: 0.917500868789783\n",
      "    values: 0.9623467380249572\n",
      "    values: 0.7203079968951823\n",
      "    values: 0.5463395433735262\n",
      "    values: 0.42097157281311914\n",
      "    values: 0.4434243126542624\n",
      "    values: 0.7813308885807294\n",
      "    values: 0.32799555236085376\n",
      "    values: 0.49398675419465554\n",
      "    values: 0.16081476479455714\n",
      "    values: 0.9358687349962205\n",
      "    values: 0.8450740070791727\n",
      "    values: 0.4572271345067197\n",
      "    values: 0.021315267730505072\n",
      "    values: 0.017827547329430593\n",
      "    values: 0.7679243031413665\n",
      "    values: 0.6382635927256938\n",
      "    values: 0.709855081228824\n",
      "    values: 0.18641111389863896\n",
      "    values: 0.22947685506453952\n",
      "    values: 0.7576749231406072\n",
      "    values: 0.4052832334298716\n",
      "    values: 0.6909268697149742\n",
      "    values: 0.31497902041543757\n",
      "    values: 0.17560783037661254\n",
      "    values: 0.533001739414932\n",
      "    values: 0.08885239188240035\n",
      "    values: 0.10196178451245275\n",
      "    values: 0.28403581345724593\n",
      "    values: 0.47649630254741404\n",
      "    values: 0.5780980054017543\n",
      "    values: 0.6289215819423972\n",
      "    values: 0.5785415542377883\n",
      "    values: 0.44028176009962505\n",
      "    values: 0.037712140717157006\n",
      "    values: 0.9903521479933168\n",
      "    values: 0.5087110198613572\n",
      "    values: 0.8416513744720527\n",
      "    values: 0.7935273081514088\n",
      "    values: 0.316696757854377\n",
      "    values: 0.25618154881202204\n",
      "    values: 0.49970502444979314\n",
      "    values: 0.9750520266748518\n",
      "    values: 0.21154409314592793\n",
      "    values: 0.6820824940194924\n",
      "    values: 0.14458307275775528\n",
      "    values: 0.49871874981182696\n",
      "    values: 0.9101375182403431\n",
      "    values: 0.4566488231075235\n",
      "    values: 0.5190633313309723\n",
      "    values: 0.03761868698711612\n",
      "    values: 0.8959241076898997\n",
      "    values: 0.9956795641251495\n",
      "    values: 0.9124026885910052\n",
      "    values: 0.8771529521906841\n",
      "    values: 0.9746106339921615\n",
      "    values: 0.09050132695945712\n",
      "    values: 0.22592152552042621\n",
      "    values: 0.8351603677260705\n",
      "    values: 0.13152596819626416\n",
      "    values: 0.08222822010226938\n",
      "    values: 0.39627625715244386\n",
      "    values: 0.7455456523896848\n",
      "    values: 0.035379068839347494\n",
      "    values: 0.5351548252469137\n",
      "    values: 0.6160520773486174\n",
      "    values: 0.25895170379992405\n",
      "    values: 0.8047859700242561\n",
      "    values: 0.6133231682938037\n",
      "    values: 0.8293408626573565\n",
      "    values: 0.23571573511526933\n",
      "    values: 0.2400163020691628\n",
      "    values: 0.4659787365933422\n",
      "    values: 0.5162713562648295\n",
      "    values: 0.01544941105499864\n",
      "    values: 0.6980509488657671\n",
      "    values: 0.15839831056540865\n",
      "    values: 0.17692555786943653\n",
      "    values: 0.19493960399955046\n",
      "    values: 0.6258275742192267\n",
      "    values: 0.9834136394321094\n",
      "    values: 0.937758511910898\n",
      "    values: 0.7758392315924554\n",
      "    values: 0.30401012096778\n",
      "    values: 0.8091374292060113\n",
      "    values: 0.9014075875503795\n",
      "    values: 0.5215968586017095\n",
      "    values: 0.9765602588360804\n",
      "    values: 0.49927384461803603\n",
      "    values: 0.6526239780264143\n",
      "    values: 0.4185468883801359\n",
      "    values: 0.38887382347259225\n",
      "    values: 0.30807500760458484\n",
      "    values: 0.005068956303728167\n",
      "    values: 0.443686331502256\n",
      "    values: 0.3379990233075232\n",
      "    values: 0.9685275757426719\n",
      "    values: 0.9042808932150844\n",
      "    values: 0.16226525177610174\n",
      "    values: 0.3692737836496336\n",
      "    values: 0.4416839847934151\n",
      "    values: 0.7651496536985714\n",
      "    values: 0.8213035714568577\n",
      "    values: 0.033262250669029325\n",
      "  }\n",
      "}\n",
      "\n",
      "Response:\n",
      "{'data': {'names': ['t:0', 't:1', 't:2', 't:3', 't:4', 't:5', 't:6', 't:7', 't:8', 't:9'], 'tensor': {'shape': [1, 10], 'values': [7.59969411e-22, 5.43922634e-34, 0.414125204, 0.585804045, 3.04793821e-29, 7.07601721e-05, 1.3436987e-12, 1.51228816e-17, 2.97117952e-09, 3.69011927e-20]}}, 'meta': {'requestPath': {'mnist-model': 'seldonio/tfserving-proxy:1.18.0'}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"rest\", shape=(1, 784))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### gRPC Request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "id": "80a51896-e6cb-417f-8963-46e36ca9f96e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "{'meta': {}, 'data': {'tensor': {'shape': [1, 784], 'values': [0.36172822444251884, 0.4699208601130421, 0.04599955578582804, 0.34137886436611387, 0.40651901318089734, 0.11725893073124771, 0.08081417032125349, 0.6187537605124053, 0.6572519990872054, 0.22070654283129476, 0.14889592272489427, 0.6104922652730083, 0.10322906851201841, 0.27183310335921584, 0.5855684888514012, 0.42078354827751796, 0.672782225323545, 0.2211601879314843, 0.5768984691728823, 0.42885906901903037, 0.40884498929123125, 0.08394924428324924, 0.24106290579477363, 0.6005980762674409, 0.3923047694739721, 0.8139942032757727, 0.6311803028136543, 0.4017185415933836, 0.6317912433120589, 0.8611012609994128, 0.2575202918955518, 0.9212772313103703, 0.4970618587075495, 0.22865387697765416, 0.4863167559295244, 0.9278355712441783, 0.9985559720177586, 0.7497805206325209, 0.2175860779635055, 0.2512634254597744, 0.9202650545933119, 0.46307333534983375, 0.911638722228198, 0.10659921520125704, 0.7065009093400467, 0.21393703737481884, 0.23544369648563712, 0.04763143531310843, 0.33148046643489193, 0.9570226964010952, 0.029099045305251647, 0.03738442865438951, 0.23575487637626347, 0.051413536135447524, 0.5927533526661493, 0.4077456693428043, 0.9657847947662764, 0.6894412555790713, 0.8298825864592176, 0.23209967395724895, 0.04894910268377517, 0.24911449696463905, 0.35025428559729055, 0.3210785258552953, 0.25188424911556073, 0.6214430769553738, 0.8918294656861748, 0.14083496959371444, 0.9037711777796075, 0.19874132515136256, 0.04002312798305829, 0.5036141546871115, 0.1547324112321683, 0.2701211153191131, 0.6762711217517801, 0.9092341266692568, 0.28183117134794444, 0.42715079757397845, 0.4931476335384225, 0.11640370088721408, 0.8378652827895045, 0.6723328499818876, 0.36491868464622534, 0.4658626742953912, 0.553191028093085, 0.6711799551640173, 0.22003488852281983, 0.23797382410492174, 0.9886983315383041, 0.5333168793794157, 0.05011542170278149, 0.6573869266081146, 0.23058722314161706, 0.17610486052386076, 0.0017029854098656871, 0.3809875641504479, 0.15487573838305713, 0.9681279055140405, 0.7937132876125764, 0.7731697342063326, 0.35224069101576705, 0.33378014111819443, 0.6380160614548303, 0.369616485215967, 0.17342566047971564, 0.1110222248915873, 0.7638350285547162, 0.18920600529077092, 0.13664681223880049, 0.368472611630176, 0.8680508691416866, 0.8015524553475534, 0.03513240032985976, 0.8685825973826709, 0.3755927580322652, 0.5132133849810869, 0.7180276972483199, 0.3795832946622183, 0.6095944043385754, 0.714062344190817, 0.17645832417380736, 0.003817765141316709, 0.6713681240529326, 0.027265993190543236, 0.2582361440333826, 0.123239134004822, 0.26004413128089565, 0.6358006283681316, 0.2214610229646592, 0.8731669051276718, 0.7560781731379562, 0.5970044076759972, 0.5776659854155435, 0.3172453941539888, 0.05014425900154673, 0.7971601247265422, 0.016950910821909915, 0.4541873916140997, 0.22826836843130738, 0.9466524049563818, 0.15047007983752336, 0.038187720711298945, 0.7859658299353078, 0.6137380779510436, 0.9504043532759825, 0.5807343943548439, 0.5343780996004118, 0.8285473359218566, 0.4275048777417326, 0.3839646867709884, 0.3810736029970343, 0.7974471807343236, 0.6242735570296196, 0.00971878886776778, 0.5581922135075682, 0.165003148980851, 0.6344671937113595, 0.5580509523542617, 0.7112867380874452, 0.11398835347179481, 0.7636441810106097, 0.9126315668027544, 0.38649411664331457, 0.4278748475538925, 0.41814584937543187, 0.7178709356920981, 0.7773153569313319, 0.8222237278195554, 0.31664510816819125, 0.7664068945052029, 0.7608344862932269, 0.3978418582694502, 0.8721658464609401, 0.1901974472233049, 0.3142831317719974, 0.16930520286795192, 0.028849289227044328, 0.037390216128801335, 0.46181680672586345, 0.40399432844086924, 0.919592367938116, 0.8312484039242999, 0.5994427931604619, 0.5181677917587025, 0.8245009147097506, 0.2162743925743349, 0.49840058263957865, 0.6316456225389769, 0.1160402940126064, 0.02602401524886655, 0.7554512212044698, 0.7376368018177055, 0.40580211758828133, 0.34388623635226845, 0.8146066260265905, 0.7119681683350676, 0.1040364317859398, 0.6408268904196793, 0.6713909390755525, 0.12026041256113884, 0.08503614029634554, 0.8622927549632072, 0.6155371377792849, 0.0163917067307201, 0.25865838870222535, 0.028954582824555763, 0.5955277148193534, 0.9632329837662499, 0.6470832170458125, 0.2171294095433295, 0.6699718255271659, 0.03937910138117828, 0.01593847359320244, 0.011386005357569395, 0.0021949778334018344, 0.8110601598509197, 0.8570134922859485, 0.6755054395104271, 0.21348205503125584, 0.221674840574153, 0.1001815027921319, 0.1590459345123697, 0.9306865103658774, 0.5625490137238318, 0.32909674171166947, 0.48245615796166985, 0.29518820837971305, 0.677639420617533, 0.9466928571641985, 0.7639407374530865, 0.9872720889667421, 0.41215970076070074, 0.8730648593350384, 0.1876832016849842, 0.635077948190888, 0.8169513049189893, 0.7902270500193581, 0.052818569319401654, 0.8873326754291323, 0.8057775805191949, 0.676449162442058, 0.8554561297827931, 0.180054638857865, 0.700105282220434, 0.20541093361386054, 0.4434799627758741, 0.6038336124702441, 0.5176476499915855, 0.48002138707542985, 0.10569080685836396, 0.05347188238240741, 0.6457583580389301, 0.46735511724859824, 0.016178550918255707, 0.30570601873154457, 0.9779955435735376, 0.1099451855203939, 0.032187942347736476, 0.09729687155129207, 0.6921144830915181, 0.13232565643546979, 0.45542955583648004, 0.07912590528826391, 0.5754788937669151, 0.8990541273272563, 0.50422286067046, 0.964932582280955, 0.21284444713540718, 0.6681766827111973, 0.8255006421136156, 0.16905920625504256, 0.9227334300918478, 0.9524913052148344, 0.3849200699535986, 0.8562712849626998, 0.2759527893657805, 0.3900195813884323, 0.8493654156681852, 0.48455075503642786, 0.010966107491165111, 0.221609157944615, 0.8242147544597527, 0.6539414941341996, 0.46850767052334974, 0.03295784546560898, 0.5969874645058812, 0.7991493164045156, 0.3317323277767359, 0.00952470597082078, 0.42452987310666324, 0.215017573692042, 0.9671197256298467, 0.5272275093125877, 0.5820122753184706, 0.28492736736636826, 0.6409045238536697, 0.23648123532981236, 0.9967641128432538, 0.3285365854966682, 0.7282305218876363, 0.4365008838978639, 0.2262544603553388, 0.18438485116943815, 0.81517454564111, 0.670765511344401, 0.287560588438839, 0.12512387294112237, 0.7168328568366962, 0.7989748576921452, 0.8371656558827189, 0.4461854282423453, 0.016220102314146834, 0.9904045202696792, 0.6774446693011152, 0.5515115613709252, 0.46206501101227826, 0.38542396738593154, 0.7786017732469093, 0.5498091766075733, 0.7313162423975503, 0.4180560275082045, 0.9035358215338767, 0.7490241926523257, 0.513780343152126, 0.4489836337923335, 0.39133619168688194, 0.27274950229284867, 0.17626909192710227, 0.8924252461383168, 0.49409914576754377, 0.5345116670989836, 0.7401857846781463, 0.25207284025506993, 0.182523412241404, 0.4968633367319527, 0.6601263521401585, 0.30753533301842917, 0.25272917606935874, 0.29980513258940067, 0.6732569921902026, 0.722081657854739, 0.49709427032128684, 0.2155902441195805, 0.04600485498745832, 0.5714267999178096, 0.7185559472528144, 0.23133483467513927, 0.4202290687955842, 0.9219503801336316, 0.0871241580964861, 0.4875751041176749, 0.4315411360799686, 0.5089938526460501, 0.07489955594032449, 0.44439875635382464, 0.3987576070475104, 0.05702753215844936, 0.3120611897693133, 0.2799182073165809, 0.10770920520997129, 0.8822989822403345, 0.5272719446038057, 0.6268964110536602, 0.3617123473191003, 0.4168993184173069, 0.8512341964854155, 0.24349409716866433, 0.48630293983557216, 0.6164463436828407, 0.06719774634575926, 0.8005282820261004, 0.6758246279243451, 0.6947374442370678, 0.14429375464536642, 0.3338738924296306, 0.05549157377925684, 0.6481998758367093, 0.40964105180478816, 0.1289216965693757, 0.3819577233941325, 0.9376761532992717, 0.2175709769270433, 0.5691163132968087, 0.3323166678751658, 0.7773021896971989, 0.018180916766491584, 0.3722535435680677, 0.3481026294876146, 0.23880384484734507, 0.733614265717547, 0.24279288307094393, 0.3965637252759765, 0.2201337753095326, 0.4471118789507452, 0.5415772926249419, 0.04640160539004523, 0.4679767609016281, 0.7762113871038807, 0.767622335988761, 0.6248144420421096, 0.0757119956392387, 0.33851236817404207, 0.7919861625320312, 0.2759819136018802, 0.38073137451412475, 0.2114668134433737, 0.2669446439716967, 0.5957750915791805, 0.14901192614861214, 0.9624125030489701, 0.6492635310115435, 0.7259204110876233, 0.4723974569525097, 0.5352481319483444, 0.6638508802850236, 0.9455972354799081, 0.46370177878557917, 0.18534766878864617, 0.8417908207490746, 0.1822897082119186, 0.6094381838605868, 0.9815449232406227, 0.6068173630144823, 0.9339302164145744, 0.4966189163275848, 0.880269950242075, 0.6804751459264525, 0.33012306112464296, 0.27273658488228225, 0.5260032957321898, 0.8272611470004578, 0.465390099476221, 0.9798296935876867, 0.87205474127178, 0.8371079847043617, 0.8600555541116339, 0.14310596501907746, 0.4570340047546829, 0.11254757166472673, 0.7293817420556507, 0.8734991003213904, 0.7977729194031019, 0.022516587907774666, 0.5432931613378962, 0.25402034327745127, 0.3629023378350461, 0.7480295382887021, 0.6362184103776304, 0.7755734390027632, 0.5046453535874189, 0.2692866854037279, 0.9752523062622446, 0.599426177904707, 0.673407804871196, 0.31518414241791237, 0.23899599082793577, 0.8293157714681563, 0.9368685899271084, 0.11454047304602633, 0.5330388239545322, 0.509838105556933, 0.6973423144745339, 0.7339068597928691, 0.06514239665853061, 0.10742681430112433, 0.2754204182558214, 0.7901999947827151, 0.2475079135775009, 0.5573536326280896, 0.7713463525163728, 0.744584022959336, 0.44501826532292044, 0.3038046268390062, 0.024606165907325672, 0.7160323167789014, 0.4366346677327386, 0.8090948746512642, 0.5770366855305255, 0.4303613935334716, 0.5839778034567215, 0.5524211173905371, 0.03154232266062773, 0.5124929072269153, 0.4533573107113379, 0.004179309517334806, 0.29238677129550217, 0.843590384267741, 0.4514742164442478, 0.47398598735335506, 0.6709344995247073, 0.7469226064295367, 0.8409405235192349, 0.8911099341370381, 0.367922039335859, 0.4560500517582783, 0.17320436581088705, 0.9136652576308856, 0.18941037202222932, 0.41758839569331796, 0.818976132596428, 0.777181705356038, 0.17956647787144087, 0.6851057842851747, 0.8656971963992834, 0.46713236987191775, 0.1969133816544293, 0.1153712779967897, 0.6479089005941221, 0.22841091746256437, 0.34835990641789694, 0.8268224498484742, 0.837101752985444, 0.35787056332052625, 0.8194304097323828, 0.508740591647388, 0.9084758298239752, 0.05187436378921162, 0.15348447978502655, 0.1367973791270064, 0.8263912806339319, 0.9445503872580792, 0.7226982213475992, 0.9078856251092489, 0.5356297224429316, 0.9024261522673223, 0.9477204095425602, 0.05135119913302799, 0.071287603842903, 0.16147744966735755, 0.9858466844457068, 0.3227666976208563, 0.9637406870971075, 0.060842668225130625, 0.9159489325806027, 0.820857057349457, 0.26716198195910046, 0.9704892606229839, 0.01631620036901882, 0.4254302863916116, 0.0012357482848420753, 0.35870927549571263, 0.4920511197840505, 0.8440210513651434, 0.9942543334719585, 0.8051992439530637, 0.7277071724192206, 0.22738473988876962, 0.9937476584942749, 0.38668157059577757, 0.010553993712293375, 0.7306155278611159, 0.4096914302017377, 0.18038583073717207, 0.04300853283640538, 0.27147382531726894, 0.538711947268488, 0.694376653219236, 0.9130270974261688, 0.6960424462398794, 0.9241026342450148, 0.20826634299076707, 0.07417635669252454, 0.4967971232358235, 0.21026412213601242, 0.1988117569534178, 0.37945200935010437, 0.6192278727118555, 0.10470273298301536, 0.4152189262157976, 0.6763621605519803, 0.8000805836878234, 0.7818285462335886, 0.04783543884651276, 0.14288348671189954, 0.7065028507599759, 0.9343068339153886, 0.8037658509200442, 0.26320949409373706, 0.6472272533125383, 0.18756030256799772, 0.9383000394782798, 0.721221548777633, 0.33132273298991144, 0.23920701954397827, 0.32746247600235223, 0.9183905590776535, 0.43907354058729375, 0.3403024674321262, 0.7026286725486899, 0.23628617069977542, 0.9068898296672114, 0.1252887120751054, 0.6757961950400322, 0.05203870520031384, 0.3032729881552577, 0.893071501757929, 0.5320707365613705, 0.3292529704822045, 0.9754645513550672, 0.1301012180696588, 0.9807534753189012, 0.06784164302212903, 0.712520362613039, 0.04405024418648429, 0.3899052833359733, 0.7046375289926156, 0.042065534547056194, 0.4844731793523469, 0.5554940951392807, 0.9537340470865582, 0.08511243318497952, 0.8753664778931017, 0.00736706514327834, 0.859259256742231, 0.7889082881724792, 0.13832417312906353, 0.4705314259181852, 0.16589656762755522, 0.7948403546460092, 0.5061953631612541, 0.9638514490798734, 0.3791858223222698, 0.16956008204465323, 0.4585852002226971, 0.6387660747743582, 0.2780332528935979, 0.2234661105102861, 0.7941072995479508, 0.4759425934483301, 0.8790249027799297, 0.31510770682371847, 0.2550831378102911, 0.027809048339305753, 0.009009002815855593, 0.5637305850425558, 0.003467245196914992, 0.4905683184857107, 0.7048920080232052, 0.8851649314326937, 0.6994497606782164, 0.6083633349667298, 0.9802749571823776, 0.8558290330634459, 0.42996174973223966, 0.17623182701502116, 0.26906937456331803, 0.6371898320903062, 0.9794912476384843, 0.22808477245342396, 0.10858324170953848, 0.7398622432232416, 0.6062665691702482, 0.20585148098516282, 0.2080539213516488, 0.9011332822136765, 0.35739539756255767, 0.2305698164006309, 0.02827091824975947, 0.6433451558581672, 0.39981410225959657, 0.5037549648597693, 0.03146299365652638, 0.7723121612370175, 0.7648955075923967, 0.3564612792945471, 0.6781058092152946, 0.11276187757058309, 0.7349814250145641, 0.7819197645972432, 0.8294261218556405, 0.1494415793140884, 0.7545724516422709, 0.3321866115899804, 0.10670047539712291, 0.5022879639259178, 0.1628006046463828, 0.29739831438499453, 0.05950268453460705, 0.2464917886197725, 0.8148807187821981, 0.4690447071246139, 0.2949095281526698, 0.7416230380595642, 0.005940578395413398, 0.4657636296423261, 0.22096331053476592, 0.694911199746227, 0.11030565102268486, 0.1604059397829568, 0.1967968433837487, 0.8143757483895765, 0.7574818080341754, 0.9538634833257315, 0.9826579126047408, 0.5955880403975019, 0.7972569716188557, 0.9443329497825466, 0.9462284339037835, 0.9089310522468065, 0.3943161425587247, 0.6552510921667604, 0.11107108184317538, 0.8284805211122891, 0.9816716829408518, 0.14067118553190805, 0.7188689411375162, 0.3781486815078553, 0.5862083765446148, 0.22962420682484852, 0.18745568534104218, 0.491700879948591, 0.36663737165466015, 0.6368353807632656, 0.9569602524196934, 0.7662018152718745, 0.8014941033116278, 0.8479271299599976, 0.5777586126147234, 0.6612690358680527, 0.5893925089042976, 0.5706828976817035, 0.4097568708453512, 0.8374657307884478, 0.051915814879910815, 0.5717604147561465, 0.8281736096390033, 0.6820977994060075, 0.8612875981688402, 0.017328877021735578, 0.5156209955398405, 0.44574743854917287, 0.616642868895055, 0.35785650731738894, 0.5720619959487966, 0.13959897863528714, 0.8803954435107009, 0.7326605981010983, 0.6719095549106356, 0.9924842720850703, 0.5256994568094662, 0.855540663997085, 0.48752210418606445, 0.20399154573630474, 0.22210206719933834, 0.08598069172071454, 0.9317430977066767, 0.3340322951924174, 0.9700725811598967, 0.8712027584095137, 0.9215887823413008, 0.11963766645587781, 0.3545338282367604, 0.3768584616518422, 0.25274336790611585, 0.42797036523360976, 0.46903868452022257, 0.7887774884730114, 0.20703188248944093, 0.02148408912083799, 0.33078507700961557, 0.9062172200642712, 0.0853297344012387, 0.6764361708963051, 0.7839276159847391, 0.0383403194391424, 0.8663691706547855, 0.2674448156304343, 0.6516343016908693, 0.5108898718355328, 0.4299453699747384, 0.5407619404085043, 0.7628579525808299, 0.2129035029296943, 0.2172721695416101, 0.6216465164695023, 0.348777988195458, 0.8339409116853125, 0.40458851521703354, 0.04968991766666253, 0.4475062776140257, 0.7304953412093921, 0.33601192720028106, 0.5446406078462704, 0.19544744163054317]}}}\n",
      "Response:\n",
      "{'data': {'tftensor': {'dtype': 'DT_FLOAT', 'tensorShape': {'dim': [{'size': '1'}, {'size': '10'}]}, 'floatVal': [1.0785601e-24, 6.920695e-34, 0.9999403, 5.9354432e-05, 4.8367345e-34, 3.337407e-07, 4.5185787e-18, 1.992394e-21, 7.845255e-12, 5.545996e-20]}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"grpc\", shape=(1, 784))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And delete the model we deployed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "id": "b011ff3c-1940-44f6-b8e4-6bb3c60ee290"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"tfserving\" deleted\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f ./resources/mnist_rest.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Serve Tensorflow Model with Tensorflow protocol\n",
    "\n",
    "The config file below shows how you can deploy your Tensorflow model which\n",
    "exposes the Tensorflow protocol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "id": "87c69588-f339-460d-b8a3-34bbddb66852"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./resources/halfplustwo_rest.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./resources/halfplustwo_rest.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: hpt\n",
    "spec:\n",
    "  name: hpt\n",
    "  protocol: tensorflow\n",
    "  transport: rest\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: TENSORFLOW_SERVER\n",
    "      modelUri: gs://seldon-models/tfserving/half_plus_two\n",
    "      name:  halfplustwo\n",
    "      parameters:\n",
    "        - name: model_name\n",
    "          type: STRING\n",
    "          value: halfplustwo\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "id": "7e6a7f1a-0114-44c7-85e2-e32114ced15f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/hpt created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f ./resources/halfplustwo_rest.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "id": "1f6e5870-7f29-447c-b64a-6e41ea6affe6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"hpt-default-0-halfplustwo\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"hpt-default-0-halfplustwo\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=hpt -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "id": "828e7eb3-a4c1-49e9-80ab-8e29d5c88782"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'predictions': [2.5, 3.0, 4.5]}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "X=!curl -s -d '{\"instances\": [1.0, 2.0, 5.0]}' \\\n",
    "   -X POST http://localhost:8004/seldon/seldon/hpt/v1/models/halfplustwo/:predict \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "d=json.loads(\"\".join(X))\n",
    "print(d)\n",
    "assert(d[\"predictions\"][0] == 2.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "id": "d6ab5245-271f-4fa1-b55c-fab43f98da20"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'outputs': {'x': {'dtype': 'DT_FLOAT', 'tensorShape': {'dim': [{'size': '3'}]}, 'floatVal': [2.5, 3, 3.5]}}, 'modelSpec': {'name': 'halfplustwo', 'version': '123', 'signatureName': 'serving_default'}}\n"
     ]
    }
   ],
   "source": [
    "X=!cd ../executor/proto && grpcurl \\\n",
    "   -d '{\"model_spec\":{\"name\":\"halfplustwo\"},\"inputs\":{\"x\":{\"dtype\": 1, \"tensor_shape\": {\"dim\":[{\"size\": 3}]}, \"floatVal\" : [1.0, 2.0, 3.0]}}}' \\\n",
    "   -rpc-header seldon:hpt -rpc-header namespace:seldon \\\n",
    "   -plaintext -proto ./prediction_service.proto \\\n",
    "   0.0.0.0:8004 tensorflow.serving.PredictionService/Predict\n",
    "d=json.loads(\"\".join(X))\n",
    "print(d)\n",
    "assert(d[\"outputs\"][\"x\"][\"floatVal\"][0] == 2.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "id": "63d6bd7c-2a55-4e3a-b8bc-5e47b6eb740a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"hpt\" deleted\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f ./resources/halfplustwo_rest.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serve MLFlow Elasticnet Wines Model\n",
    "\n",
    "In order to deploy MLflow models, we can leverage the [pre-packaged MLflow inference server](https://docs.seldon.io/projects/seldon-core/en/latest/servers/mlflow.html).\n",
    "The exposed API can follow either:\n",
    "\n",
    "- The default Seldon protocol. \n",
    "- The V2 protocol."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Default Seldon protocol\n",
    "\n",
    "We can deploy an MLFlow model uploaded to an object store by using the MLFlow model server implementation as the config below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "id": "404d46bf-d0dd-4eff-8018-bfd88a978813"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./resources/elasticnet_wine.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./resources/elasticnet_wine.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: mlflow\n",
    "spec:\n",
    "  name: wines\n",
    "  predictors:\n",
    "  - componentSpecs:\n",
    "    - spec:\n",
    "        # We are setting high failureThreshold as installing conda dependencies\n",
    "        # can take long time and we want to avoid k8s killing the container prematurely\n",
    "        containers:\n",
    "        - name: classifier\n",
    "          livenessProbe:\n",
    "            initialDelaySeconds: 80\n",
    "            failureThreshold: 200\n",
    "            periodSeconds: 5\n",
    "            successThreshold: 1\n",
    "            httpGet:\n",
    "              path: /health/ping\n",
    "              port: http\n",
    "              scheme: HTTP\n",
    "          readinessProbe:\n",
    "            initialDelaySeconds: 80\n",
    "            failureThreshold: 200\n",
    "            periodSeconds: 5\n",
    "            successThreshold: 1\n",
    "            httpGet:\n",
    "              path: /health/ping\n",
    "              port: http\n",
    "              scheme: HTTP\n",
    "    graph:\n",
    "      children: []\n",
    "      implementation: MLFLOW_SERVER\n",
    "      modelUri: gs://seldon-models/v1.18.0/mlflow/elasticnet_wine\n",
    "      name: classifier\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "id": "1b1684bc-fd47-44be-8651-8bcce6182e51"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/mlflow created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f ./resources/elasticnet_wine.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "id": "f51824e7-7928-4343-9a55-82ae2835cde7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"mlflow-default-0-classifier\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"mlflow-default-0-classifier\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=mlflow -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### REST requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "id": "bb9cb6c7-da6f-444f-87b3-b17fe55211cf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': {'names': [], 'ndarray': [5.275558760255375]}, 'meta': {'requestPath': {'classifier': 'seldonio/mlflowserver:1.18.0'}}}\n"
     ]
    }
   ],
   "source": [
    "X=!curl -s -d '{\"data\": {\"ndarray\":[[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0,1.1]]}}' \\\n",
    "   -X POST http://localhost:8004/seldon/seldon/mlflow/api/v1.0/predictions \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "d=json.loads(X[0])\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "id": "be48da39-e33a-474b-bc73-da989dacf806"
   },
   "outputs": [],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "\n",
    "sc = SeldonClient(deployment_name=\"mlflow\", namespace=\"seldon\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "id": "59002292-fa5b-4f65-ad49-a11508c3972a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "meta {\n",
      "}\n",
      "data {\n",
      "  tensor {\n",
      "    shape: 1\n",
      "    shape: 11\n",
      "    values: 0.5653756616147143\n",
      "    values: 0.8420723980045103\n",
      "    values: 0.5062962018951862\n",
      "    values: 0.5278571532571091\n",
      "    values: 0.8664864216170011\n",
      "    values: 0.45973619467487525\n",
      "    values: 0.9263453669947618\n",
      "    values: 0.6150362295808135\n",
      "    values: 0.5089439556920178\n",
      "    values: 0.9282257119213217\n",
      "    values: 0.7953848591020709\n",
      "  }\n",
      "}\n",
      "\n",
      "Response:\n",
      "{'data': {'names': [], 'tensor': {'shape': [1], 'values': [5.244154959197477]}}, 'meta': {'requestPath': {'classifier': 'seldonio/mlflowserver:1.18.0'}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"rest\", shape=(1, 11))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### gRPC Requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'meta': {'requestPath': {'classifier': 'seldonio/mlflowserver:1.18.0'}}, 'data': {'ndarray': [5.275558760255375]}}\n"
     ]
    }
   ],
   "source": [
    "X=!cd ../executor/proto && grpcurl -d '{\"data\":{\"ndarray\":[[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0,1.1]]}}' \\\n",
    "         -rpc-header seldon:mlflow -rpc-header namespace:seldon \\\n",
    "         -plaintext \\\n",
    "         -proto ./prediction.proto  0.0.0.0:8004 seldon.protos.Seldon/Predict\n",
    "d=json.loads(\"\".join(X))\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "id": "e9685ef6-56b0-4604-a3af-233b9e0e9637"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "{'meta': {}, 'data': {'tensor': {'shape': [1, 11], 'values': [0.49331471179570063, 0.6224359102414585, 0.9804140644783785, 0.3380112388366434, 0.8942524358731484, 0.2837043142995588, 0.9867737754039229, 0.47015800689221765, 0.5398691845908773, 0.6674452878134565, 0.4007242201724862]}}}\n",
      "Response:\n",
      "{'meta': {'requestPath': {'classifier': 'seldonio/mlflowserver:1.18.0'}}, 'data': {'tensor': {'shape': [1], 'values': [5.204512993729161]}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"grpc\", shape=(1, 11))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "id": "b061cdba-9607-4c5b-a694-429cb616aca1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"mlflow\" deleted\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f ./resources/elasticnet_wine.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### V2 protocol\n",
    "\n",
    "We can deploy a MLflow model, exposing an API compatible with v2 protocol by specifying the `protocol` of our `SeldonDeployment` as `v2`.\n",
    "For example, we can consider the config below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "id": "f857d6fa-c769-442c-b26a-29aaf12a8588"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./resources/elasticnet_wine_v2.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./resources/elasticnet_wine_v2.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: mlflow\n",
    "spec:\n",
    "  protocol: v2  # Activate v2 protocol\n",
    "  name: wines\n",
    "  predictors:\n",
    "    - graph:\n",
    "        children: []\n",
    "        implementation: MLFLOW_SERVER\n",
    "        modelUri: gs://seldon-models/v1.12.0-dev/mlflow/elasticnet_wine\n",
    "        name: classifier\n",
    "      name: default\n",
    "      replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "id": "3fed1409-b072-4fca-b974-3592b5257915"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/mlflow created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f ./resources/elasticnet_wine_v2.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "id": "b6a0d511-48ca-422d-b489-cf6745401b29"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"mlflow-default-0-classifier\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"mlflow-default-0-classifier\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=mlflow -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's deployed, we can send inference requests to our model.\n",
    "Note that, since it's using the V2 Protocol, these requests will be different to the ones using the default Seldon Protocol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "id": "8f8f7612-0806-463d-b089-d2dd21c10266"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"model_name\": \"classifier\",\n",
      "  \"model_version\": \"v1\",\n",
      "  \"id\": \"732f2602-cc8a-479d-86c8-ee99bafd1a1e\",\n",
      "  \"parameters\": null,\n",
      "  \"outputs\": [\n",
      "    {\n",
      "      \"name\": \"predict\",\n",
      "      \"shape\": [\n",
      "        1\n",
      "      ],\n",
      "      \"datatype\": \"FP64\",\n",
      "      \"parameters\": null,\n",
      "      \"data\": [\n",
      "        6.016145744177844\n",
      "      ]\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "import requests\n",
    "\n",
    "inference_request = {\n",
    "    \"parameters\": {\"content_type\": \"pd\"},\n",
    "    \"inputs\": [\n",
    "        {\n",
    "            \"name\": \"fixed acidity\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [7.4],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"volatile acidity\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [0.7000],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"citric acidity\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [0],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"residual sugar\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [1.9],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"chlorides\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [0.076],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"free sulfur dioxide\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [11],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"total sulfur dioxide\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [34],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"density\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [0.9978],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"pH\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [3.51],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"sulphates\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [0.56],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"alcohol\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [9.4],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "    ],\n",
    "}\n",
    "\n",
    "endpoint = \"http://localhost:8004/seldon/seldon/mlflow/v2/models/infer\"\n",
    "response = requests.post(endpoint, json=inference_request)\n",
    "\n",
    "print(json.dumps(response.json(), indent=2))\n",
    "assert response.ok"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "id": "c80971d9-e533-461f-abd8-0191135d8207"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"mlflow\" deleted\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f ./resources/elasticnet_wine_v2.yaml"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "seldon-core",
   "language": "python",
   "name": "seldon-core"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}