notebooks/protocol_examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Examples with Different Protocols\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    " * A kubernetes cluster with kubectl configured\n",
    " * curl\n",
    " * grpcurl\n",
    " * pygmentize\n",
    " \n",
    "## Examples\n",
    "\n",
    "  * [Seldon Protocol](#Seldon-Protocol-Model)\n",
    "  * [Tensorflow Protocol](#Tensorflow-Protocol-Model)\n",
    "  * [KFServing V2 Protocol](#KFServing-V2-Protocol-Model)\n",
    " \n",
    "\n",
    "## Setup Seldon Core\n",
    "\n",
    "Use the setup notebook to [Setup Cluster](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html) to setup Seldon Core with an ingress - either Ambassador or Istio.\n",
    "\n",
    "Then port-forward to that ingress on localhost:8003 in a separate terminal either with:\n",
    "\n",
    " * Ambassador: `kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080`\n",
    " * Istio: `kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8003:80`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "namespace/seldon created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create namespace seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context \"kind-kind\" modified.\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl config set-context $(kubectl config current-context) --namespace=seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seldon Protocol Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will deploy a REST model that uses the SELDON Protocol namely by specifying the attribute `protocol: seldon`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting resources/model_seldon.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile resources/model_seldon.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: example-seldon\n",
    "spec:\n",
    "  protocol: seldon\n",
    "  predictors:\n",
    "  - componentSpecs:\n",
    "    - spec:\n",
    "        containers:\n",
    "        - image: seldonio/mock_classifier:1.3.0-dev\n",
    "          name: classifier\n",
    "    graph:\n",
    "      name: classifier\n",
    "      type: MODEL\n",
    "    name: model\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/example-seldon created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f resources/model_seldon.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deployment \"example-seldon-model-0-classifier\" successfully rolled out\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example-seldon -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Available\n"
     ]
    }
   ],
   "source": [
    "for i in range(60):\n",
    "    state=!kubectl get sdep example-seldon -o jsonpath='{.status.state}'\n",
    "    state=state[0]\n",
    "    print(state)\n",
    "    if state==\"Available\":\n",
    "        break\n",
    "    time.sleep(1)\n",
    "assert(state==\"Available\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': {'names': ['proba'], 'ndarray': [[0.43782349911420193]]}, 'meta': {}}\n"
     ]
    }
   ],
   "source": [
    "X=!curl -s -d '{\"data\": {\"ndarray\":[[1.0, 2.0, 5.0]]}}' \\\n",
    "   -X POST http://localhost:8003/seldon/seldon/example-seldon/api/v1.0/predictions \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "d=json.loads(X[0])\n",
    "print(d)\n",
    "assert(d[\"data\"][\"ndarray\"][0][0] > 0.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'meta': {}, 'data': {'names': ['proba'], 'ndarray': [[0.43782349911420193]]}}\n"
     ]
    }
   ],
   "source": [
    "X=!cd ../executor/proto && grpcurl -d '{\"data\":{\"ndarray\":[[1.0,2.0,5.0]]}}' \\\n",
    "         -rpc-header seldon:example-seldon -rpc-header namespace:seldon \\\n",
    "         -plaintext \\\n",
    "         -proto ./prediction.proto  0.0.0.0:8003 seldon.protos.Seldon/Predict\n",
    "d=json.loads(\"\".join(X))\n",
    "print(d)\n",
    "assert(d[\"data\"][\"ndarray\"][0][0] > 0.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"example-seldon\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f resources/model_seldon.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensorflow Protocol Model\n",
    "We will deploy a model that uses the TENSORLFOW Protocol namely by specifying the attribute `protocol: tensorflow`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting resources/model_tfserving.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile resources/model_tfserving.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: example-tfserving\n",
    "spec:\n",
    "  protocol: tensorflow\n",
    "  predictors:\n",
    "  - componentSpecs:\n",
    "    - spec:\n",
    "        containers:\n",
    "        - args: \n",
    "          - --port=8500\n",
    "          - --rest_api_port=8501\n",
    "          - --model_name=halfplustwo\n",
    "          - --model_base_path=gs://seldon-models/tfserving/half_plus_two\n",
    "          image: tensorflow/serving\n",
    "          name: halfplustwo\n",
    "          ports:\n",
    "          - containerPort: 8501\n",
    "            name: http\n",
    "            protocol: TCP\n",
    "          - containerPort: 8500\n",
    "            name: grpc\n",
    "            protocol: TCP\n",
    "    graph:\n",
    "      name: halfplustwo\n",
    "      type: MODEL\n",
    "      endpoint:\n",
    "        http_port: 8501\n",
    "        grpc_port: 8500\n",
    "    name: model\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/example-tfserving created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f resources/model_tfserving.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"example-tfserving-model-0-halfplustwo\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"example-tfserving-model-0-halfplustwo\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example-tfserving \\\n",
    "                                 -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Available\n"
     ]
    }
   ],
   "source": [
    "for i in range(60):\n",
    "    state=!kubectl get sdep example-tfserving -o jsonpath='{.status.state}'\n",
    "    state=state[0]\n",
    "    print(state)\n",
    "    if state==\"Available\":\n",
    "        break\n",
    "    time.sleep(1)\n",
    "assert(state==\"Available\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'predictions': [2.5, 3.0, 4.5]}\n"
     ]
    }
   ],
   "source": [
    "X=!curl -s -d '{\"instances\": [1.0, 2.0, 5.0]}' \\\n",
    "   -X POST http://localhost:8003/seldon/seldon/example-tfserving/v1/models/halfplustwo/:predict \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "d=json.loads(\"\".join(X))\n",
    "print(d)\n",
    "assert(d[\"predictions\"][0] == 2.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'outputs': {'x': {'dtype': 'DT_FLOAT', 'tensorShape': {'dim': [{'size': '3'}]}, 'floatVal': [2.5, 3, 3.5]}}, 'modelSpec': {'name': 'halfplustwo', 'version': '123', 'signatureName': 'serving_default'}}\n"
     ]
    }
   ],
   "source": [
    "X=!cd ../executor/proto && grpcurl \\\n",
    "   -d '{\"model_spec\":{\"name\":\"halfplustwo\"},\"inputs\":{\"x\":{\"dtype\": 1, \"tensor_shape\": {\"dim\":[{\"size\": 3}]}, \"floatVal\" : [1.0, 2.0, 3.0]}}}' \\\n",
    "   -rpc-header seldon:example-tfserving -rpc-header namespace:seldon \\\n",
    "   -plaintext -proto ./prediction_service.proto \\\n",
    "   0.0.0.0:8003 tensorflow.serving.PredictionService/Predict\n",
    "d=json.loads(\"\".join(X))\n",
    "print(d)\n",
    "assert(d[\"outputs\"][\"x\"][\"floatVal\"][0] == 2.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"example-tfserving\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f resources/model_tfserving.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## KFServing V2 Protocol Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will deploy a REST model that uses the KFServing V2 Protocol namely by specifying the attribute `protocol: kfserving`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing resources/model_v2.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile resources/model_v2.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: triton\n",
    "spec:\n",
    "  protocol: kfserving\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: TRITON_SERVER\n",
    "      modelUri: gs://seldon-models/trtis/simple-model\n",
    "      name: simple\n",
    "    name: simple\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/triton created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f resources/model_v2.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deployment \"triton-simple-0-simple\" successfully rolled out\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=triton -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Available\n"
     ]
    }
   ],
   "source": [
    "for i in range(60):\n",
    "    state=!kubectl get sdep triton -o jsonpath='{.status.state}'\n",
    "    state=state[0]\n",
    "    print(state)\n",
    "    if state==\"Available\":\n",
    "        break\n",
    "    time.sleep(1)\n",
    "assert(state==\"Available\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'model_name': 'simple', 'model_version': '1', 'outputs': [{'name': 'OUTPUT0', 'datatype': 'INT32', 'shape': [1, 16], 'data': [2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]}, {'name': 'OUTPUT1', 'datatype': 'INT32', 'shape': [1, 16], 'data': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}]}\n"
     ]
    }
   ],
   "source": [
    "X=!curl -s -d '{\"inputs\":[{\"name\":\"INPUT0\",\"data\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],\"datatype\":\"INT32\",\"shape\":[1,16]},{\"name\":\"INPUT1\",\"data\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],\"datatype\":\"INT32\",\"shape\":[1,16]}]}'  \\\n",
    "        -X POST http://0.0.0.0:8003/seldon/seldon/triton/v2/models/simple/infer \\\n",
    "        -H \"Content-Type: application/json\"\n",
    "d=json.loads(X[0])\n",
    "print(d)\n",
    "assert(d[\"outputs\"][0][\"data\"][0]==2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{  \"modelName\": \"simple\",  \"modelVersion\": \"1\",  \"outputs\": [    {      \"name\": \"OUTPUT0\",      \"datatype\": \"INT32\",      \"shape\": [        \"1\",        \"16\"      ]    },    {      \"name\": \"OUTPUT1\",      \"datatype\": \"INT32\",      \"shape\": [        \"1\",        \"16\"      ]    }  ],  \"rawOutputContents\": [    \"AgAAAAQAAAAGAAAACAAAAAoAAAAMAAAADgAAABAAAAASAAAAFAAAABYAAAAYAAAAGgAAABwAAAAeAAAAIAAAAA==\",    \"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==\"  ]}\n"
     ]
    }
   ],
   "source": [
    "X=!cd ../executor/api/grpc/kfserving/inference && \\\n",
    "        grpcurl -d '{\"model_name\":\"simple\",\"inputs\":[{\"name\":\"INPUT0\",\"contents\":{\"int_contents\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},\"datatype\":\"INT32\",\"shape\":[1,16]},{\"name\":\"INPUT1\",\"contents\":{\"int_contents\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},\"datatype\":\"INT32\",\"shape\":[1,16]}]}' \\\n",
    "        -plaintext -proto ./grpc_service.proto \\\n",
    "        -rpc-header seldon:triton -rpc-header namespace:seldon \\\n",
    "        0.0.0.0:8003 inference.GRPCInferenceService/ModelInfer\n",
    "X=\"\".join(X)\n",
    "print(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"triton\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f resources/model_v2.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}