examples/models/metrics/metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "18a9259d-8e0f-4a03-9884-d5e755fa0a93",
   "metadata": {},
   "source": [
    "# Basic Examples of Metrics with Prometheus Operator\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    " * A kubernetes cluster with kubectl configured\n",
    " * curl\n",
    " \n",
    "\n",
    "## Setup Seldon Core\n",
    "\n",
    "Install Seldon Core as described in [docs](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/install.html).\n",
    "\n",
    "Then port-forward to that ingress on localhost:8003 in a separate terminal either with:\n",
    "```bash\n",
    "kubectl port-forward -n istio-system svc/istio-ingressgateway 8003:80\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "daeea512-4a13-4c06-9e66-c09d3d125d32",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Error from server (AlreadyExists): namespaces \"seldon\" already exists\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Seldon namespace already exists\n",
      "Context \"kind-ansible\" modified.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "kubectl create namespace seldon || echo \"Seldon namespace already exists\"\n",
    "kubectl config set-context $(kubectl config current-context) --namespace=seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f3d861f-1b1c-4125-be2c-2323138fc86e",
   "metadata": {},
   "source": [
    "## Install Prometheus Operator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "64b71ab8-ba35-420c-8160-3c7bbff657c0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Error from server (AlreadyExists): namespaces \"seldon-monitoring\" already exists\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Release \"seldon-monitoring\" does not exist. Installing it now.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0509 15:22:58.165020  425776 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n",
      "W0509 15:22:58.166767  425776 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n",
      "W0509 15:22:58.168184  425776 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n",
      "W0509 15:22:58.169785  425776 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n",
      "W0509 15:22:58.171310  425776 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n",
      "W0509 15:22:58.487981  425776 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n",
      "W0509 15:22:58.489698  425776 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n",
      "W0509 15:22:58.489729  425776 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n",
      "W0509 15:22:58.489886  425776 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n",
      "W0509 15:22:58.489921  425776 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME: seldon-monitoring\n",
      "LAST DEPLOYED: Mon May  9 15:22:57 2022\n",
      "NAMESPACE: seldon-monitoring\n",
      "STATUS: deployed\n",
      "REVISION: 1\n",
      "TEST SUITE: None\n",
      "NOTES:\n",
      "CHART NAME: kube-prometheus\n",
      "CHART VERSION: 6.9.5\n",
      "APP VERSION: 0.55.1\n",
      "\n",
      "** Please be patient while the chart is being deployed **\n",
      "\n",
      "Watch the Prometheus Operator Deployment status using the command:\n",
      "\n",
      "    kubectl get deploy -w --namespace seldon-monitoring -l app.kubernetes.io/name=kube-prometheus-operator,app.kubernetes.io/instance=seldon-monitoring\n",
      "\n",
      "Watch the Prometheus StatefulSet status using the command:\n",
      "\n",
      "    kubectl get sts -w --namespace seldon-monitoring -l app.kubernetes.io/name=kube-prometheus-prometheus,app.kubernetes.io/instance=seldon-monitoring\n",
      "\n",
      "Prometheus can be accessed via port \"9090\" on the following DNS name from within your cluster:\n",
      "\n",
      "    seldon-monitoring-prometheus.seldon-monitoring.svc.cluster.local\n",
      "\n",
      "To access Prometheus from outside the cluster execute the following commands:\n",
      "\n",
      "    echo \"Prometheus URL: http://127.0.0.1:9090/\"\n",
      "    kubectl port-forward --namespace seldon-monitoring svc/seldon-monitoring-prometheus 9090:9090\n",
      "\n",
      "Watch the Alertmanager StatefulSet status using the command:\n",
      "\n",
      "    kubectl get sts -w --namespace seldon-monitoring -l app.kubernetes.io/name=kube-prometheus-alertmanager,app.kubernetes.io/instance=seldon-monitoring\n",
      "\n",
      "Alertmanager can be accessed via port \"9093\" on the following DNS name from within your cluster:\n",
      "\n",
      "    seldon-monitoring-alertmanager.seldon-monitoring.svc.cluster.local\n",
      "\n",
      "To access Alertmanager from outside the cluster execute the following commands:\n",
      "\n",
      "    echo \"Alertmanager URL: http://127.0.0.1:9093/\"\n",
      "    kubectl port-forward --namespace seldon-monitoring svc/seldon-monitoring-alertmanager 9093:9093\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "kubectl create namespace seldon-monitoring\n",
    "\n",
    "# Note: we set prometheus.scrapeInterval=1s for CI tests reliability here\n",
    "helm upgrade --install seldon-monitoring kube-prometheus \\\n",
    "    --version 6.9.5 \\\n",
    "    --set fullnameOverride=seldon-monitoring \\\n",
    "    --set prometheus.scrapeInterval=1s \\\n",
    "    --namespace seldon-monitoring \\\n",
    "    --repo https://charts.bitnami.com/bitnami"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0a808114-e0cf-4033-9374-bc9d8e1bb018",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"seldon-monitoring-operator\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"seldon-monitoring-operator\" successfully rolled out\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Error from server (NotFound): deployments.apps \"prometheus-kube-state-metrics\" not found\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "statefulset rolling update complete 1 pods at revision prometheus-seldon-monitoring-prometheus-5f84fbf5d4...\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Extra sleep as statefulset is not always present right away\n",
    "sleep 5 \n",
    "kubectl rollout status -n seldon-monitoring deployment/seldon-monitoring-operator\n",
    "kubectl rollout status -n seldon-monitoring deployment/prometheus-kube-state-metrics\n",
    "kubectl rollout status -n seldon-monitoring statefulsets/prometheus-seldon-monitoring-prometheus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "02d2c5c9-704c-48dc-b5e6-c1b7bf700384",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning: resource podmonitors/seldon-podmonitor is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "podmonitor.monitoring.coreos.com/seldon-podmonitor configured\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat <<EOF | kubectl apply -f -\n",
    "apiVersion: monitoring.coreos.com/v1\n",
    "kind: PodMonitor\n",
    "metadata:\n",
    "  name: seldon-podmonitor\n",
    "  namespace: seldon-monitoring\n",
    "spec:\n",
    "  selector:\n",
    "    matchLabels:\n",
    "      app.kubernetes.io/managed-by: seldon-core\n",
    "  podMetricsEndpoints:\n",
    "    - port: metrics\n",
    "      path: /prometheus\n",
    "  namespaceSelector:\n",
    "    any: true\n",
    "EOF"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9d0052c-ee64-44b0-a259-dc56e8287da2",
   "metadata": {},
   "source": [
    "## Deploy Example Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ef44380c-7a4e-4deb-ac3a-0dba88098c50",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting echo-sdep.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile echo-sdep.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: echo\n",
    "  namespace: seldon\n",
    "spec:\n",
    "  predictors:\n",
    "  - name: default\n",
    "    replicas: 1\n",
    "    graph:\n",
    "      name: classifier\n",
    "      type: MODEL\n",
    "    componentSpecs:\n",
    "    - spec:\n",
    "        containers:\n",
    "        - image: seldonio/echo-model:1.19.0-dev\n",
    "          name: classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "68615758-07a3-420e-977a-88d9962f6ae6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/echo created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f echo-sdep.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "15b44a10-2fd0-4ccf-89fc-f57561f8f66f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"echo-default-0-classifier\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"echo-default-0-classifier\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "deployment=$(kubectl get deploy -l seldon-deployment-id=echo -o jsonpath='{.items[0].metadata.name}')\n",
    "kubectl rollout status deploy/${deployment}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2fb60f9-6eea-46c6-9cd2-ad6e69ce0c6d",
   "metadata": {},
   "source": [
    "## Sent series of REST requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "65961c3c-2eba-4610-a751-1faee739f723",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting 5s for model to fully ready\n",
      "Waiting 10s for Prometheus to scrape metrics\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "# Wait for the model to become fully ready\n",
    "echo \"Waiting 5s for model to fully ready\"\n",
    "sleep 5\n",
    "\n",
    "# Send 20 requests to REST endpoint\n",
    "for i in `seq 1 10`; do sleep 0.1 && \\\n",
    "   curl -s -H \"Content-Type: application/json\" \\\n",
    "   -d '{\"data\": {\"ndarray\":[[1.0, 2.0, 5.0]]}}' \\\n",
    "   http://localhost:8003/seldon/seldon/echo/api/v1.0/predictions > /dev/null ; \\\n",
    "done\n",
    "\n",
    "# Give time for metrics to get collected by Prometheus\n",
    "echo \"Waiting 10s for Prometheus to scrape metrics\"\n",
    "sleep 10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7be3a3f-fa33-4942-afa7-0ccc13ac12cc",
   "metadata": {},
   "source": [
    "## Check Metrics (REST)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8f7658f3-6dc6-4d42-96c2-664a29dea9b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "856542d5-5d19-4954-bbcd-6ff9d55c07da",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting get-metrics.sh\n"
     ]
    }
   ],
   "source": [
    "%%writefile get-metrics.sh\n",
    "QUERY='query=seldon_api_executor_client_requests_seconds_count{deployment_name=~\"echo\",namespace=~\"seldon\",method=~\"post\"}'\n",
    "QUERY_URL=http://seldon-monitoring-prometheus.seldon-monitoring.svc.cluster.local:9090/api/v1/query\n",
    "\n",
    "kubectl run --quiet=true -it --rm curlmetrics-$(date +%s) --image=radial/busyboxplus:curl --restart=Never -- \\\n",
    "    curl --data-urlencode ${QUERY} ${QUERY_URL}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4d99a102-2edf-427a-9dd2-589a675ae3c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics = ! bash get-metrics.sh\n",
    "metrics = json.loads(metrics[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4482f74e-d9e4-4162-aff3-547096bc4f64",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'status': 'success',\n",
       " 'data': {'resultType': 'vector',\n",
       "  'result': [{'metric': {'__name__': 'seldon_api_executor_client_requests_seconds_count',\n",
       "     'code': '200',\n",
       "     'container': 'seldon-container-engine',\n",
       "     'deployment_name': 'echo',\n",
       "     'endpoint': 'metrics',\n",
       "     'instance': '10.244.0.43:8000',\n",
       "     'job': 'seldon-monitoring/seldon-podmonitor',\n",
       "     'method': 'post',\n",
       "     'model_image': 'seldonio/echo-model',\n",
       "     'model_name': 'classifier',\n",
       "     'model_version': '1.15.0-dev',\n",
       "     'namespace': 'seldon',\n",
       "     'pod': 'echo-default-0-classifier-6fcd878bc5-pzzsc',\n",
       "     'predictor_name': 'default',\n",
       "     'service': '/predict'},\n",
       "    'value': [1652106256.269, '10']},\n",
       "   {'metric': {'__name__': 'seldon_api_executor_client_requests_seconds_count',\n",
       "     'code': '200',\n",
       "     'container': 'seldon-container-engine',\n",
       "     'deployment_name': 'echo',\n",
       "     'endpoint': 'metrics',\n",
       "     'instance': '10.244.0.43:8000',\n",
       "     'job': 'seldon-system/seldon-podmonitor',\n",
       "     'method': 'post',\n",
       "     'model_image': 'seldonio/echo-model',\n",
       "     'model_name': 'classifier',\n",
       "     'model_version': '1.15.0-dev',\n",
       "     'namespace': 'seldon',\n",
       "     'pod': 'echo-default-0-classifier-6fcd878bc5-pzzsc',\n",
       "     'predictor_name': 'default',\n",
       "     'service': '/predict'},\n",
       "    'value': [1652106256.269, '10']}]}}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c6747dd2-b57a-4c06-ae24-f9d33d2d12d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "counter = int(metrics[\"data\"][\"result\"][0][\"value\"][1])\n",
    "assert counter == 10, f\"expected 10 requests, got {counter}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71d8bfa6-9d74-4e57-94fd-de77e5aa62ab",
   "metadata": {},
   "source": [
    "## Send series GRPC requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3837706f-e3d4-4769-8baf-28b5b93a27a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting 10s for Prometheus to scrape metrics\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cd ../../../executor/proto && for i in `seq 1 10`; do sleep 0.1 && \\\n",
    "    grpcurl -d '{\"data\": {\"ndarray\":[[1.0, 2.0, 5.0]]}}' \\\n",
    "    -rpc-header seldon:echo -rpc-header namespace:seldon \\\n",
    "    -plaintext -proto ./prediction.proto \\\n",
    "     0.0.0.0:8003 seldon.protos.Seldon/Predict > /dev/null ; \\\n",
    "done\n",
    "\n",
    "# Give time for metrics to get collected by Prometheus\n",
    "echo \"Waiting 10s for Prometheus to scrape metrics\"\n",
    "sleep 10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ab45941-b7bc-4185-bce7-65b2dfceebea",
   "metadata": {},
   "source": [
    "## Check metrics (GRPC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "35e322fe-0231-4a80-9ab5-1266d8293686",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting get-metrics.sh\n"
     ]
    }
   ],
   "source": [
    "%%writefile get-metrics.sh\n",
    "QUERY='query=seldon_api_executor_client_requests_seconds_count{deployment_name=~\"echo\",namespace=~\"seldon\",method=~\"unary\"}'\n",
    "QUERY_URL=http://seldon-monitoring-prometheus.seldon-monitoring.svc.cluster.local:9090/api/v1/query\n",
    "\n",
    "kubectl run --quiet=true -it --rm curlmetrics-$(date +%s) --image=radial/busyboxplus:curl --restart=Never -- \\\n",
    "    curl --data-urlencode ${QUERY} ${QUERY_URL}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4c2803f4-ae28-4bd3-b413-7b46905562d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics = ! bash get-metrics.sh\n",
    "metrics = json.loads(metrics[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6f049fb7-a180-40c7-91ed-e93916e12ff5",
   "metadata": {},
   "outputs": [],
   "source": [
    "counter = int(metrics[\"data\"][\"result\"][0][\"value\"][1])\n",
    "assert counter == 10, f\"expected 10 requests, got {counter}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18f24cd5-dc78-47d8-91a2-29cf4d9e15ce",
   "metadata": {},
   "source": [
    "## Check Custom Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16ef0e4f-54ac-46e0-8802-3671f59b6301",
   "metadata": {},
   "source": [
    "This model defines a few custom metrics in its `.py` class definition:\n",
    "```Python\n",
    "    def metrics(self):\n",
    "        print(\"metrics called\")\n",
    "        return [\n",
    "            # a counter which will increase by the given value\n",
    "            {\"type\": \"COUNTER\", \"key\": \"mycounter\", \"value\": 1},\n",
    "\n",
    "            # a gauge which will be set to given value\n",
    "            {\"type\": \"GAUGE\", \"key\": \"mygauge\", \"value\": 100},\n",
    "\n",
    "            # a timer (in msecs) which  will be aggregated into HISTOGRAM\n",
    "            {\"type\": \"TIMER\", \"key\": \"mytimer\", \"value\": 20.2},\n",
    "        ]\n",
    "```      \n",
    "\n",
    "We will be checking value of `mygaguge` metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a068adcf-69d4-4c36-ade2-2633eff3a121",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting get-metrics.sh\n"
     ]
    }
   ],
   "source": [
    "%%writefile get-metrics.sh\n",
    "QUERY='query=mygauge{deployment_name=~\"echo\",namespace=~\"seldon\"}'\n",
    "QUERY_URL=http://seldon-monitoring-prometheus.seldon-monitoring.svc.cluster.local:9090/api/v1/query\n",
    "\n",
    "kubectl run --quiet=true -it --rm curlmetrics-$(date +%s) --image=radial/busyboxplus:curl --restart=Never -- \\\n",
    "    curl --data-urlencode ${QUERY} ${QUERY_URL}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5a6bd6c8-979a-485a-8a03-a1fdf8f57ae4",
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics = ! bash get-metrics.sh\n",
    "metrics = json.loads(metrics[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "13dcac52-9291-4a6b-a7c0-3585540653f9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'status': 'success',\n",
       " 'data': {'resultType': 'vector',\n",
       "  'result': [{'metric': {'__name__': 'mygauge',\n",
       "     'container': 'classifier',\n",
       "     'deployment_name': 'echo',\n",
       "     'endpoint': 'metrics',\n",
       "     'image_name': 'seldonio/echo-model',\n",
       "     'image_version': '1.15.0-dev',\n",
       "     'instance': '10.244.0.43:6000',\n",
       "     'job': 'seldon-monitoring/seldon-podmonitor',\n",
       "     'method': 'predict',\n",
       "     'model_image': 'seldonio/echo-model',\n",
       "     'model_name': 'classifier',\n",
       "     'model_version': '1.15.0-dev',\n",
       "     'namespace': 'seldon',\n",
       "     'pod': 'echo-default-0-classifier-6fcd878bc5-pzzsc',\n",
       "     'predictor_name': 'default',\n",
       "     'predictor_version': 'default',\n",
       "     'seldon_deployment_name': 'echo',\n",
       "     'worker_id': '50'},\n",
       "    'value': [1652106270.106, '100']},\n",
       "   {'metric': {'__name__': 'mygauge',\n",
       "     'container': 'classifier',\n",
       "     'deployment_name': 'echo',\n",
       "     'endpoint': 'metrics',\n",
       "     'image_name': 'seldonio/echo-model',\n",
       "     'image_version': '1.15.0-dev',\n",
       "     'instance': '10.244.0.43:6000',\n",
       "     'job': 'seldon-monitoring/seldon-podmonitor',\n",
       "     'method': 'predict',\n",
       "     'model_image': 'seldonio/echo-model',\n",
       "     'model_name': 'classifier',\n",
       "     'model_version': '1.15.0-dev',\n",
       "     'namespace': 'seldon',\n",
       "     'pod': 'echo-default-0-classifier-6fcd878bc5-pzzsc',\n",
       "     'predictor_name': 'default',\n",
       "     'predictor_version': 'default',\n",
       "     'seldon_deployment_name': 'echo',\n",
       "     'worker_id': '58'},\n",
       "    'value': [1652106270.106, '100']},\n",
       "   {'metric': {'__name__': 'mygauge',\n",
       "     'container': 'classifier',\n",
       "     'deployment_name': 'echo',\n",
       "     'endpoint': 'metrics',\n",
       "     'image_name': 'seldonio/echo-model',\n",
       "     'image_version': '1.15.0-dev',\n",
       "     'instance': '10.244.0.43:6000',\n",
       "     'job': 'seldon-system/seldon-podmonitor',\n",
       "     'method': 'predict',\n",
       "     'model_image': 'seldonio/echo-model',\n",
       "     'model_name': 'classifier',\n",
       "     'model_version': '1.15.0-dev',\n",
       "     'namespace': 'seldon',\n",
       "     'pod': 'echo-default-0-classifier-6fcd878bc5-pzzsc',\n",
       "     'predictor_name': 'default',\n",
       "     'predictor_version': 'default',\n",
       "     'seldon_deployment_name': 'echo',\n",
       "     'worker_id': '50'},\n",
       "    'value': [1652106270.106, '100']},\n",
       "   {'metric': {'__name__': 'mygauge',\n",
       "     'container': 'classifier',\n",
       "     'deployment_name': 'echo',\n",
       "     'endpoint': 'metrics',\n",
       "     'image_name': 'seldonio/echo-model',\n",
       "     'image_version': '1.15.0-dev',\n",
       "     'instance': '10.244.0.43:6000',\n",
       "     'job': 'seldon-system/seldon-podmonitor',\n",
       "     'method': 'predict',\n",
       "     'model_image': 'seldonio/echo-model',\n",
       "     'model_name': 'classifier',\n",
       "     'model_version': '1.15.0-dev',\n",
       "     'namespace': 'seldon',\n",
       "     'pod': 'echo-default-0-classifier-6fcd878bc5-pzzsc',\n",
       "     'predictor_name': 'default',\n",
       "     'predictor_version': 'default',\n",
       "     'seldon_deployment_name': 'echo',\n",
       "     'worker_id': '58'},\n",
       "    'value': [1652106270.106, '100']}]}}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a09b418c-fff1-4a40-8f2d-c66a575b5794",
   "metadata": {},
   "outputs": [],
   "source": [
    "gauge = int(metrics[\"data\"][\"result\"][0][\"value\"][1])\n",
    "assert gauge == 100, f\"expected 100 on guage, got {gauge}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5367eb4d-90c8-4383-a4b8-30e8dfa66907",
   "metadata": {},
   "source": [
    "## Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a836ece4-879f-42b8-9622-c01a0d1f1bb9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"echo\" deleted\n",
      "W0509 15:24:31.442158  428564 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n",
      "W0509 15:24:31.443380  428564 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n",
      "W0509 15:24:31.443570  428564 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n",
      "W0509 15:24:31.443720  428564 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n",
      "W0509 15:24:31.443934  428564 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\n",
      "release \"seldon-monitoring\" uninstalled\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete sdep -n seldon echo\n",
    "!helm uninstall -n seldon-monitoring seldon-monitoring"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}