examples/batch/argo-workflows-batch/README.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch processing with Argo Worfklows\n",
    "\n",
    "In this notebook we will dive into how you can run batch processing with Argo Workflows and Seldon Core.\n",
    "\n",
    "Dependencies:\n",
    "\n",
    "* Seldon core installed as per the docs with an ingress\n",
    "* Minio running in your cluster to use as local (s3) object storage\n",
    "* Argo Workfklows installed in cluster (and argo CLI for commands)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup\n",
    "\n",
    "#### Install Seldon Core\n",
    "Use the notebook to [set-up Seldon Core with Ambassador or Istio Ingress](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html).\n",
    "\n",
    "Note: If running with KIND you need to make sure do follow [these steps](https://github.com/argoproj/argo-workflows/issues/2376#issuecomment-595593237) as workaround to the `/.../docker.sock` known issue.\n",
    "\n",
    "#### Set up Minio in your cluster\n",
    "Use the notebook to [set-up Minio in your cluster](https://docs.seldon.io/projects/seldon-core/en/latest/examples/minio_setup.html).\n",
    "\n",
    "#### Create rclone configuration\n",
    "In this example, our workflow stages responsible for pulling / pushing data to in-cluster MinIO S3 storage will use `rclone` CLI.\n",
    "In order to configure the CLI we will create a following secret:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting rclone-config.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile rclone-config.yaml\n",
    "apiVersion: v1\n",
    "kind: Secret\n",
    "metadata:\n",
    "  name: rclone-config-secret\n",
    "type: Opaque\n",
    "stringData:\n",
    "  rclone.conf: |\n",
    "    [cluster-minio]\n",
    "    type = s3\n",
    "    provider = minio\n",
    "    env_auth = false\n",
    "    access_key_id = minioadmin\n",
    "    secret_access_key = minioadmin\n",
    "    endpoint = http://minio.minio-system.svc.cluster.local:9000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "secret/rclone-config-secret created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -n default -f rclone-config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Install Argo Workflows\n",
    "You can follow the instructions from the official [Argo Workflows Documentation](https://github.com/argoproj/argo#quickstart).\n",
    "\n",
    "You also need to make sure that argo has permissions to create seldon deployments - for this you can create a role:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting role.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile role.yaml\n",
    "apiVersion: rbac.authorization.k8s.io/v1\n",
    "kind: Role\n",
    "metadata:\n",
    "  name: workflow\n",
    "rules:\n",
    "- apiGroups:\n",
    "  - \"\"\n",
    "  resources:\n",
    "  - pods\n",
    "  verbs:\n",
    "  - \"*\"\n",
    "- apiGroups:\n",
    "  - \"apps\"\n",
    "  resources:\n",
    "  - deployments\n",
    "  verbs:\n",
    "  - \"*\"\n",
    "- apiGroups:\n",
    "  - \"\"\n",
    "  resources:\n",
    "  - pods/log\n",
    "  verbs:\n",
    "  - \"*\"\n",
    "- apiGroups:\n",
    "  - machinelearning.seldon.io\n",
    "  resources:\n",
    "  - \"*\"\n",
    "  verbs:\n",
    "  - \"*\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['role.rbac.authorization.k8s.io/workflow created']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!!kubectl apply -n default -f role.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A service account:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "serviceaccount/workflow created\n"
     ]
    }
   ],
   "source": [
    "!kubectl create -n default serviceaccount workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And a binding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rolebinding.rbac.authorization.k8s.io/workflow created\n"
     ]
    }
   ],
   "source": [
    "!kubectl create rolebinding workflow -n default --role=workflow --serviceaccount=default:workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create some input for our model\n",
    "\n",
    "We will create a file that will contain the inputs that will be sent to our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir -p assets/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "\n",
    "random.seed(0)\n",
    "with open(\"assets/input-data.txt\", \"w\") as f:\n",
    "    for _ in range(10000):\n",
    "        data = [random.random() for _ in range(4)]\n",
    "        data = \"[[\" + \", \".join(str(x) for x in data) + \"]]\\n\"\n",
    "        f.write(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check the contents of the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000 assets/input-data.txt\n",
      "[[0.8444218515250481, 0.7579544029403025, 0.420571580830845, 0.25891675029296335]]\n",
      "[[0.5112747213686085, 0.4049341374504143, 0.7837985890347726, 0.30331272607892745]]\n",
      "[[0.4765969541523558, 0.5833820394550312, 0.9081128851953352, 0.5046868558173903]]\n",
      "[[0.28183784439970383, 0.7558042041572239, 0.6183689966753316, 0.25050634136244054]]\n",
      "[[0.9097462559682401, 0.9827854760376531, 0.8102172359965896, 0.9021659504395827]]\n",
      "[[0.3101475693193326, 0.7298317482601286, 0.8988382879679935, 0.6839839319154413]]\n",
      "[[0.47214271545271336, 0.1007012080683658, 0.4341718354537837, 0.6108869734438016]]\n",
      "[[0.9130110532378982, 0.9666063677707588, 0.47700977655271704, 0.8653099277716401]]\n",
      "[[0.2604923103919594, 0.8050278270130223, 0.5486993038355893, 0.014041700164018955]]\n",
      "[[0.7197046864039541, 0.39882354222426875, 0.824844977148233, 0.6681532012318508]]\n"
     ]
    }
   ],
   "source": [
    "!wc -l assets/input-data.txt\n",
    "!head assets/input-data.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Upload the file to our minio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[m\u001b[32;1mBucket created successfully `minio-seldon/data`.\u001b[0m\n",
      "...-data.txt:  820.96 KiB / 820.96 KiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 71.44 MiB/s 0s\u001b[0m\u001b[0m\u001b[m\u001b[32;1m"
     ]
    }
   ],
   "source": [
    "!mc mb minio-seldon/data\n",
    "!mc cp assets/input-data.txt minio-seldon/data/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Argo Workflow\n",
    "\n",
    "In order to create our argo workflow we have made it simple so you can leverage the power of the helm charts.\n",
    "\n",
    "Before we dive into the contents of the full helm chart, let's first give it a try with some of the settings.\n",
    "\n",
    "We will run a batch job that will set up a Seldon Deployment with 10 replicas and 100 batch client workers to send requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:                seldon-batch-process\n",
      "Namespace:           default\n",
      "ServiceAccount:      workflow\n",
      "Status:              Pending\n",
      "Created:             Fri Jan 15 11:44:56 +0000 (now)\n",
      "Progress:            \n"
     ]
    }
   ],
   "source": [
    "!helm template seldon-batch-workflow helm-charts/seldon-batch-workflow/ \\\n",
    "    --set workflow.name=seldon-batch-process \\\n",
    "    --set seldonDeployment.name=sklearn \\\n",
    "    --set seldonDeployment.replicas=10 \\\n",
    "    --set seldonDeployment.serverWorkers=1 \\\n",
    "    --set seldonDeployment.serverThreads=10 \\\n",
    "    --set batchWorker.workers=100 \\\n",
    "    --set batchWorker.payloadType=ndarray \\\n",
    "    --set batchWorker.dataType=data \\\n",
    "    | argo submit --serviceaccount workflow -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                   STATUS    AGE   DURATION   PRIORITY\n",
      "seldon-batch-process   Running   10s   10s        0\n"
     ]
    }
   ],
   "source": [
    "!argo list -n default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:                seldon-batch-process\n",
      "Namespace:           default\n",
      "ServiceAccount:      workflow\n",
      "Status:              Succeeded\n",
      "Conditions:          \n",
      " Completed           True\n",
      "Created:             Fri Jan 15 11:44:56 +0000 (2 minutes ago)\n",
      "Started:             Fri Jan 15 11:44:56 +0000 (2 minutes ago)\n",
      "Finished:            Fri Jan 15 11:47:00 +0000 (36 seconds ago)\n",
      "Duration:            2 minutes 4 seconds\n",
      "Progress:            6/6\n",
      "ResourcesDuration:   2m18s*(1 cpu),2m18s*(100Mi memory)\n",
      "\n",
      "\u001b[39mSTEP\u001b[0m                           TEMPLATE                         PODNAME                          DURATION  MESSAGE\n",
      " \u001b[32m✔\u001b[0m seldon-batch-process        seldon-batch-process                                                          \n",
      " ├───\u001b[32m✔\u001b[0m create-seldon-resource  create-seldon-resource-template  seldon-batch-process-3626514072  2s          \n",
      " ├───\u001b[32m✔\u001b[0m wait-seldon-resource    wait-seldon-resource-template    seldon-batch-process-2052519094  31s         \n",
      " ├───\u001b[32m✔\u001b[0m download-object-store   download-object-store-template   seldon-batch-process-1257652469  4s          \n",
      " ├───\u001b[32m✔\u001b[0m process-batch-inputs    process-batch-inputs-template    seldon-batch-process-2033515954  33s         \n",
      " ├───\u001b[32m✔\u001b[0m upload-object-store     upload-object-store-template     seldon-batch-process-2123074048  3s          \n",
      " └───\u001b[32m✔\u001b[0m delete-seldon-resource  delete-seldon-resource-template  seldon-batch-process-2070809024  9s          \n"
     ]
    }
   ],
   "source": [
    "!argo get -n default seldon-batch-process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mseldon-batch-process-3626514072: time=\"2021-01-15T11:44:57.620Z\" level=info msg=\"Starting Workflow Executor\" version=v2.12.3\u001b[0m\n",
      "\u001b[32mseldon-batch-process-3626514072: time=\"2021-01-15T11:44:57.622Z\" level=info msg=\"Creating a K8sAPI executor\"\u001b[0m\n",
      "\u001b[32mseldon-batch-process-3626514072: time=\"2021-01-15T11:44:57.622Z\" level=info msg=\"Executor (version: v2.12.3, build_date: 2021-01-05T00:54:54Z) initialized (pod: default/seldon-batch-process-3626514072) with template:\\n{\\\"name\\\":\\\"create-seldon-resource-template\\\",\\\"arguments\\\":{},\\\"inputs\\\":{},\\\"outputs\\\":{},\\\"metadata\\\":{\\\"annotations\\\":{\\\"sidecar.istio.io/inject\\\":\\\"false\\\"}},\\\"resource\\\":{\\\"action\\\":\\\"create\\\",\\\"manifest\\\":\\\"apiVersion: machinelearning.seldon.io/v1\\\\nkind: SeldonDeployment\\\\nmetadata:\\\\n  name: \\\\\\\"sklearn\\\\\\\"\\\\n  namespace: default\\\\n  ownerReferences:\\\\n  - apiVersion: argoproj.io/v1alpha1\\\\n    blockOwnerDeletion: true\\\\n    kind: Workflow\\\\n    name: \\\\\\\"seldon-batch-process\\\\\\\"\\\\n    uid: \\\\\\\"511f64a2-0699-42eb-897a-c0a57b24072c\\\\\\\"\\\\nspec:\\\\n  name: \\\\\\\"sklearn\\\\\\\"\\\\n  predictors:\\\\n    - componentSpecs:\\\\n      - spec:\\\\n        containers:\\\\n        - name: classifier\\\\n          env:\\\\n          - name: GUNICORN_THREADS\\\\n            value: 10\\\\n          - name: GUNICORN_WORKERS\\\\n            value: 1\\\\n          resources:\\\\n            requests:\\\\n              cpu: 50m\\\\n              memory: 100Mi\\\\n            limits:\\\\n              cpu: 50m\\\\n              memory: 1000Mi\\\\n      graph:\\\\n        children: []\\\\n        implementation: SKLEARN_SERVER\\\\n        modelUri: gs://seldon-models/sklearn/iris\\\\n        name: classifier\\\\n      name: default\\\\n      replicas: 10\\\\n\\\"}}\"\u001b[0m\n",
      "\u001b[32mseldon-batch-process-3626514072: time=\"2021-01-15T11:44:57.622Z\" level=info msg=\"Loading manifest to /tmp/manifest.yaml\"\u001b[0m\n",
      "\u001b[32mseldon-batch-process-3626514072: time=\"2021-01-15T11:44:57.622Z\" level=info msg=\"kubectl create -f /tmp/manifest.yaml -o json\"\u001b[0m\n",
      "\u001b[32mseldon-batch-process-3626514072: time=\"2021-01-15T11:44:58.044Z\" level=info msg=default/SeldonDeployment.machinelearning.seldon.io/sklearn\u001b[0m\n",
      "\u001b[32mseldon-batch-process-3626514072: time=\"2021-01-15T11:44:58.044Z\" level=info msg=\"Starting SIGUSR2 signal monitor\"\u001b[0m\n",
      "\u001b[32mseldon-batch-process-3626514072: time=\"2021-01-15T11:44:58.045Z\" level=info msg=\"No output parameters\"\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2052519094: Waiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 0 of 10 updated replicas are available...\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2052519094: Waiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 1 of 10 updated replicas are available...\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2052519094: Waiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 2 of 10 updated replicas are available...\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2052519094: Waiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 3 of 10 updated replicas are available...\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2052519094: Waiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 4 of 10 updated replicas are available...\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2052519094: Waiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 5 of 10 updated replicas are available...\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2052519094: Waiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 6 of 10 updated replicas are available...\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2052519094: Waiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 7 of 10 updated replicas are available...\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2052519094: Waiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 8 of 10 updated replicas are available...\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2052519094: Waiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 9 of 10 updated replicas are available...\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2052519094: deployment \"sklearn-default-0-classifier\" successfully rolled out\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:00,306 - batch_processor.py:167 - INFO:  Processed instances: 100\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:00,321 - batch_processor.py:167 - INFO:  Processed instances: 200\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:00,411 - batch_processor.py:167 - INFO:  Processed instances: 300\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:00,464 - batch_processor.py:167 - INFO:  Processed instances: 400\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:00,768 - batch_processor.py:167 - INFO:  Processed instances: 500\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:01,055 - batch_processor.py:167 - INFO:  Processed instances: 600\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:01,439 - batch_processor.py:167 - INFO:  Processed instances: 700\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:01,757 - batch_processor.py:167 - INFO:  Processed instances: 800\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:02,025 - batch_processor.py:167 - INFO:  Processed instances: 900\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:02,303 - batch_processor.py:167 - INFO:  Processed instances: 1000\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:02,563 - batch_processor.py:167 - INFO:  Processed instances: 1100\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:02,864 - batch_processor.py:167 - INFO:  Processed instances: 1200\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:03,151 - batch_processor.py:167 - INFO:  Processed instances: 1300\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:03,447 - batch_processor.py:167 - INFO:  Processed instances: 1400\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:03,848 - batch_processor.py:167 - INFO:  Processed instances: 1500\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:04,244 - batch_processor.py:167 - INFO:  Processed instances: 1600\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:04,547 - batch_processor.py:167 - INFO:  Processed instances: 1700\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:04,783 - batch_processor.py:167 - INFO:  Processed instances: 1800\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:05,080 - batch_processor.py:167 - INFO:  Processed instances: 1900\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:05,478 - batch_processor.py:167 - INFO:  Processed instances: 2000\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:06,074 - batch_processor.py:167 - INFO:  Processed instances: 2100\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:06,438 - batch_processor.py:167 - INFO:  Processed instances: 2200\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:06,817 - batch_processor.py:167 - INFO:  Processed instances: 2300\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:07,101 - batch_processor.py:167 - INFO:  Processed instances: 2400\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:07,312 - batch_processor.py:167 - INFO:  Processed instances: 2500\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:07,538 - batch_processor.py:167 - INFO:  Processed instances: 2600\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:07,766 - batch_processor.py:167 - INFO:  Processed instances: 2700\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:07,979 - batch_processor.py:167 - INFO:  Processed instances: 2800\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:08,265 - batch_processor.py:167 - INFO:  Processed instances: 2900\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:08,565 - batch_processor.py:167 - INFO:  Processed instances: 3000\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:08,820 - batch_processor.py:167 - INFO:  Processed instances: 3100\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:09,069 - batch_processor.py:167 - INFO:  Processed instances: 3200\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:09,326 - batch_processor.py:167 - INFO:  Processed instances: 3300\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:09,566 - batch_processor.py:167 - INFO:  Processed instances: 3400\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:09,786 - batch_processor.py:167 - INFO:  Processed instances: 3500\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:09,993 - batch_processor.py:167 - INFO:  Processed instances: 3600\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:10,209 - batch_processor.py:167 - INFO:  Processed instances: 3700\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:10,487 - batch_processor.py:167 - INFO:  Processed instances: 3800\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:10,757 - batch_processor.py:167 - INFO:  Processed instances: 3900\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:10,967 - batch_processor.py:167 - INFO:  Processed instances: 4000\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:11,221 - batch_processor.py:167 - INFO:  Processed instances: 4100\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:11,449 - batch_processor.py:167 - INFO:  Processed instances: 4200\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:11,705 - batch_processor.py:167 - INFO:  Processed instances: 4300\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:11,914 - batch_processor.py:167 - INFO:  Processed instances: 4400\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:12,130 - batch_processor.py:167 - INFO:  Processed instances: 4500\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:12,345 - batch_processor.py:167 - INFO:  Processed instances: 4600\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:12,621 - batch_processor.py:167 - INFO:  Processed instances: 4700\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:12,962 - batch_processor.py:167 - INFO:  Processed instances: 4800\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:13,219 - batch_processor.py:167 - INFO:  Processed instances: 4900\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:13,463 - batch_processor.py:167 - INFO:  Processed instances: 5000\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:13,730 - batch_processor.py:167 - INFO:  Processed instances: 5100\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:13,966 - batch_processor.py:167 - INFO:  Processed instances: 5200\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:14,246 - batch_processor.py:167 - INFO:  Processed instances: 5300\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:14,479 - batch_processor.py:167 - INFO:  Processed instances: 5400\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:14,764 - batch_processor.py:167 - INFO:  Processed instances: 5500\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:15,041 - batch_processor.py:167 - INFO:  Processed instances: 5600\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:15,319 - batch_processor.py:167 - INFO:  Processed instances: 5700\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:15,561 - batch_processor.py:167 - INFO:  Processed instances: 5800\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:15,795 - batch_processor.py:167 - INFO:  Processed instances: 5900\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:16,038 - batch_processor.py:167 - INFO:  Processed instances: 6000\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:16,284 - batch_processor.py:167 - INFO:  Processed instances: 6100\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:16,565 - batch_processor.py:167 - INFO:  Processed instances: 6200\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:16,862 - batch_processor.py:167 - INFO:  Processed instances: 6300\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:17,145 - batch_processor.py:167 - INFO:  Processed instances: 6400\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:17,416 - batch_processor.py:167 - INFO:  Processed instances: 6500\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:17,669 - batch_processor.py:167 - INFO:  Processed instances: 6600\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:17,881 - batch_processor.py:167 - INFO:  Processed instances: 6700\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:18,096 - batch_processor.py:167 - INFO:  Processed instances: 6800\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:18,357 - batch_processor.py:167 - INFO:  Processed instances: 6900\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:18,609 - batch_processor.py:167 - INFO:  Processed instances: 7000\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:18,996 - batch_processor.py:167 - INFO:  Processed instances: 7100\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:19,426 - batch_processor.py:167 - INFO:  Processed instances: 7200\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:19,804 - batch_processor.py:167 - INFO:  Processed instances: 7300\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:20,139 - batch_processor.py:167 - INFO:  Processed instances: 7400\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:20,528 - batch_processor.py:167 - INFO:  Processed instances: 7500\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:21,047 - batch_processor.py:167 - INFO:  Processed instances: 7600\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:21,439 - batch_processor.py:167 - INFO:  Processed instances: 7700\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:22,050 - batch_processor.py:167 - INFO:  Processed instances: 7800\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:22,435 - batch_processor.py:167 - INFO:  Processed instances: 7900\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:22,812 - batch_processor.py:167 - INFO:  Processed instances: 8000\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:23,226 - batch_processor.py:167 - INFO:  Processed instances: 8100\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:23,547 - batch_processor.py:167 - INFO:  Processed instances: 8200\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:23,898 - batch_processor.py:167 - INFO:  Processed instances: 8300\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:24,308 - batch_processor.py:167 - INFO:  Processed instances: 8400\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:24,672 - batch_processor.py:167 - INFO:  Processed instances: 8500\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:24,986 - batch_processor.py:167 - INFO:  Processed instances: 8600\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:25,284 - batch_processor.py:167 - INFO:  Processed instances: 8700\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:25,545 - batch_processor.py:167 - INFO:  Processed instances: 8800\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:25,851 - batch_processor.py:167 - INFO:  Processed instances: 8900\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:26,124 - batch_processor.py:167 - INFO:  Processed instances: 9000\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:26,445 - batch_processor.py:167 - INFO:  Processed instances: 9100\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:26,817 - batch_processor.py:167 - INFO:  Processed instances: 9200\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:27,165 - batch_processor.py:167 - INFO:  Processed instances: 9300\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:27,483 - batch_processor.py:167 - INFO:  Processed instances: 9400\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:27,794 - batch_processor.py:167 - INFO:  Processed instances: 9500\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:28,099 - batch_processor.py:167 - INFO:  Processed instances: 9600\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:28,479 - batch_processor.py:167 - INFO:  Processed instances: 9700\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:28,912 - batch_processor.py:167 - INFO:  Processed instances: 9800\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:29,465 - batch_processor.py:167 - INFO:  Processed instances: 9900\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:30,012 - batch_processor.py:167 - INFO:  Processed instances: 10000\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:30,012 - batch_processor.py:168 - INFO:  Total processed instances: 10000\u001b[0m\n",
      "\u001b[33mseldon-batch-process-2033515954: 2021-01-15 11:46:30,012 - batch_processor.py:116 - INFO:  Elapsed time: 30.899641513824463\u001b[0m\n",
      "\u001b[35mseldon-batch-process-2070809024: seldondeployment.machinelearning.seldon.io \"sklearn\" deleted\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!argo -n default logs seldon-batch-process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check output in object store\n",
    "\n",
    "We can now visualise the output that we obtained in the object store.\n",
    "\n",
    "First we can check that the file is present:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Workflow UID is 511f64a2-0699-42eb-897a-c0a57b24072c\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "wf_arr = !argo get -n default seldon-batch-process -o json\n",
    "wf = json.loads(\"\".join(wf_arr))\n",
    "WF_UID = wf[\"metadata\"][\"uid\"]\n",
    "print(f\"Workflow UID is {WF_UID}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[m\u001b[32m[2021-01-15 11:46:42 GMT] \u001b[0m\u001b[33m 3.4MiB \u001b[0moutput-data-511f64a2-0699-42eb-897a-c0a57b24072c.txt\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!mc ls minio-seldon/data/output-data-\"$WF_UID\".txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can output the contents of the file created using the `mc head` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...4072c.txt:  3.36 MiB / 3.36 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 192.59 MiB/s 0s\u001b[0m\u001b[0m\u001b[m\u001b[32;1m{\"data\": {\"names\": [\"t:0\", \"t:1\", \"t:2\"], \"ndarray\": [[0.1859090109477526, 0.46433848375587844, 0.349752505296369]]}, \"meta\": {\"requestPath\": {\"classifier\": \"seldonio/sklearnserver:1.6.0-dev\"}, \"tags\": {\"tags\": {\"batch_id\": \"3c4000b8-5727-11eb-91c1-6e88dc41eb63\", \"batch_index\": 1.0, \"batch_instance_id\": \"3c40e1e0-5727-11eb-9fe5-6e88dc41eb63\"}}}}\n",
      "{\"data\": {\"names\": [\"t:0\", \"t:1\", \"t:2\"], \"ndarray\": [[0.1679456497678022, 0.42318259169768935, 0.4088717585345084]]}, \"meta\": {\"requestPath\": {\"classifier\": \"seldonio/sklearnserver:1.6.0-dev\"}, \"tags\": {\"tags\": {\"batch_id\": \"3c4000b8-5727-11eb-91c1-6e88dc41eb63\", \"batch_index\": 22.0, \"batch_instance_id\": \"3c42efb2-5727-11eb-9fe5-6e88dc41eb63\"}}}}\n",
      "{\"data\": {\"names\": [\"t:0\", \"t:1\", \"t:2\"], \"ndarray\": [[0.5329356306409886, 0.2531124742231082, 0.21395189513590318]]}, \"meta\": {\"requestPath\": {\"classifier\": \"seldonio/sklearnserver:1.6.0-dev\"}, \"tags\": {\"tags\": {\"batch_id\": \"3c4000b8-5727-11eb-91c1-6e88dc41eb63\", \"batch_index\": 25.0, \"batch_instance_id\": \"3c43dac6-5727-11eb-9fe5-6e88dc41eb63\"}}}}\n",
      "{\"data\": {\"names\": [\"t:0\", \"t:1\", \"t:2\"], \"ndarray\": [[0.5057216294927378, 0.37562353221834527, 0.11865483828891676]]}, \"meta\": {\"requestPath\": {\"classifier\": \"seldonio/sklearnserver:1.6.0-dev\"}, \"tags\": {\"tags\": {\"batch_id\": \"3c4000b8-5727-11eb-91c1-6e88dc41eb63\", \"batch_index\": 20.0, \"batch_instance_id\": \"3c4294a4-5727-11eb-9fe5-6e88dc41eb63\"}}}}\n",
      "{\"data\": {\"names\": [\"t:0\", \"t:1\", \"t:2\"], \"ndarray\": [[0.16020781530738484, 0.49084414063547427, 0.3489480440571409]]}, \"meta\": {\"requestPath\": {\"classifier\": \"seldonio/sklearnserver:1.6.0-dev\"}, \"tags\": {\"tags\": {\"batch_id\": \"3c4000b8-5727-11eb-91c1-6e88dc41eb63\", \"batch_index\": 24.0, \"batch_instance_id\": \"3c439a34-5727-11eb-9fe5-6e88dc41eb63\"}}}}\n",
      "{\"data\": {\"names\": [\"t:0\", \"t:1\", \"t:2\"], \"ndarray\": [[0.49551509682202705, 0.4192462053867995, 0.08523869779117352]]}, \"meta\": {\"requestPath\": {\"classifier\": \"seldonio/sklearnserver:1.6.0-dev\"}, \"tags\": {\"tags\": {\"batch_id\": \"3c4000b8-5727-11eb-91c1-6e88dc41eb63\", \"batch_index\": 0.0, \"batch_instance_id\": \"3c40c6d8-5727-11eb-9fe5-6e88dc41eb63\"}}}}\n",
      "{\"data\": {\"names\": [\"t:0\", \"t:1\", \"t:2\"], \"ndarray\": [[0.17817271417040353, 0.4160568279837039, 0.4057704578458926]]}, \"meta\": {\"requestPath\": {\"classifier\": \"seldonio/sklearnserver:1.6.0-dev\"}, \"tags\": {\"tags\": {\"batch_id\": \"3c4000b8-5727-11eb-91c1-6e88dc41eb63\", \"batch_index\": 6.0, \"batch_instance_id\": \"3c41aa58-5727-11eb-9fe5-6e88dc41eb63\"}}}}\n",
      "{\"data\": {\"names\": [\"t:0\", \"t:1\", \"t:2\"], \"ndarray\": [[0.31086648314817084, 0.43371070280306884, 0.25542281404876027]]}, \"meta\": {\"requestPath\": {\"classifier\": \"seldonio/sklearnserver:1.6.0-dev\"}, \"tags\": {\"tags\": {\"batch_id\": \"3c4000b8-5727-11eb-91c1-6e88dc41eb63\", \"batch_index\": 27.0, \"batch_instance_id\": \"3c44420e-5727-11eb-9fe5-6e88dc41eb63\"}}}}\n",
      "{\"data\": {\"names\": [\"t:0\", \"t:1\", \"t:2\"], \"ndarray\": [[0.4381942165350952, 0.39483980719426687, 0.16696597627063794]]}, \"meta\": {\"requestPath\": {\"classifier\": \"seldonio/sklearnserver:1.6.0-dev\"}, \"tags\": {\"tags\": {\"batch_id\": \"3c4000b8-5727-11eb-91c1-6e88dc41eb63\", \"batch_index\": 34.0, \"batch_instance_id\": \"3c448ff2-5727-11eb-9fe5-6e88dc41eb63\"}}}}\n",
      "{\"data\": {\"names\": [\"t:0\", \"t:1\", \"t:2\"], \"ndarray\": [[0.2975075875929912, 0.25439317776178244, 0.44809923464522644]]}, \"meta\": {\"requestPath\": {\"classifier\": \"seldonio/sklearnserver:1.6.0-dev\"}, \"tags\": {\"tags\": {\"batch_id\": \"3c4000b8-5727-11eb-91c1-6e88dc41eb63\", \"batch_index\": 4.0, \"batch_instance_id\": \"3c41837a-5727-11eb-9fe5-6e88dc41eb63\"}}}}\n"
     ]
    }
   ],
   "source": [
    "!mc cp minio-seldon/data/output-data-\"$WF_UID\".txt assets/output-data.txt\n",
    "!head assets/output-data.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Workflow 'seldon-batch-process' deleted\n"
     ]
    }
   ],
   "source": [
    "!argo delete -n default seldon-batch-process"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}