tutorials/Tutorial_Python_SDK.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6c4dae06",
   "metadata": {},
   "source": [
    "# Python SDK Tutorial\n",
    "This notebook explains how to use the Python SDK of ModelBox and explains the major concepts and how to use the API independent of any Deep Learning Framework. Please follow the PyTorch notebook to see how the SDK can be integrated with a PyTorch trainer. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "063f9e83-e9d5-4583-ac45-b1dc5a7c3bc1",
   "metadata": {},
   "source": [
    "## Install ModelBox SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "46cfbb6a-951b-4c9c-8e78-4ccff15dfc79",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting modelbox==0.0.5\n",
      "  Downloading modelbox-0.0.5-py3-none-any.whl (24 kB)\n",
      "Requirement already satisfied: grpcio in /opt/conda/lib/python3.10/site-packages (1.49.1)\n",
      "Requirement already satisfied: tabulate>=0.8.10 in /opt/conda/lib/python3.10/site-packages (from modelbox==0.0.5) (0.9.0)\n",
      "Requirement already satisfied: grpcio-tools>=1.46.1 in /opt/conda/lib/python3.10/site-packages (from modelbox==0.0.5) (1.49.1)\n",
      "Collecting protobuf>=3.20.1\n",
      "  Using cached protobuf-4.21.7-cp37-abi3-manylinux2014_x86_64.whl (408 kB)\n",
      "Requirement already satisfied: six>=1.5.2 in /opt/conda/lib/python3.10/site-packages (from grpcio) (1.16.0)\n",
      "Requirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from grpcio-tools>=1.46.1->modelbox==0.0.5) (63.3.0)\n",
      "Installing collected packages: protobuf, modelbox\n",
      "  Attempting uninstall: protobuf\n",
      "    Found existing installation: protobuf 3.19.6\n",
      "    Uninstalling protobuf-3.19.6:\n",
      "      Successfully uninstalled protobuf-3.19.6\n",
      "  Attempting uninstall: modelbox\n",
      "    Found existing installation: modelbox 0.0.4\n",
      "    Uninstalling modelbox-0.0.4:\n",
      "      Successfully uninstalled modelbox-0.0.4\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "tensorboard 2.10.1 requires protobuf<3.20,>=3.9.2, but you have protobuf 4.21.7 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed modelbox-0.0.5 protobuf-4.21.7\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install modelbox==0.0.5 grpcio"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "631504a4",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Initialize the ModelBox Client\n",
    "First, we initialize the client by pointing it to the address of the ModelBox Server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9c02d955",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from modelbox.modelbox import ModelBox, MLFramework, Artifact, ArtifactMime, MetricValue, LocalFile\n",
    "\n",
    "mbox = ModelBox(addr=\"127.0.0.1:8085\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a03eb91b",
   "metadata": {},
   "source": [
    "## Create an Experiment \n",
    "Once we have a client, we can start using it to create a new Experiment to train a model or track an existing pre-trained model. Let us first see how to create an experiment. We are going to create an experiment to train a Wav2Vec Model with Pytorch and store it in a namespace called *langtech*. If you are using an experiment management service like Weights and Biases or Nepute, you could associate the ID from that service with modelbox to create a lineage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b5a51e68",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'8649388612666864736'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment = mbox.new_experiment(\"wav2vec\", \"owner@pytorch.org\", \"langtech\", \"extern123\", MLFramework.PYTORCH)\n",
    "experiment.id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d810f5f",
   "metadata": {},
   "source": [
    "The above code is going to create a new experiment and give us an ID. You can list the experiments of a namespace by -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ecf2d8e5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ListExperimentsResponse(experiments=[Experiment(id='8649388612666864736', name='wav2vec', owner='owner@pytorch.org', namespace='langtech', external_id='extern123', created_at=seconds: 41\n",
       "nanos: 674434000\n",
       ", updated_at=seconds: 41\n",
       "nanos: 674440000\n",
       ", framework=<MLFramework.PYTORCH: 2>)])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mbox.experiments(namespace=\"langtech\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8926dae8",
   "metadata": {},
   "source": [
    "#### Adding metadata\n",
    "Metadata can be added to any of the objects in ModelBox after they have been created. For example, once an experiment is created, metadata can be added and listed at any stage -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8025ffb1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ListMetadataResponse(metadata={'foo/bar': 12})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment.update_metadata(\"foo/bar\", 12)\n",
    "experiment.metadata()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "774a768d",
   "metadata": {},
   "source": [
    "## Working with Checkpoints\n",
    "Once we have an experiment we can create model checkpoints from the trainers. Let's assume the file stored in assets/mnist_checkpoint1.pt is a checkpoint created by the trainer. We will now associate this checkpoint with ModelBox.\n",
    "\n",
    "We could either track the path of the checkpoint or upload the blob and let ModelBox store it in the configured blob store. The benefit of letting ModelBox store the checkpoint is that the trainer doesn't need to have access to the blob store directly. However, in some cases, it's more optimal to have the trainer store the blob directly when the path to IO to the blob store from the trainer is much faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "eb6323ac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogMetricsResponse(updated_at=1678663308)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "file =LocalFile.from_path('artifacts/mnist_cnn.pt')\n",
    "metrics = {'val_accu': 98.5, 'train_accu': 99.2}\n",
    "experiment.track_file(\"checkpoint-0\", file)\n",
    "artifacts = experiment.artifacts\n",
    "artifacts[0].log_metrics(metrics=metrics, step=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c91a59cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'train_accu': [MetricValue(step=1, wallclock_time=64, value=99.19999694824219),\n",
       "  MetricValue(step=1, wallclock_time=64, value=99.19999694824219),\n",
       "  MetricValue(step=1, wallclock_time=64, value=99.19999694824219),\n",
       "  MetricValue(step=1, wallclock_time=64, value=99.19999694824219),\n",
       "  MetricValue(step=1, wallclock_time=64, value=99.19999694824219)],\n",
       " 'val_accu': [MetricValue(step=1, wallclock_time=64, value=98.5),\n",
       "  MetricValue(step=1, wallclock_time=64, value=98.5),\n",
       "  MetricValue(step=1, wallclock_time=64, value=98.5),\n",
       "  MetricValue(step=1, wallclock_time=64, value=98.5),\n",
       "  MetricValue(step=1, wallclock_time=64, value=98.5)]}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "artifacts[0].all_metrics()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "489ea146",
   "metadata": {},
   "source": [
    "This returns us the artifactID and tracks the path of the checkpoint created by the trainer.\n",
    "\n",
    "Now let's say that we also want ModelBox to store the checkpoint, we will simply set the flag `upload` in the above api"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6a2ab84d",
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = experiment.upload_file(artifact_name=\"checkpoint-0\", f=file)\n",
    "resp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "783bd85b",
   "metadata": {},
   "source": [
    "Once checkpoints are created they can be listed by passing the experiment name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4daee87e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Artifact(name='checkpoint-0', id='10820184364444362974', parent_id='8649388612666864736', assets=[ArtifactAsset(parent='8649388612666864736', src_path='/home/diptanuc/Projects/modelbox/tutorials/artifacts/mnist_cnn.pt', upload_path='', mime_type=<ArtifactMime.Unknown: 0>, checksum='', id='2639677390021709401'), ArtifactAsset(parent='8649388612666864736', src_path='/home/diptanuc/Projects/modelbox/tutorials/artifacts/mnist_cnn.pt', upload_path='', mime_type=<ArtifactMime.Unknown: 0>, checksum='31c437e7d87fa749a7e049f0ccc46dd0', id='12632304970294682093')]),\n",
       " Artifact(name='checkpoint1', id='16406521829230001821', parent_id='8649388612666864736', assets=[ArtifactAsset(parent='8649388612666864736', src_path='/home/diptanuc/Projects/modelbox/tutorials/artifacts/mnist_cnn.pt', upload_path='modelbox/artifacts/8649388612666864736/155603561208062398', mime_type=<ArtifactMime.Checkpoint: 2>, checksum='31c437e7d87fa749a7e049f0ccc46dd0', id='155603561208062398')])]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment.artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c690dee",
   "metadata": {},
   "source": [
    "## Working with Models and ModelVersions\n",
    "\n",
    "Model objects describe tasks performed, metadata, which datasets are used to train, how to use the models during inference, etc. ModelVersions are trained instances of a model. So for example over time an English ASR(speech to text) model can have multiple model versions as they are trained with different datasets and such. \n",
    "\n",
    "We don't prescribe the granularity of Models and ModelVersions. If it's easier to create different Models every time a new model is trained with different hyperparameters and a single ModelVersion pointing to the model artifacts and all the metrics that is fine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0e9bbc1c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'14142242426467866576'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = mbox.new_model(name='asr_en', owner='owner@owner.org', namespace='langtech', task='asr', description='ASR for english')\n",
    "model.update_metadata(key='x', value='y')\n",
    "model.id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4760fe7",
   "metadata": {},
   "source": [
    "In the same way a ModelVersion can be created by the client, and track the associated artifacts and metadata.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a48f0d9c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'16818571058352993853'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags =[\"test\"]\n",
    "model_version = model.new_model_version(name=\"asr_en_july\", version=\"1\", description='ASR for english',unique_tags=tags, artifacts=[], framework=MLFramework.PYTORCH)\n",
    "model_version.id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26fe5d37",
   "metadata": {},
   "source": [
    "Once a modelversion is created we can upload the model and associate with the model version object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "99487ba8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Artifact(name='model1', id='6984394584362292612', parent_id='16818571058352993853', assets=[ArtifactAsset(parent='16818571058352993853', src_path='/home/diptanuc/Projects/modelbox/tutorials/artifacts/mnist_cnn.pt', upload_path='modelbox/artifacts/16818571058352993853/5402444647884978184', mime_type=<ArtifactMime.Checkpoint: 2>, checksum='31c437e7d87fa749a7e049f0ccc46dd0', id='5402444647884978184'), ArtifactAsset(parent='16818571058352993853', src_path='/home/diptanuc/Projects/modelbox/tutorials/artifacts/mnist_cnn.pt', upload_path='modelbox/artifacts/16818571058352993853/9243652877292513617', mime_type=<ArtifactMime.Unknown: 0>, checksum='31c437e7d87fa749a7e049f0ccc46dd0', id='9243652877292513617')])]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_path = LocalFile.from_path('artifacts/mnist_cnn.pt')\n",
    "model_version.upload_file(artifact_name=\"model1\", f=model_path)\n",
    "model_version.artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "912b53a2",
   "metadata": {},
   "source": [
    "The model file can now be served by the file server built into Model Box to inference servers. Inference services can either use the language specific SDKs in Python, Rust or Go or call the GRPC `DownloadFile` API directly which streams the files."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "293fe5be",
   "metadata": {},
   "source": [
    "> **_NOTE: Checkpoints Transforms to ModelVersions_** \n",
    "Usually in production engineers look at checkpoints/models created during training and select a version which has the best metrics. Once we have the worker infrastructure in place, we will create APIs which to do automatic convertion of checkpoints to ModelVersions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d145152f",
   "metadata": {},
   "source": [
    "## Tracking Artifacts and Working with Files\n",
    "Modelbox can track artifacts used in training and also users can upload Files and associate them with experiments, models and model versions. For example, a user can track the dataset files used for training stored in S3 or even upload them to ModelBox. A trained model can be uploaded and then later streamed to applications for inferencing.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "0a325e46",
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = experiment.track_file(artifact_name=\"dataset\", f=LocalFile.from_path(\"artifacts/test_artifact.txt\"))\n",
    "resp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c029e37d",
   "metadata": {},
   "source": [
    "Modelbox is now tracking the artifact and has information about the checksum, local path of the file, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "4629da68",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Artifact(name='checkpoint-0', id='10820184364444362974', parent_id='8649388612666864736', assets=[ArtifactAsset(parent='8649388612666864736', src_path='/home/diptanuc/Projects/modelbox/tutorials/artifacts/mnist_cnn.pt', upload_path='', mime_type=<ArtifactMime.Unknown: 0>, checksum='', id='2639677390021709401'), ArtifactAsset(parent='8649388612666864736', src_path='/home/diptanuc/Projects/modelbox/tutorials/artifacts/mnist_cnn.pt', upload_path='', mime_type=<ArtifactMime.Unknown: 0>, checksum='31c437e7d87fa749a7e049f0ccc46dd0', id='12632304970294682093')]),\n",
       " Artifact(name='dataset', id='2562978676191970736', parent_id='8649388612666864736', assets=[ArtifactAsset(parent='8649388612666864736', src_path='/home/diptanuc/Projects/modelbox/tutorials/artifacts/test_artifact.txt', upload_path='', mime_type=<ArtifactMime.Unknown: 0>, checksum='0019d23bef56a136a1891211d7007f6f', id='15017724009377637244')]),\n",
       " Artifact(name='checkpoint1', id='16406521829230001821', parent_id='8649388612666864736', assets=[ArtifactAsset(parent='8649388612666864736', src_path='/home/diptanuc/Projects/modelbox/tutorials/artifacts/mnist_cnn.pt', upload_path='modelbox/artifacts/8649388612666864736/155603561208062398', mime_type=<ArtifactMime.Checkpoint: 2>, checksum='31c437e7d87fa749a7e049f0ccc46dd0', id='155603561208062398')])]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment.artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dc63ee7",
   "metadata": {},
   "source": [
    "## Metrics \n",
    "ModelBox supports adding Metrics to experiments. Metrics can be logged to a key with values being a float, string or bytes. Metric values are associated with a step unit, and wallclock time when the metric was emitted by the application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "4c67a930",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'val_accu': [MetricValue(step=1, wallclock_time=64, value=0.7300000190734863),\n",
       "  MetricValue(step=2, wallclock_time=64, value=0.7699999809265137)],\n",
       " 'loss': [MetricValue(step=1, wallclock_time=64, value=0.15000000596046448),\n",
       "  MetricValue(step=2, wallclock_time=64, value=0.11999999731779099)]}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import time\n",
    "experiment.log_metrics(metrics={'val_accu': 0.73, 'loss': 0.15}, step=1, wallclock=int(time.time()))\n",
    "experiment.log_metrics(metrics={'val_accu': 0.77, 'loss': 0.12}, step=2, wallclock=int(time.time()))\n",
    "\n",
    "experiment.all_metrics()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7da99f9",
   "metadata": {},
   "source": [
    "Metrics can be added to any of the modelbox objects including Model, ModelVersion."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "899bba05-d7bc-40a4-9c3c-e1514acf0f98",
   "metadata": {},
   "source": [
    "## Events\n",
    "Events can be logged into the system against an experiment or a model to aid observability of the processes involved with training models or deployment of models in production."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "82d9d857-94df-4d48-be97-403408dab973",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Event(name='data_download_start', source=EventSource(name='trainer'), wallclock_time=seconds: 19\n",
       " nanos: 895433000\n",
       " , metadata={}),\n",
       " Event(name='data_download_finish', source=EventSource(name='trainer'), wallclock_time=seconds: 19\n",
       " nanos: 908435000\n",
       " , metadata={})]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from modelbox.modelbox import Event, EventSource\n",
    "\n",
    "# Log events to \n",
    "experiment.log_event(Event(name=\"data_download_start\", source=EventSource(name=\"trainer\"), wallclock_time = int(time.time()) , metadata={}))\n",
    "#..... load training data from the network in memory or on the disk of the trainer\n",
    "experiment.log_event(Event(name=\"data_download_finish\", source=EventSource(name=\"trainer\"), wallclock_time = int(time.time()) , metadata={}))\n",
    "\n",
    "experiment.events() # same events can be logged repeatedly as long as wallclock time is incremented"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e10a5c30-471e-44b4-990b-c414ec3a5acd",
   "metadata": {},
   "source": [
    "Events generally help in troubleshooting training processes or any other MLOps processes when a stage in the pipeline takes a long time to execute or the pipeline doesn't work as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "09fc0332-1526-4ec8-8cc2-a9db443b1ffe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Event(name='prod_deployment', source=EventSource(name='CD_Service_name'), wallclock_time=seconds: 22\n",
       " nanos: 411304000\n",
       " , metadata={})]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_version.log_event(Event(name=\"prod_deployment\", source=EventSource(name=\"CD_Service_name\"), wallclock_time = int(time.time()) , metadata={}))\n",
    "model_version.events()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3544d4c5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "345px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}