interactive/hf_interactive.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d3260669-c7ec-4d06-a655-590c5e7ab152",
   "metadata": {},
   "source": [
    "# Transfer learning with Huggingface using CodeFlare"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4acfb10-1aa1-445d-947e-396ea5ebed1a",
   "metadata": {},
   "source": [
    "In this notebook you will learn how to leverage the **[huggingface](https://huggingface.co/)** support in ray ecosystem to carry out a text classification task using transfer learning. We will be referencing the example **[here](https://huggingface.co/docs/transformers/tasks/sequence_classification)**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70b77929-e96c-434e-ada3-8b14795bfbb1",
   "metadata": {},
   "source": [
    "The example carries out a text classification task on **[imdb dataset](https://huggingface.co/datasets/imdb)** and tries to classify the movie reviews as positive or negative. Huggingface library provides an easy way to build a model and the dataset to carry out this classification task. In this case we will be using **distilbert-base-uncased** model which is a **BERT** based model.\n",
    "\n",
    "Huggingface has a **[built in support for ray ecosystem](https://docs.ray.io/en/releases-1.13.0/_modules/ray/ml/train/integrations/huggingface/huggingface_trainer.html)** which allows the huggingface trainer to scale on CodeFlare and can scale the training as we add additional gpus and can run distributed training across multiple GPUs that will help scale out the training.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02593d04-40b9-4a07-a32e-40b649444ab5",
   "metadata": {},
   "source": [
    "### Getting all the requirements in place"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c737a768-6e31-4767-a301-60ae932b4ed9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pieces from codeflare-sdk\n",
    "from codeflare_sdk.cluster.cluster import Cluster, ClusterConfiguration\n",
    "from codeflare_sdk.cluster.auth import TokenAuthentication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0734734b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create authentication object for oc user permissions and login\n",
    "auth = TokenAuthentication(\n",
    "    token = \"sha256~6W9_zO5Rvz_-s4gUU-j580g9L-O2RRbB29JzBnCRixg\",\n",
    "    server = \"https://api.mini2.mydomain.com:6443\",\n",
    "    skip_tls = True\n",
    ")\n",
    "auth.login()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc27f84c",
   "metadata": {},
   "source": [
    "Here, we want to define our cluster by specifying the resources we require for our batch workload. Below, we define our cluster object (which generates a corresponding AppWrapper)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "220b9d85-3a3c-4c0c-aaf2-0d866823dcd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create our cluster and submit appwrapper\n",
    "cluster = Cluster(ClusterConfiguration(name='hfgputest', min_worker=1, max_worker=3, min_cpus=8, max_cpus=8, min_memory=16, max_memory=16, gpu=1, instascale=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12eef53c",
   "metadata": {},
   "source": [
    "Next, we want to bring our cluster up, so we call the `up()` function below to submit our cluster AppWrapper yaml onto the MCAD queue, and begin the process of obtaining our resource cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ae1d861-b743-4c05-903b-5799072b942b",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.up()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "657ebdfb",
   "metadata": {},
   "source": [
    "Now, we want to check on the initial status of our resource cluster, then wait until it is finally ready for use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d0db5f5-22f1-4806-ae7e-a0ee865625c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.status()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d26275e",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.wait_ready()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2969a4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.status()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "477ac246",
   "metadata": {},
   "source": [
    "Let's quickly verify that the specs of the cluster are as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06a54428-f186-4c27-948e-4eaf9c0e34b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.details()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ac46c87-70f1-4c70-9648-881151665355",
   "metadata": {},
   "outputs": [],
   "source": [
    "ray_cluster_uri = cluster.cluster_uri()\n",
    "print(ray_cluster_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44dba6a0-8275-4726-8911-6b6ec467b6a3",
   "metadata": {},
   "source": [
    "**NOTE**: Now we have our resource cluster with the desired GPUs, so we can interact with it to train the HuggingFace model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c458589-5a17-47c6-a8db-625427ae4fe7",
   "metadata": {},
   "outputs": [],
   "source": [
    "#before proceeding make sure the cluster exists and the uri is not empty\n",
    "assert ray_cluster_uri, \"Ray cluster needs to be started and set before proceeding\"\n",
    "\n",
    "import ray\n",
    "from ray.air.config import ScalingConfig\n",
    "\n",
    "# reset the ray context in case there's already one. \n",
    "ray.shutdown()\n",
    "# establish connection to ray cluster\n",
    "\n",
    "#install additionall libraries that will be required for this training\n",
    "runtime_env = {\"pip\": [\"accelerate\", \"transformers\", \"datasets\", \"evaluate\", \"pyarrow<7.0.0\"]}\n",
    "\n",
    "ray.init(address=f'{ray_cluster_uri}', runtime_env=runtime_env)\n",
    "\n",
    "print(\"Ray cluster is up and running: \", ray.is_initialized())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94a38146-1321-4b7b-9152-9ebca4eb9444",
   "metadata": {},
   "source": [
    "**NOTE** : in this case since we are running a task for which we need additional pip packages. we can install those by passing them in the `runtime_env` variable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76a1945b-d6c8-49b8-9a4c-b82724cffba9",
   "metadata": {},
   "source": [
    "### Transfer learning code from huggingface"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bdbe888-4f38-4e9a-ae43-67ce89ff9d42",
   "metadata": {},
   "source": [
    "We are using the code based on the example **[here](https://huggingface.co/docs/transformers/tasks/sequence_classification)** . "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e69994b4-1a13-43fe-b698-2a5374cb941b",
   "metadata": {},
   "outputs": [],
   "source": [
    "@ray.remote\n",
    "def train_fn():\n",
    "    from datasets import load_dataset\n",
    "    import transformers\n",
    "    from transformers import AutoTokenizer, TrainingArguments\n",
    "    from transformers import AutoModelForSequenceClassification\n",
    "    import numpy as np\n",
    "    from datasets import load_metric\n",
    "    import ray\n",
    "    from ray import tune\n",
    "    from ray.train.huggingface import HuggingFaceTrainer\n",
    "\n",
    "    dataset = load_dataset(\"imdb\")\n",
    "    tokenizer = AutoTokenizer.from_pretrained(\"distilbert-base-uncased\")\n",
    "\n",
    "    def tokenize_function(examples):\n",
    "        return tokenizer(examples[\"text\"], padding=\"max_length\", truncation=True)\n",
    "\n",
    "    tokenized_datasets = dataset.map(tokenize_function, batched=True)\n",
    "\n",
    "    # Using a fraction of dataset but you can run with the full dataset\n",
    "    # hmm, this does not limit to 1000 when we later use the ray.data.from_huggingface\n",
    "    small_train_dataset = tokenized_datasets[\"train\"]#.shuffle(seed=42).select(range(1000))\n",
    "    small_eval_dataset = tokenized_datasets[\"test\"]#.shuffle(seed=42).select(range(1000))\n",
    "    print(f\"len of train_dataset {small_train_dataset} and eval_dataset {small_eval_dataset}\")\n",
    "\n",
    "    # Using a fraction of dataset - The limit here works\n",
    "    ray_train_ds = ray.data.from_huggingface(small_train_dataset).random_shuffle(seed=42).limit(1000)\n",
    "    ray_evaluation_ds = ray.data.from_huggingface(small_eval_dataset).random_shuffle(seed=42).limit(1000)\n",
    "    print(f\"len of small_train_dataset {small_train_dataset} and small_eval_dataset {small_eval_dataset}\")\n",
    "\n",
    "    def compute_metrics(eval_pred):\n",
    "        metric = load_metric(\"accuracy\")\n",
    "        logits, labels = eval_pred\n",
    "        predictions = np.argmax(logits, axis=-1)\n",
    "        return metric.compute(predictions=predictions, references=labels)\n",
    "\n",
    "    def trainer_init_per_worker(train_dataset, eval_dataset, **config):\n",
    "        model = AutoModelForSequenceClassification.from_pretrained(\"distilbert-base-uncased\", num_labels=2)\n",
    "\n",
    "        training_args = TrainingArguments(\"/tmp/hf_imdb/test\", eval_steps=1, disable_tqdm=True, \n",
    "                                          num_train_epochs=3, skip_memory_metrics=True,\n",
    "                                          learning_rate=2e-5,\n",
    "                                          per_device_train_batch_size=16,\n",
    "                                          per_device_eval_batch_size=16,                                \n",
    "                                          weight_decay=0.01,)\n",
    "        return transformers.Trainer(\n",
    "            model=model,\n",
    "            args=training_args,\n",
    "            train_dataset=train_dataset,\n",
    "            eval_dataset=eval_dataset,\n",
    "            compute_metrics=compute_metrics\n",
    "        )\n",
    "\n",
    "    scaling_config = ScalingConfig(num_workers=3, use_gpu=True) #num workers is the number of gpus\n",
    "\n",
    "    # we are using the ray native HuggingFaceTrainer, but you can swap out to use non ray Huggingface Trainer. Both have the same method signature. \n",
    "    # the ray native HFTrainer has built in support for scaling to multiple GPUs\n",
    "    trainer = HuggingFaceTrainer(\n",
    "        trainer_init_per_worker=trainer_init_per_worker,\n",
    "        scaling_config=scaling_config,\n",
    "        datasets={\"train\": ray_train_ds, \"evaluation\": ray_evaluation_ds},\n",
    "    )\n",
    "    result = trainer.fit()\n",
    "    print(f\"metrics: {result.metrics}\")\n",
    "    print(f\"checkpoint: {result.checkpoint}\")\n",
    "    print(f\"log_dir: {result.log_dir}\")\n",
    "    return result.checkpoint\n",
    "    #return result.log_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9593fee-2b2b-415f-8902-bceec014385f",
   "metadata": {},
   "source": [
    "**NOTE:** This code will produce a lot of output and will run for **approximately 2 minutes.** As a part of execution it will download the `imdb` dataset, `distilbert-base-uncased` model and then will start transfer learning task for training the model with this dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f0985e9-5e88-4d36-ab38-c3001c13f97c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#call the above cell as a remote ray function\n",
    "result=ray.get(train_fn.remote())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec421113-0e49-4043-a3b5-66efa5021cdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ray.train.torch import TorchCheckpoint\n",
    "checkpoint: TorchCheckpoint = result\n",
    "path = checkpoint.to_directory()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c108cd0-4d29-4b7b-856b-e5d961f8afb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(path)\n",
    "!ls {path}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f31f441-c363-435a-bafb-da59454f33ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "#log_dir=result.log_dir\n",
    "#print(f\"log_dir: {log_dir}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fda215e-f8f0-4f41-9000-946219c1e100",
   "metadata": {},
   "source": [
    "# Inference using the checkpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "949abe38-377d-46bd-8193-6c628a40f762",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n",
    "#DistilbertTokenizerFast\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"distilbert-base-uncased\")\n",
    "id2label = {0: \"NEGATIVE\", 1: \"POSITIVE\"}\n",
    "label2id = {\"NEGATIVE\": 0, \"POSITIVE\": 1}\n",
    "model = AutoModelForSequenceClassification.from_pretrained(path,num_labels=2, id2label=id2label, label2id=label2id)\n",
    "text1 = \"This was a masterpiece. Not completely faithful to the books, but enthralling from beginning to end. Might be my favorite of the three.\"\n",
    "text2 = \"This is a catastrophe. Each of the three movies had different actors that made it difficult to follow.\"\n",
    "#inputs = tokenizer(text, return_tensors=\"pt\")\n",
    "batch=[text1,text2]\n",
    "inputs = tokenizer(batch, padding=True, truncation=True, max_length=512, return_tensors=\"pt\")\n",
    "\n",
    "with torch.no_grad(): logits = model(**inputs).logits # For pytorch you have to unpack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28508028-40f5-4cb3-a864-7649206599d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(logits)\n",
    "print(torch.nn.Softmax(dim=1)(logits)) #tf.math.softmax(logits, axis=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0fa12d2-87b1-4ab7-98f3-7220cd4ca717",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "print(np.array(logits))\n",
    "predicted_class_id = np.array(logits).argmax(axis=1)\n",
    "print(predicted_class_id)\n",
    "print([model.config.id2label[i] for i in predicted_class_id])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d992216f-ac9e-43cb-839b-92c25778a561",
   "metadata": {},
   "source": [
    "# Convert to onyx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f71df32-ab7f-403e-83e2-c44c9fc52fc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.onnx.export(\n",
    "    model, \n",
    "    tuple(inputs.values()),\n",
    "    f=\"torch-model.onnx\",  \n",
    "    input_names=['input_ids', 'attention_mask'], \n",
    "    output_names=['logits'], \n",
    "    dynamic_axes={'input_ids': {0: 'batch_size', 1: 'sequence'}, \n",
    "                  'attention_mask': {0: 'batch_size', 1: 'sequence'}, \n",
    "                  'logits': {0: 'batch_size', 1: 'sequence'}}, \n",
    "    do_constant_folding=True, \n",
    "    opset_version=13, \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "372b0017-6f6b-4c4b-93b4-060a83d246b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "dataset = load_dataset(\"imdb\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ca6efd7-a378-4fbc-8c9a-dd6ef1b88226",
   "metadata": {},
   "outputs": [],
   "source": [
    "import onnx\n",
    "import onnxruntime\n",
    "import torch\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ace73dbb-d017-41ec-a7f8-bbc58879eb93",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForSequenceClassification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d56bc496-ef5f-4a27-93af-5056cc749e02",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(\"distilbert-base-uncased\")\n",
    "print(tokenizer)\n",
    "\n",
    "providers=['CUDAExecutionProvider', 'CPUExecutionProvider']\n",
    "\n",
    "session = onnxruntime.InferenceSession('torch-model.onnx', providers=providers)\n",
    "text=\"This movie is a catastrophe.\"\n",
    "inputs = tokenizer(text, return_tensors=\"np\")\n",
    "print(inputs)\n",
    "\n",
    "result1 = session.run([i.name for i in session.get_outputs()], dict(inputs))\n",
    "print(result1)\n",
    "\n",
    "id2label = {0: \"NEGATIVE\", 1: \"POSITIVE\"}\n",
    "label2id = {\"NEGATIVE\": 0, \"POSITIVE\": 1}\n",
    "predicted_class_id = np.array(result1).argmax().item()\n",
    "print(id2label[predicted_class_id])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3334d19-c11e-444e-91ca-d1baf1a9eb53",
   "metadata": {},
   "outputs": [],
   "source": [
    "#import tensorflow as tf\n",
    "#predictions = tf.math.softmax(result, axis=-1)\n",
    "print(torch.nn.Softmax(dim=1)(torch.tensor(result1[0])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6282eeb6-eae7-48f9-b357-875d41da45a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "text1 = \"This was a masterpiece. Not completely faithful to the books, but enthralling from beginning to end. Might be my favorite of the three.\"\n",
    "text2 = \"This is a catastrophe.\"\n",
    "batch=[text1,text2]\n",
    "inputs = tokenizer(batch, padding=True, truncation=True, max_length=512, return_tensors=\"np\")\n",
    "print(inputs)\n",
    "result2 = session.run([i.name for i in session.get_outputs()], dict(inputs))\n",
    "print(result2)\n",
    "torch.nn.Softmax(dim=1)(torch.tensor(result2[0]))\n",
    "print(np.argmax(torch.nn.Softmax(dim=1)(torch.tensor(result2[0])),axis=1))\n",
    "print([id2label[i.item()] for i in torch.argmax(torch.nn.Softmax(dim=1)(torch.tensor(result2[0])),axis=1)])\n",
    "labels=[id2label[labelid] for labelid in torch.argmax(torch.nn.Softmax(dim=1)(torch.tensor(result2[0])),axis=1).tolist()]\n",
    "print(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe649550-8536-4624-a9c8-9ef3b19b527c",
   "metadata": {},
   "source": [
    "# Upload the model to S3 Bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "787e15cc-cde4-49d5-a16a-f42d6feb6561",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import boto3\n",
    "from boto3 import session\n",
    "\n",
    "key_id = os.environ.get('AWS_ACCESS_KEY_ID')\n",
    "secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY')\n",
    "endpoint_url = os.environ.get('AWS_S3_ENDPOINT')\n",
    "session = boto3.session.Session(aws_access_key_id=key_id, aws_secret_access_key=secret_key)\n",
    "s3_client = boto3.client('s3', aws_access_key_id=key_id, aws_secret_access_key=secret_key,endpoint_url=endpoint_url,verify=False)\n",
    "buckets=s3_client.list_buckets()\n",
    "for bucket in buckets['Buckets']: print(bucket['Name'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e04e2fe-e37d-4565-9bc8-dcad76e1cf37",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(bucket['Name'])\n",
    "modelfile='torch-model.onnx'\n",
    "s3_client.upload_file(modelfile, bucket['Name'],'hf_model.onnx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "425943e1-6c53-4bae-b557-70ea314156a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "[item.get(\"Key\") for item in s3_client.list_objects_v2(Bucket=bucket['Name']).get(\"Contents\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ef500bc-3cdb-4db8-9173-b99ff4035c26",
   "metadata": {},
   "source": [
    "Now manually deploy the model from Data Science Projects\n",
    "\n",
    "---\n",
    "# Submit inferencing request to Deployed model using HTTP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d47b99b4-48a4-4e51-9714-d5e7ab7e862d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "URL='http://modelmesh-serving.huggingface.svc.cluster.local:8008/v2/models/hfmodel/infer' # underscore characters are removed\n",
    "headers = {}\n",
    "payload = {\n",
    "        \"inputs\": [{ \"name\": \"input_ids\", \"shape\": inputs.get('input_ids').shape, \"datatype\": \"INT64\", \"data\": inputs.get('input_ids').tolist()},{ \"name\": \"attention_mask\", \"shape\": inputs.get('attention_mask').shape, \"datatype\": \"INT64\", \"data\": inputs.get('attention_mask').tolist()}]\n",
    "    }\n",
    "print(payload)\n",
    "headers = {\"content-type\": \"application/json\"}\n",
    "res = requests.post(URL, json=payload, headers=headers)\n",
    "print(res)\n",
    "print(res.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "649c4865-15c2-45f1-891e-5ddf6edc5637",
   "metadata": {},
   "outputs": [],
   "source": [
    "result=[np.array(res.json().get('outputs')[0].get('data')).reshape(res.json().get('outputs')[0].get('shape'))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9932ffb6-981e-439f-a568-b1fc3a26b41f",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.nn.Softmax(dim=1)(torch.tensor(result[0]))\n",
    "print(np.argmax(torch.nn.Softmax(dim=1)(torch.tensor(result[0])),axis=1))\n",
    "print('Using item',[id2label[i.item()] for i in torch.argmax(torch.nn.Softmax(dim=1)(torch.tensor(result[0])),axis=1)])\n",
    "labels=[id2label[labelid] for labelid in torch.argmax(torch.nn.Softmax(dim=1)(torch.tensor(result[0])),axis=1).tolist()]\n",
    "print('Using to_list',labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bfbb860-dc0f-4f4c-b9ee-f66468972565",
   "metadata": {},
   "source": [
    "# Submit inferencing request to Deployed model using GRPC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1170e88c-0f43-4ed4-abbc-e0a13f5993dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install grpcio grpcio-tools==1.46.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2fcc311-30d1-4de3-bb99-4ae5545fb35f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!wget https://raw.githubusercontent.com/kserve/kserve/master/docs/predict-api/v2/grpc_predict_v2.proto\n",
    "!wget https://raw.githubusercontent.com/kserve/modelmesh-serving/main/fvt/proto/kfs_inference_v2.proto\n",
    "!python3 -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. ./kfs_inference_v2.proto"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5633d01f-c033-452f-bb42-c376987b0352",
   "metadata": {},
   "outputs": [],
   "source": [
    "payload = { \"model_name\": \"hfmodel\",\n",
    "        \"inputs\": [{ \"name\": \"input_ids\", \"shape\": inputs.get('input_ids').shape, \"datatype\": \"INT64\", \n",
    "                     \"contents\": {\"int64_contents\":[y for x in inputs.get('input_ids').tolist() for y in x]}},\n",
    "                   { \"name\": \"attention_mask\", \"shape\": inputs.get('attention_mask').shape, \"datatype\": \"INT64\", \n",
    "                     \"contents\": {\"int64_contents\":[y for x in inputs.get('attention_mask').tolist() for y in x]}}]\n",
    "    }\n",
    "print(json.dumps(payload))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bb63c22-5b51-456a-88bb-f151e68a6a44",
   "metadata": {},
   "outputs": [],
   "source": [
    "import grpc\n",
    "import kfs_inference_v2_pb2, kfs_inference_v2_pb2_grpc\n",
    "grpc_url=\"modelmesh-serving.huggingface.svc.cluster.local:8033\"\n",
    "request=kfs_inference_v2_pb2.ModelInferRequest(model_name=\"hfmodel\",inputs=payload[\"inputs\"])\n",
    "grpc_channel = grpc.insecure_channel(grpc_url)\n",
    "grpc_stub = kfs_inference_v2_pb2_grpc.GRPCInferenceServiceStub(grpc_channel)\n",
    "response = grpc_stub.ModelInfer(request)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a151141-2a10-4d97-a420-dc28f65432a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(response.outputs),type(response.raw_output_contents))\n",
    "from google.protobuf.json_format import MessageToDict\n",
    "d = MessageToDict(response.outputs[0])\n",
    "print(d)\n",
    "binary_data=bytes([x for x in response.raw_output_contents[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fba3e10-4d74-45dd-b7d0-0c27da4304c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import struct\n",
    "import base64\n",
    "FLOAT = 'f'\n",
    "fmt = '<' + FLOAT * (len(binary_data) // struct.calcsize(FLOAT))\n",
    "numbers = struct.unpack(fmt, binary_data)\n",
    "print(numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a3cac62-37b6-47ec-9815-cad2e930ff88",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.array(numbers).reshape(*[int(i) for i in d.get(\"shape\")])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5af8cd32",
   "metadata": {},
   "source": [
    "Finally, we bring our resource cluster down and release/terminate the associated resources, bringing everything back to the way it was before our cluster was brought up."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b7a183b-5e8e-4adb-b9a6-a349e13512a0",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "As shown in the above example, you can easily run your Huggingface transfer learning tasks easily and natively on CodeFlare. You can scale them from 1 to n GPUs without requiring you to make any significant code changes and leveraging the native Huggingface trainer. \n",
    "\n",
    "Also refer to additional notebooks that showcase other use cases\n",
    "In our next notebook [./02_codeflare_workflows_encoding.ipynb ] shows an sklearn example and how you can leverage workflows to run experiment pipelines and explore multiple pipelines in parallel on CodeFlare cluster. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "630e78d9-ed9e-487f-b933-adde53ae5cbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.down()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "345f8933-5999-4f75-9626-f292d60356df",
   "metadata": {},
   "outputs": [],
   "source": [
    "auth.logout()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.13",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "f9f85f796d01129d0dd105a088854619f454435301f6ffec2fea96ecbd9be4ac"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}