examples/end-to-end-session-based/02-End-to-end-session-based-with-Yoochoose-PyT.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0fb18ebc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2022 NVIDIA Corporation. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# ==============================================================================\n",
    "\n",
    "# Each user is responsible for checking the content of datasets and the\n",
    "# applicable licenses and determining if suitable for the intended use."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "378ff149",
   "metadata": {
    "tags": []
   },
   "source": [
    "<img src=\"https://developer.download.nvidia.com/notebooks/dlsw-notebooks/merlin_transformers4rec_end-to-end-session-based-02-end-to-end-session-based-with-yoochoose-pyt/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# End-to-end session-based recommendations with PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe975de6",
   "metadata": {},
   "source": [
    "In recent years, several deep learning-based algorithms have been proposed for recommendation systems while its adoption in industry deployments have been steeply growing. In particular, NLP inspired approaches have been successfully adapted for sequential and session-based recommendation problems, which are important for many domains like e-commerce, news and streaming media. Session-Based Recommender Systems (SBRS) have been proposed to model the sequence of interactions within the current user session, where a session is a short sequence of user interactions typically bounded by user inactivity. They have recently gained popularity due to their ability to capture short-term or contextual user preferences towards items. \n",
    "\n",
    "The field of NLP has evolved significantly within the last decade, particularly due to the increased usage of deep learning. As a result, state of the art NLP approaches have inspired RecSys practitioners and researchers to adapt those architectures, especially for sequential and session-based recommendation problems. Here, we leverage one of the state-of-the-art Transformer-based architecture, [XLNet](https://arxiv.org/abs/1906.08237) with Masked Language Modeling (MLM) training technique (see our [tutorial](https://github.com/NVIDIA-Merlin/Transformers4Rec/tree/stable/examples/tutorial) for details) for training a session-based model.\n",
    "\n",
    "In this end-to-end-session-based recommnender model example, we use `Transformers4Rec` library, which leverages the popular [HuggingFace’s Transformers](https://github.com/huggingface/transformers) NLP library and make it possible to experiment with cutting-edge implementation of such architectures for sequential and session-based recommendation problems. For detailed explanations of the building blocks of Transformers4Rec meta-architecture visit [getting-started-session-based](https://github.com/NVIDIA-Merlin/Transformers4Rec/tree/stable/examples/getting-started-session-based) and [tutorial](https://github.com/NVIDIA-Merlin/Transformers4Rec/tree/stable/examples/tutorial) example notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7516d46",
   "metadata": {},
   "source": [
    "## 1. Model definition using Transformers4Rec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8642b00",
   "metadata": {},
   "source": [
    "In the previous notebook, we have created sequential features and saved our processed data frames as parquet files. Now we use these processed parquet files to train a session-based recommendation model with the XLNet architecture."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34bc0fdb",
   "metadata": {},
   "source": [
    "### 1.1 Get the schema "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78d52653",
   "metadata": {},
   "source": [
    "The library uses a schema format to configure the input features and automatically creates the necessary layers. This *protobuf* text file contains the description of each input feature by defining: the name, the type, the number of elements of a list column,  the cardinality of a categorical feature and the min and max values of each feature. In addition, the annotation field contains the tags such as specifying the `continuous` and `categorical` features, the `target` column or the `item_id` feature, among others."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83162988-58c7-45b2-9be2-a1039a786553",
   "metadata": {},
   "source": [
    "We create the schema object by reading the processed train parquet file generated by NVTabular pipeline in the previous, 01-ETL-with-NVTabular, notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bdc53326-860b-4cc3-b465-9cf6d4586c53",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\"\n",
    "INPUT_DATA_DIR = os.environ.get(\"INPUT_DATA_DIR\", \"/workspace/data\")\n",
    "OUTPUT_DIR = os.environ.get(\"OUTPUT_DIR\", f\"{INPUT_DATA_DIR}/preproc_sessions_by_day\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c80c3dd1-7390-46b1-97ba-0768b95502ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/merlin/dtypes/mappings/tf.py:52: UserWarning: Tensorflow dtype mappings did not load successfully due to an error: No module named 'tensorflow'\n",
      "  warn(f\"Tensorflow dtype mappings did not load successfully due to an error: {exc.msg}\")\n"
     ]
    }
   ],
   "source": [
    "from merlin.schema import Schema\n",
    "from merlin.io import Dataset\n",
    "\n",
    "train = Dataset(os.path.join(INPUT_DATA_DIR, \"processed_nvt/part_0.parquet\"))\n",
    "schema = train.schema"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ab6407d",
   "metadata": {},
   "source": [
    "We can select the subset of features we want to use for training the model by their tags or their names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f2a57f70",
   "metadata": {},
   "outputs": [],
   "source": [
    "schema = schema.select_by_name(\n",
    "   ['item_id-list', 'category-list', 'product_recency_days_log_norm-list', 'et_dayofweek_sin-list']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaba9e00-23cc-49fb-9c57-1ec4f2a7c7eb",
   "metadata": {},
   "source": [
    "We can print out the schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "25fc2490-1daa-43f7-a052-1eb0d262bd5c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>tags</th>\n",
       "      <th>dtype</th>\n",
       "      <th>is_list</th>\n",
       "      <th>is_ragged</th>\n",
       "      <th>properties.num_buckets</th>\n",
       "      <th>properties.freq_threshold</th>\n",
       "      <th>properties.max_size</th>\n",
       "      <th>properties.cat_path</th>\n",
       "      <th>properties.embedding_sizes.cardinality</th>\n",
       "      <th>properties.embedding_sizes.dimension</th>\n",
       "      <th>properties.domain.min</th>\n",
       "      <th>properties.domain.max</th>\n",
       "      <th>properties.domain.name</th>\n",
       "      <th>properties.value_count.min</th>\n",
       "      <th>properties.value_count.max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>item_id-list</td>\n",
       "      <td>(Tags.CATEGORICAL, Tags.LIST, Tags.ID, Tags.ITEM)</td>\n",
       "      <td>DType(name='int64', element_type=&lt;ElementType....</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>.//categories/unique.item_id.parquet</td>\n",
       "      <td>52742.0</td>\n",
       "      <td>512.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>52741.0</td>\n",
       "      <td>item_id</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>category-list</td>\n",
       "      <td>(Tags.CATEGORICAL, Tags.LIST)</td>\n",
       "      <td>DType(name='int64', element_type=&lt;ElementType....</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>.//categories/unique.category.parquet</td>\n",
       "      <td>337.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>336.0</td>\n",
       "      <td>category</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>product_recency_days_log_norm-list</td>\n",
       "      <td>(Tags.LIST, Tags.CONTINUOUS)</td>\n",
       "      <td>DType(name='float32', element_type=&lt;ElementTyp...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>et_dayofweek_sin-list</td>\n",
       "      <td>(Tags.LIST, Tags.CONTINUOUS)</td>\n",
       "      <td>DType(name='float64', element_type=&lt;ElementTyp...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "[{'name': 'item_id-list', 'tags': {<Tags.CATEGORICAL: 'categorical'>, <Tags.LIST: 'list'>, <Tags.ID: 'id'>, <Tags.ITEM: 'item'>}, 'properties': {'num_buckets': None, 'freq_threshold': 0.0, 'max_size': 0.0, 'cat_path': './/categories/unique.item_id.parquet', 'embedding_sizes': {'cardinality': 52742.0, 'dimension': 512.0}, 'domain': {'min': 0, 'max': 52741, 'name': 'item_id'}, 'value_count': {'min': 0, 'max': 20}}, 'dtype': DType(name='int64', element_type=<ElementType.Int: 'int'>, element_size=64, element_unit=None, signed=True, shape=Shape(dims=(Dimension(min=0, max=None), Dimension(min=0, max=20)))), 'is_list': True, 'is_ragged': True}, {'name': 'category-list', 'tags': {<Tags.CATEGORICAL: 'categorical'>, <Tags.LIST: 'list'>}, 'properties': {'num_buckets': None, 'freq_threshold': 0.0, 'max_size': 0.0, 'cat_path': './/categories/unique.category.parquet', 'embedding_sizes': {'cardinality': 337.0, 'dimension': 42.0}, 'domain': {'min': 0, 'max': 336, 'name': 'category'}, 'value_count': {'min': 0, 'max': 20}}, 'dtype': DType(name='int64', element_type=<ElementType.Int: 'int'>, element_size=64, element_unit=None, signed=True, shape=Shape(dims=(Dimension(min=0, max=None), Dimension(min=0, max=20)))), 'is_list': True, 'is_ragged': True}, {'name': 'product_recency_days_log_norm-list', 'tags': {<Tags.LIST: 'list'>, <Tags.CONTINUOUS: 'continuous'>}, 'properties': {'value_count': {'min': 0, 'max': 20}}, 'dtype': DType(name='float32', element_type=<ElementType.Float: 'float'>, element_size=32, element_unit=None, signed=True, shape=Shape(dims=(Dimension(min=0, max=None), Dimension(min=0, max=20)))), 'is_list': True, 'is_ragged': True}, {'name': 'et_dayofweek_sin-list', 'tags': {<Tags.LIST: 'list'>, <Tags.CONTINUOUS: 'continuous'>}, 'properties': {'value_count': {'min': 0, 'max': 20}}, 'dtype': DType(name='float64', element_type=<ElementType.Float: 'float'>, element_size=64, element_unit=None, signed=True, shape=Shape(dims=(Dimension(min=0, max=None), Dimension(min=0, max=20)))), 'is_list': True, 'is_ragged': True}]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c79ca4c0",
   "metadata": {},
   "source": [
    "### 1.2 Define the end-to-end Session-based Transformer-based recommendation model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e74ac27",
   "metadata": {
    "tags": []
   },
   "source": [
    "For defining a session-based recommendation model, the end-to-end model definition requires four steps:\n",
    "\n",
    "1. Instantiate [TabularSequenceFeatures](https://nvidia-merlin.github.io/Transformers4Rec/stable/api/transformers4rec.tf.features.html?highlight=tabularsequence#transformers4rec.tf.features.sequence.TabularSequenceFeatures) input-module from schema to prepare the embedding tables of categorical variables and project continuous features, if specified. In addition, the module provides different aggregation methods (e.g. 'concat', 'elementwise-sum') to merge input features and generate the sequence of interactions embeddings. The module also supports language modeling tasks to prepare masked labels for training and evaluation (e.g: 'mlm' for masked language modeling) \n",
    "\n",
    "2. Next, we need to define one or multiple prediction tasks. For this demo, we are going to use [NextItemPredictionTask](https://nvidia-merlin.github.io/Transformers4Rec/stable/api/transformers4rec.tf.model.html?highlight=nextitem#transformers4rec.tf.model.prediction_task.NextItemPredictionTask) with `Masked Language modeling`: during training, randomly selected items are masked and predicted using the unmasked sequence items. For inference, it is meant to always predict the next item to be interacted with.\n",
    "\n",
    "3. Then we construct a `transformer_config` based on the architectures provided by [Hugging Face Transformers](https://github.com/huggingface/transformers) framework. </a>\n",
    "\n",
    "4. Finally we link the transformer-body to the inputs and the prediction tasks to get the final pytorch `Model` class.\n",
    "    \n",
    "For more details about the features supported by each sub-module, please check out the library [documentation](https://nvidia-merlin.github.io/Transformers4Rec/) page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d79c0e1b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "Projecting inputs of NextItemPredictionTask to'64' As weight tying requires the input dimension '320' to be equal to the item-id embedding dimension '64'\n"
     ]
    }
   ],
   "source": [
    "from transformers4rec import torch as tr\n",
    "\n",
    "max_sequence_length, d_model = 20, 320\n",
    "# Define input module to process tabular input-features and to prepare masked inputs\n",
    "input_module = tr.TabularSequenceFeatures.from_schema(\n",
    "    schema,\n",
    "    max_sequence_length=max_sequence_length,\n",
    "    continuous_projection=64,\n",
    "    aggregation=\"concat\",\n",
    "    d_output=d_model,\n",
    "    masking=\"mlm\",\n",
    ")\n",
    "\n",
    "# Define Next item prediction-task \n",
    "prediction_task = tr.NextItemPredictionTask(weight_tying=True)\n",
    "\n",
    "# Define the config of the XLNet Transformer architecture\n",
    "transformer_config = tr.XLNetConfig.build(\n",
    "    d_model=d_model, n_head=8, n_layer=2, total_seq_length=max_sequence_length\n",
    ")\n",
    "\n",
    "# Get the end-to-end model \n",
    "model = transformer_config.to_torch_model(input_module, prediction_task)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "847757ca-3c91-4504-b9dc-29127563da01",
   "metadata": {},
   "source": [
    "You can print out the model structure by uncommenting the line below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d9d4a3f7-f495-4504-bfe1-94f1a6cc8328",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12fa0aa0",
   "metadata": {},
   "source": [
    "### 1.3. Daily Fine-Tuning: Training over a time window¶"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdecb72a",
   "metadata": {},
   "source": [
    "Now that the model is defined, we are going to launch training. For that, Transfromers4rec extends HF Transformers Trainer class to adapt the evaluation loop for session-based recommendation task and the calculation of ranking metrics. The original `train()` method is not modified meaning that we leverage the efficient training implementation from that library, which manages, for example, half-precision (FP16) training."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6186d69",
   "metadata": {},
   "source": [
    "#### Set the training arguments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "818197e2",
   "metadata": {},
   "source": [
    "An additional argument `data_loader_engine` is defined to automatically load the features needed for training using the schema. The default value is `merlin` for optimized GPU-based data-loading.  Optionally a `PyarrowDataLoader` (`pyarrow`) can also be used as a basic option, but it is slower and works only for small datasets, as the full data is loaded to CPU memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8fec3739",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE_TRAIN = int(os.environ.get(\"BATCH_SIZE_TRAIN\", \"512\"))\n",
    "BATCH_SIZE_VALID = int(os.environ.get(\"BATCH_SIZE_VALID\", \"256\"))\n",
    "training_args = tr.trainer.T4RecTrainingArguments(\n",
    "            output_dir=\"./tmp\",\n",
    "            max_sequence_length=20,\n",
    "            data_loader_engine='merlin',\n",
    "            num_train_epochs=10, \n",
    "            dataloader_drop_last=False,\n",
    "            per_device_train_batch_size = BATCH_SIZE_TRAIN,\n",
    "            per_device_eval_batch_size = BATCH_SIZE_VALID,\n",
    "            learning_rate=0.0005,\n",
    "            fp16=True,\n",
    "            report_to = [],\n",
    "            logging_steps=200\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "836df805",
   "metadata": {},
   "source": [
    "#### Instantiate the trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3e707cb8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using amp fp16 backend\n"
     ]
    }
   ],
   "source": [
    "recsys_trainer = tr.Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    schema=schema,\n",
    "    compute_metrics=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f116800b",
   "metadata": {},
   "source": [
    "#### Launch daily training and evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "709592a2",
   "metadata": {},
   "source": [
    "In this demo, we will use the `fit_and_evaluate` method that allows us to conduct a time-based finetuning by iteratively training and evaluating using a sliding time window: At each iteration, we use the training data of a specific time index $t$ to train the model; then we evaluate on the validation data of the next index $t + 1$. Particularly, we set start time to 178 and end time to 180.\n",
    "\n",
    "If you have generated a synthetic dataset in the previous notebook, remember to change the values 178 and 180 accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "85be5b12",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "***** Launch training for day 178: *****\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running training *****\n",
      "  Num examples = 28672\n",
      "  Num Epochs = 10\n",
      "  Instantaneous batch size per device = 512\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 512\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 560\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='560' max='560' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [560/560 00:25, Epoch 10/10]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>7.585600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>400</td>\n",
       "      <td>6.608700</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to ./tmp/checkpoint-500\n",
      "Trainer.model is not a `PreTrainedModel`, only saving its state dict.\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='25' max='11' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [11/11 00:33]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running training *****\n",
      "  Num examples = 20480\n",
      "  Num Epochs = 10\n",
      "  Instantaneous batch size per device = 512\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 512\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 400\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "***** Evaluation results for day 179:*****\n",
      "\n",
      " eval_/next-item/avg_precision@10 = 0.07277625054121017\n",
      " eval_/next-item/avg_precision@20 = 0.077287457883358\n",
      " eval_/next-item/ndcg@10 = 0.1008271649479866\n",
      " eval_/next-item/ndcg@20 = 0.11763089150190353\n",
      " eval_/next-item/recall@10 = 0.18959537148475647\n",
      " eval_/next-item/recall@20 = 0.25549131631851196\n",
      "\n",
      "***** Launch training for day 179: *****\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='400' max='400' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [400/400 00:17, Epoch 10/10]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>6.838400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>400</td>\n",
       "      <td>6.304600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "***** Running training *****\n",
      "  Num examples = 16896\n",
      "  Num Epochs = 10\n",
      "  Instantaneous batch size per device = 512\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 512\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 330\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "***** Evaluation results for day 180:*****\n",
      "\n",
      " eval_/next-item/avg_precision@10 = 0.059328265488147736\n",
      " eval_/next-item/avg_precision@20 = 0.06352042406797409\n",
      " eval_/next-item/ndcg@10 = 0.08318208903074265\n",
      " eval_/next-item/ndcg@20 = 0.09845318645238876\n",
      " eval_/next-item/recall@10 = 0.16083915531635284\n",
      " eval_/next-item/recall@20 = 0.2209790199995041\n",
      "\n",
      "***** Launch training for day 180: *****\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='330' max='330' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [330/330 00:14, Epoch 10/10]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>6.708000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "***** Evaluation results for day 181:*****\n",
      "\n",
      " eval_/next-item/avg_precision@10 = 0.12736327946186066\n",
      " eval_/next-item/avg_precision@20 = 0.13500627875328064\n",
      " eval_/next-item/ndcg@10 = 0.16738776862621307\n",
      " eval_/next-item/ndcg@20 = 0.19680777192115784\n",
      " eval_/next-item/recall@10 = 0.29406309127807617\n",
      " eval_/next-item/recall@20 = 0.41187384724617004\n"
     ]
    }
   ],
   "source": [
    "from transformers4rec.torch.utils.examples_utils import fit_and_evaluate\n",
    "start_time_idx = int(os.environ.get(\"START_TIME_INDEX\", \"178\"))\n",
    "end_time_idx = int(os.environ.get(\"END_TIME_INDEX\", \"180\"))\n",
    "OT_results = fit_and_evaluate(recsys_trainer, start_time_index=start_time_idx, end_time_index=end_time_idx, input_dir=OUTPUT_DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25c0aeb3",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Visualize the average of metrics over time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93aaace3",
   "metadata": {},
   "source": [
    "`OT_results` is a list of scores (accuracy metrics) for evaluation based on given start and end time_index. Since in this example we do evaluation on days 179, 180 and 181, we get three metrics in the list one for each day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7c90423f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'indexed_by_time_eval_/next-item/avg_precision@10': [0.07277625054121017,\n",
       "  0.059328265488147736,\n",
       "  0.12736327946186066],\n",
       " 'indexed_by_time_eval_/next-item/avg_precision@20': [0.077287457883358,\n",
       "  0.06352042406797409,\n",
       "  0.13500627875328064],\n",
       " 'indexed_by_time_eval_/next-item/ndcg@10': [0.1008271649479866,\n",
       "  0.08318208903074265,\n",
       "  0.16738776862621307],\n",
       " 'indexed_by_time_eval_/next-item/ndcg@20': [0.11763089150190353,\n",
       "  0.09845318645238876,\n",
       "  0.19680777192115784],\n",
       " 'indexed_by_time_eval_/next-item/recall@10': [0.18959537148475647,\n",
       "  0.16083915531635284,\n",
       "  0.29406309127807617],\n",
       " 'indexed_by_time_eval_/next-item/recall@20': [0.25549131631851196,\n",
       "  0.2209790199995041,\n",
       "  0.41187384724617004]}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "OT_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6eaf2040",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " indexed_by_time_eval_/next-item/avg_precision@10 = 0.08648926516373952\n",
      " indexed_by_time_eval_/next-item/avg_precision@20 = 0.09193805356820424\n",
      " indexed_by_time_eval_/next-item/ndcg@10 = 0.1171323408683141\n",
      " indexed_by_time_eval_/next-item/ndcg@20 = 0.13763061662515005\n",
      " indexed_by_time_eval_/next-item/recall@10 = 0.21483253935972849\n",
      " indexed_by_time_eval_/next-item/recall@20 = 0.2961147278547287\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "# take the average of metric values over time\n",
    "avg_results = {k: np.mean(v) for k,v in OT_results.items()}\n",
    "for key in sorted(avg_results.keys()): \n",
    "    print(\" %s = %s\" % (key, str(avg_results[key]))) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51feee51",
   "metadata": {},
   "source": [
    "#### Trace the model\n",
    "\n",
    "We serve the model with the PyTorch backend that is used to execute TorchScript models. All models created in PyTorch using the python API must be traced/scripted to produce a TorchScript model. For tracing the model, we use [torch.jit.trace](https://pytorch.org/docs/stable/generated/torch.jit.trace.html) api that takes the model as a Python function or torch.nn.Module, and an example input  that will be passed to the function while tracing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8939fd41",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import torch\n",
    "import cudf\n",
    "from merlin.io import Dataset\n",
    "from nvtabular import Workflow\n",
    "\n",
    "from merlin.systems.dag import Ensemble\n",
    "from merlin.systems.dag.ops.pytorch import PredictPyTorch\n",
    "from merlin.systems.dag.ops.workflow import TransformWorkflow\n",
    "from merlin.table import TensorTable, TorchColumn\n",
    "from merlin.table.conversions import convert_col"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42bebd77",
   "metadata": {},
   "source": [
    "Create a dict of tensors to feed it as example inputs in the `torch.jit.trace()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "0b668889",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.read_parquet(os.path.join(INPUT_DATA_DIR, f\"preproc_sessions_by_day/{start_time_idx}/train.parquet\"), columns=model.input_schema.column_names)\n",
    "table = TensorTable.from_df(df.iloc[:100])\n",
    "for column in table.columns:\n",
    "    table[column] = convert_col(table[column], TorchColumn)\n",
    "model_input_dict = table.to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f96c4d8d",
   "metadata": {},
   "source": [
    "Let's now add a top_k parameter to model so that we can return the top k item ids with the highest scores when we serve the model on Triton Inference Server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4a81b2e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "topk = 20\n",
    "model.top_k = topk"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75fa9c78",
   "metadata": {},
   "source": [
    "Let us now trace the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3158138f",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.eval()\n",
    "traced_model = torch.jit.trace(model, model_input_dict, strict=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6978d13",
   "metadata": {},
   "source": [
    "Let's check out the `item_id-list` column in the `model_input_dict` dictionary.\n",
    "\n",
    "The column is represented as values and offsets dictating which values belong to which example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "073f527b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([  605,   879,   743,    91,  4778,  1584,  3447,  8084,  3447,  4019,\n",
       "          743,  4778,   185, 12289,  2066,   431,     7,    31,     7,   158,\n",
       "         1988,  2591, 10856,  8218,  4211,  8712,  4243,    82,   113,  4243,\n",
       "         5733,  6811,    34,     7,    74,   665,  2313,  7125,  9114,   446,\n",
       "         1158,   775,   686,   431,  1946,   476,   598,   290,   167,    30,\n",
       "          343,   290,    34,   424,   167,   481,  2773,   289,   963,  4002,\n",
       "         2051,  3275,   500,  1220,   396,  1637, 11840, 10715, 11108,   290,\n",
       "          167,   651,  1086,   303,    89,   651,   215,   305,   178,   318,\n",
       "          424,     7,  3819,   932,  2187,  1086,   207,   688,  3832,   688,\n",
       "          203,    21,    44,    21,  2035, 10458,    21,    22,  1127,  2816,\n",
       "         4211, 34265,   831,   775,   621,  2051,  1988,  1080,  4714,  1337,\n",
       "          662,   290,   431,   864,  4830,  5787, 19157, 17271, 23366,  4210,\n",
       "         3652,  1038,  4771,   225,   278,  1021,   651,   167,  1355,   207,\n",
       "         1890,  2474,  1698,   998,   481,   775,  3842,  4317,  3842,  1231,\n",
       "         3842,  1698,  3810,   476,   982,   805,   314,   614,  1220,  1335,\n",
       "         1942,  2889,  2627,  1335,  2690,   805,   476,   982,   314,   805,\n",
       "         1220,   207,   652,   430,   606,   102,   414,  1966,   628,   815,\n",
       "          628,   815,  4714,  3676,  2790,  3770,  1284,  9541,  1252,   314,\n",
       "          686,  2498,   396,   846,  3463,  2714,  5078,   389,   341,   298,\n",
       "          389,  9642,    47,    62,   823,    62,   603,  2271,   720,  3275,\n",
       "         2557,     3,    62,    25,    97,    62,   424,    62,   476,  3461,\n",
       "         2694,  3461,  3045,  2557,  3989,   993,  1604,   123,  2705,  2788,\n",
       "         3136,   551,   517,    45,  1552,  2703,   207,  1763,    32,   475,\n",
       "          482,   199,   475,  2705,  2394,  1026, 20034,    73,  1335,   225,\n",
       "         3461,  4775,  2051,  6486, 15954,   423,  1489,  2347,  4471,  2549,\n",
       "          572,  1771,  1325,   454,   838,   124,   639,  4760,  3553,  6826,\n",
       "         2741,  5348,  5391,  1170,  4101,  1231,   805,  3589,  2450,   186,\n",
       "           17,   644,   275,   687, 18093, 10458,   610,  2970,  3481,  2970,\n",
       "           38,   610,  2970,   610,    23,  8313,   258,    38,    23,  5362,\n",
       "         7187,  7381,  6053,  7257, 13405,   558,   161,  1665,  4376,  3485,\n",
       "          686,   652,   446,   430,   446,   775,   652,  4285,  1739,  3856,\n",
       "          226,   211,  7246,  6732,   772,  1988,   158,   805,   443,   805,\n",
       "         2092,  1170,  2092,  1170,  3485,  4376,  3485,   446,   430,   431,\n",
       "          424,  1698,  1394,  1799,  2754,   207,  1154, 21589,  2190,  3705,\n",
       "         4464,  5817,  7558,   508,  1798,   815,   628,  2017,   856,  1890,\n",
       "          225,   598,    38,   598, 16534, 10256,     3,  2652,  4029,  2557,\n",
       "         2789,  6380,  1831,  1071,    31,   313,   446,  1086,  1570,  2223,\n",
       "         2665,  1951,  2099,  1673,   225,  4337,   652,   998,  1158,   831,\n",
       "          801,   598,  1086, 12431,   416, 12431,   652,   801,  1757,  1379,\n",
       "         1414,   634,  2035,  7933,  6035,  6361,  4663,   577,  4663,   577,\n",
       "         4663], device='cuda:0')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_input_dict['item_id-list__values']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3eb2d6e8",
   "metadata": {},
   "source": [
    "And here are the offsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "797f95fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([  0,  12,  14,  16,  19,  21,  23,  26,  32,  35,  45,  47,  49,  51,\n",
       "         56,  59,  61,  66,  69,  72,  76,  78,  80,  83,  87,  91,  94,  96,\n",
       "         98, 100, 102, 104, 107, 111, 113, 122, 124, 128, 130, 133, 136, 141,\n",
       "        143, 161, 164, 167, 172, 176, 180, 182, 184, 191, 193, 196, 199, 201,\n",
       "        209, 214, 234, 237, 243, 245, 265, 267, 274, 276, 281, 289, 295, 298,\n",
       "        300, 305, 307, 312, 314, 317, 320, 324, 327, 331, 335, 337, 339, 343,\n",
       "        345, 348, 351, 354, 356, 360, 362, 364, 367, 374, 376, 381, 383, 386,\n",
       "        388, 396, 401], device='cuda:0', dtype=torch.int32)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_input_dict['item_id-list__offsets']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "228f090b",
   "metadata": {},
   "source": [
    "Let's create a folder that we can store the exported models and the config files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "20a93b02",
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "ens_model_path = os.environ.get(\"ens_model_path\", f\"{INPUT_DATA_DIR}/models\")\n",
    "# Make sure we have a clean stats space for Dask\n",
    "if os.path.isdir(ens_model_path):\n",
    "    shutil.rmtree(ens_model_path)\n",
    "os.mkdir(ens_model_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "55c03a0a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "workflow = Workflow.load(os.path.join(INPUT_DATA_DIR, \"workflow_etl\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "18eaa079",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "torch_op = workflow.input_schema.column_names >> TransformWorkflow(workflow) >> PredictPyTorch(\n",
    "    traced_model, model.input_schema, model.output_schema\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bf0fe7e",
   "metadata": {},
   "source": [
    "The last step is to create the ensemble artifacts that Triton Inference Server can consume. To make these artifacts, we import the Ensemble class. The class is responsible for interpreting the graph and exporting the correct files for the server.\n",
    "\n",
    "When we create an `Ensemble` object we supply the graph and a schema representing the starting input of the graph. The inputs to the ensemble graph are the inputs to the first operator of out graph. After we created the Ensemble we export the graph, supplying an export path for the `ensemble.export` function. This returns an ensemble config which represents the entire inference pipeline and a list of node-specific configs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "64178cff",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/merlin/systems/dag/node.py:100: UserWarning: Operator 'TransformWorkflow' is producing the output column 'session_id', which is not being used by any downstream operator in the ensemble graph.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.8/dist-packages/merlin/systems/dag/node.py:100: UserWarning: Operator 'TransformWorkflow' is producing the output column 'item_id-count', which is not being used by any downstream operator in the ensemble graph.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.8/dist-packages/merlin/systems/dag/node.py:100: UserWarning: Operator 'TransformWorkflow' is producing the output column 'day_index', which is not being used by any downstream operator in the ensemble graph.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "ensemble = Ensemble(torch_op, workflow.input_schema)\n",
    "ens_config, node_configs = ensemble.export(ens_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2e4430b",
   "metadata": {},
   "source": [
    "## 2. Serving Ensemble Model to the Triton Inference Server"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b1615cc",
   "metadata": {},
   "source": [
    "NVIDIA [Triton Inference Server (TIS)](https://github.com/triton-inference-server/server) simplifies the deployment of AI models at scale in production. TIS provides a cloud and edge inferencing solution optimized for both CPUs and GPUs. It supports a number of different machine learning frameworks such as TensorFlow and PyTorch.\n",
    "\n",
    "The last step of a machine learning (ML)/deep learning (DL) pipeline is to deploy the ETL workflow and saved model to production. In the production setting, we want to transform the input data as done during training (ETL). We need to apply the same mean/std for continuous features and use the same categorical mapping to convert the categories to continuous integer before we use the DL model for a prediction. Therefore, we deploy the NVTabular workflow with the PyTorch model as an ensemble model to Triton Inference. The ensemble model guarantees that the same transformation is applied to the raw inputs.\n",
    "\n",
    "\n",
    "In this section, you will learn how to\n",
    "- to deploy saved NVTabular and PyTorch models to Triton Inference Server \n",
    "- send requests for predictions and get responses."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccafebdc",
   "metadata": {},
   "source": [
    "### 2.1 Starting Triton Server"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adb24c60",
   "metadata": {},
   "source": [
    "It is time to deploy all the models as an ensemble model to Triton Inference Serve TIS. After we export the ensemble, we are ready to start the TIS. You can start triton server by using the following command on your terminal:\n",
    "\n",
    "`tritonserver --model-repository=<ensemble_export_path>`\n",
    "\n",
    "For the `--model-repository` argument, specify the same path as the export_path that you specified previously in the `ensemble.export` method. This command will launch the server and load all the models to the server. Once all the models are loaded successfully, you should see READY status printed out in the terminal for each loaded model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e76b0ed",
   "metadata": {},
   "source": [
    "### 2.2. Connect to the Triton Inference Server and check if the server is alive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "dcfb4acd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "client created.\n",
      "GET /v2/health/live, headers None\n",
      "<HTTPSocketPoolResponse status=200 headers={'content-length': '0', 'content-type': 'text/plain'}>\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/tritonhttpclient/__init__.py:31: DeprecationWarning: The package `tritonhttpclient` is deprecated and will be removed in a future version. Please use instead `tritonclient.http`\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tritonhttpclient\n",
    "try:\n",
    "    triton_client = tritonhttpclient.InferenceServerClient(url=\"localhost:8000\", verbose=True)\n",
    "    print(\"client created.\")\n",
    "except Exception as e:\n",
    "    print(\"channel creation failed: \" + str(e))\n",
    "triton_client.is_server_live()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5067e84",
   "metadata": {},
   "source": [
    "### 2.3. Load raw data for inference\n",
    "We select the last 50 interactions and filter out sessions with less than 2 interactions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "61a27083",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "interactions_merged_df = pd.read_parquet(os.path.join(INPUT_DATA_DIR, \"interactions_merged_df.parquet\"))\n",
    "interactions_merged_df = interactions_merged_df.sort_values('timestamp')\n",
    "batch = interactions_merged_df[-50:]\n",
    "sessions_to_use = batch.session_id.value_counts()\n",
    "filtered_batch = batch[batch.session_id.isin(sessions_to_use[sessions_to_use.values>1].index.values)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74e022da",
   "metadata": {},
   "source": [
    "### 2.5. Send the request to triton server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "e98c6068",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "POST /v2/repository/index, headers None\n",
      "\n",
      "<HTTPSocketPoolResponse status=200 headers={'content-type': 'application/json', 'content-length': '188'}>\n",
      "bytearray(b'[{\"name\":\"0_transformworkflowtriton\",\"version\":\"1\",\"state\":\"READY\"},{\"name\":\"1_predictpytorchtriton\",\"version\":\"1\",\"state\":\"READY\"},{\"name\":\"executor_model\",\"version\":\"1\",\"state\":\"READY\"}]')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'name': '0_transformworkflowtriton', 'version': '1', 'state': 'READY'},\n",
       " {'name': '1_predictpytorchtriton', 'version': '1', 'state': 'READY'},\n",
       " {'name': 'executor_model', 'version': '1', 'state': 'READY'}]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "triton_client.get_model_repository_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8083b9ab",
   "metadata": {},
   "source": [
    "If all models are loaded successfully, you should be seeing `READY` status next to each model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "eea28970",
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "cannot convert NA to integer",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[26], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mmerlin\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01msystems\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtriton\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mutils\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m send_triton_request\n\u001b[0;32m----> 2\u001b[0m response \u001b[38;5;241m=\u001b[39m \u001b[43msend_triton_request\u001b[49m\u001b[43m(\u001b[49m\u001b[43mworkflow\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43minput_schema\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mfiltered_batch\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmodel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43moutput_schema\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcolumn_names\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m      3\u001b[0m \u001b[38;5;28mprint\u001b[39m(response)\n",
      "File \u001b[0;32m/usr/local/lib/python3.8/dist-packages/merlin/systems/triton/utils.py:226\u001b[0m, in \u001b[0;36msend_triton_request\u001b[0;34m(schema, inputs, outputs_list, client, endpoint, request_id, triton_model)\u001b[0m\n\u001b[1;32m    224\u001b[0m     triton_inputs \u001b[38;5;241m=\u001b[39m triton\u001b[38;5;241m.\u001b[39mconvert_table_to_triton_input(schema, inputs, grpcclient\u001b[38;5;241m.\u001b[39mInferInput)\n\u001b[1;32m    225\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 226\u001b[0m     triton_inputs \u001b[38;5;241m=\u001b[39m \u001b[43mtriton\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mconvert_df_to_triton_input\u001b[49m\u001b[43m(\u001b[49m\u001b[43mschema\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mgrpcclient\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mInferInput\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    228\u001b[0m outputs \u001b[38;5;241m=\u001b[39m [grpcclient\u001b[38;5;241m.\u001b[39mInferRequestedOutput(col) \u001b[38;5;28;01mfor\u001b[39;00m col \u001b[38;5;129;01min\u001b[39;00m outputs_list]\n\u001b[1;32m    230\u001b[0m response \u001b[38;5;241m=\u001b[39m client\u001b[38;5;241m.\u001b[39minfer(triton_model, triton_inputs, request_id\u001b[38;5;241m=\u001b[39mrequest_id, outputs\u001b[38;5;241m=\u001b[39moutputs)\n",
      "File \u001b[0;32m/usr/local/lib/python3.8/dist-packages/merlin/systems/triton/__init__.py:88\u001b[0m, in \u001b[0;36mconvert_df_to_triton_input\u001b[0;34m(schema, batch, input_class, dtype)\u001b[0m\n\u001b[1;32m     68\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mconvert_df_to_triton_input\u001b[39m(schema, batch, input_class\u001b[38;5;241m=\u001b[39mgrpcclient\u001b[38;5;241m.\u001b[39mInferInput, dtype\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mint32\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n\u001b[1;32m     69\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m     70\u001b[0m \u001b[38;5;124;03m    Convert a dataframe to a set of Triton inputs\u001b[39;00m\n\u001b[1;32m     71\u001b[0m \n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     86\u001b[0m \u001b[38;5;124;03m        A list of Triton inputs of the requested input class\u001b[39;00m\n\u001b[1;32m     87\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m---> 88\u001b[0m     df_dict \u001b[38;5;241m=\u001b[39m \u001b[43m_convert_df_to_dict\u001b[49m\u001b[43m(\u001b[49m\u001b[43mschema\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbatch\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdtype\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     89\u001b[0m     inputs \u001b[38;5;241m=\u001b[39m [\n\u001b[1;32m     90\u001b[0m         _convert_array_to_triton_input(col_name, col_values, input_class)\n\u001b[1;32m     91\u001b[0m         \u001b[38;5;28;01mfor\u001b[39;00m col_name, col_values \u001b[38;5;129;01min\u001b[39;00m df_dict\u001b[38;5;241m.\u001b[39mitems()\n\u001b[1;32m     92\u001b[0m     ]\n\u001b[1;32m     93\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m inputs\n",
      "File \u001b[0;32m/usr/local/lib/python3.8/dist-packages/merlin/systems/triton/__init__.py:183\u001b[0m, in \u001b[0;36m_convert_df_to_dict\u001b[0;34m(schema, batch, dtype)\u001b[0m\n\u001b[1;32m    181\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    182\u001b[0m         values \u001b[38;5;241m=\u001b[39m col\u001b[38;5;241m.\u001b[39mvalues \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(col, pd\u001b[38;5;241m.\u001b[39mSeries) \u001b[38;5;28;01melse\u001b[39;00m col\u001b[38;5;241m.\u001b[39mvalues_host\n\u001b[0;32m--> 183\u001b[0m         values \u001b[38;5;241m=\u001b[39m \u001b[43mvalues\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mreshape\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mshape\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mastype\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcol_schema\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdtype\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mto_numpy\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    184\u001b[0m         df_dict[col_name] \u001b[38;5;241m=\u001b[39m values\n\u001b[1;32m    185\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m df_dict\n",
      "File \u001b[0;32m/usr/local/lib/python3.8/dist-packages/pandas/core/arrays/masked.py:471\u001b[0m, in \u001b[0;36mBaseMaskedArray.astype\u001b[0;34m(self, dtype, copy)\u001b[0m\n\u001b[1;32m    469\u001b[0m \u001b[38;5;66;03m# to_numpy will also raise, but we get somewhat nicer exception messages here\u001b[39;00m\n\u001b[1;32m    470\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m is_integer_dtype(dtype) \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_hasna:\n\u001b[0;32m--> 471\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mcannot convert NA to integer\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m    472\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m is_bool_dtype(dtype) \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_hasna:\n\u001b[1;32m    473\u001b[0m     \u001b[38;5;66;03m# careful: astype_nansafe converts np.nan to True\u001b[39;00m\n\u001b[1;32m    474\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mcannot convert float NaN to bool\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
      "\u001b[0;31mValueError\u001b[0m: cannot convert NA to integer"
     ]
    }
   ],
   "source": [
    "from merlin.systems.triton.utils import send_triton_request\n",
    "response = send_triton_request(workflow.input_schema, filtered_batch, model.output_schema.column_names)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ac9f174",
   "metadata": {},
   "source": [
    "The response contains scores (logits) for each of the returned items and the returned it ids."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "928080a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['item_id_scores', 'item_ids'])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76f10a13",
   "metadata": {},
   "source": [
    "Just as we requested by setting the `top_k` parameter, only 20 predictions are returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "811bfa29",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15, 20)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response['item_ids'].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f576abb9",
   "metadata": {},
   "source": [
    "\n",
    "This is the end of the tutorial. You successfully\n",
    "\n",
    "- performed feature engineering with NVTabular\n",
    "- trained transformer architecture based session-based recommendation models with Transformers4Rec\n",
    "- deployed a trained model to Triton Inference Server, sent request and got responses from the server."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68dd532f",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30f9073e",
   "metadata": {},
   "source": [
    "- Merlin Transformers4rec: https://github.com/NVIDIA-Merlin/Transformers4Rec\n",
    "\n",
    "- Merlin NVTabular: https://github.com/NVIDIA-Merlin/NVTabular/tree/stable/nvtabular\n",
    "\n",
    "- Merlin Dataloader: https://github.com/NVIDIA-Merlin/dataloader\n",
    "\n",
    "- Triton inference server: https://github.com/triton-inference-server"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "d795d7ca5d3ec3bd6293cc80853205a74ce23d484a2b8f537732a716747107c8"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}