examples/getting-started-session-based/02-session-based-XLNet-with-PyT.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "97250792",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2022 NVIDIA Corporation. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# ==============================================================================\n",
    "\n",
    "# Each user is responsible for checking the content of datasets and the\n",
    "# applicable licenses and determining if suitable for the intended use."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a2228da",
   "metadata": {},
   "source": [
    "<img src=\"https://developer.download.nvidia.com/notebooks/dlsw-notebooks/merlin_transformers4rec_getting-started-session-based-02-session-based-xlnet-with-pyt/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# Session-based Recommendation with XLNET"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad8aa005",
   "metadata": {},
   "source": [
    "This notebook is created using the latest stable [merlin-pytorch](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-pytorch/tags) container.\n",
    "\n",
    "In this notebook we introduce the [Transformers4Rec](https://github.com/NVIDIA-Merlin/Transformers4Rec) library for sequential and session-based recommendation. This notebook uses the PyTorch API. Transformers4Rec integrates with the popular [HuggingFace’s Transformers](https://github.com/huggingface/transformers) and makes it possible to experiment with a cutting-edge implementation of the latest NLP Transformer architectures.  \n",
    "\n",
    "We demonstrate how to build a session-based recommendation model with the [XLNET](https://arxiv.org/abs/1906.08237) Transformer architecture. The XLNet architecture was designed to leverage the best of both auto-regressive language modeling and auto-encoding with its Permutation Language Modeling training method. In this example we will use XLNET with masked language modeling (MLM) training method, which showed very promising results in the experiments conducted in our [ACM RecSys'21 paper](https://github.com/NVIDIA-Merlin/publications/blob/main/2021_acm_recsys_transformers4rec/recsys21_transformers4rec_paper.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82b3e715",
   "metadata": {},
   "source": [
    "In the previous notebook we went through our ETL pipeline with the NVTabular library, and created sequential features to be used in training a session-based recommendation model. In this notebook we will learn:\n",
    "\n",
    "- Accelerating data loading of parquet files with multiple features on PyTorch using NVTabular library\n",
    "- Training and evaluating a Transformer-based (XLNET-MLM) session-based recommendation model with multiple features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "359e04dc",
   "metadata": {},
   "source": [
    "## Build a DL model with Transformers4Rec library  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2da66684",
   "metadata": {},
   "source": [
    "Transformers4Rec supports multiple input features and provides configurable building blocks that can be easily combined for custom architectures:\n",
    "\n",
    "- [TabularSequenceFeatures](https://nvidia-merlin.github.io/Transformers4Rec/stable/api/transformers4rec.torch.html#transformers4rec.torch.TabularSequenceFeatures) class that reads from schema and creates an input block. This input module combines different types of features (continuous, categorical & text) to a sequence.\n",
    "-  [MaskSequence](https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/stable/transformers4rec/torch/masking.py) to define masking schema and prepare the masked inputs and labels for the selected LM task.\n",
    "-  [TransformerBlock](https://nvidia-merlin.github.io/Transformers4Rec/stable/api/transformers4rec.torch.html#transformers4rec.torch.TransformerBlock) class that supports HuggingFace Transformers for session-based and sequential-based recommendation models.\n",
    "-  [SequentialBlock](https://nvidia-merlin.github.io/Transformers4Rec/stable/api/transformers4rec.torch.html#transformers4rec.torch.SequentialBlock) creates the body by mimicking [torch.nn.sequential](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html) class. It is designed to define our model as a sequence of layers.\n",
    "-  [Head](https://nvidia-merlin.github.io/Transformers4Rec/stable/api/transformers4rec.torch.html#transformers4rec.torch.Head) where we define the prediction task of the model.\n",
    "-  [NextItemPredictionTask](https://nvidia-merlin.github.io/Transformers4Rec/stable/api/transformers4rec.torch.html#transformers4rec.torch.NextItemPredictionTask) is the class to support next item prediction task.\n",
    "- [Trainer](https://nvidia-merlin.github.io/Transformers4Rec/stable/api/transformers4rec.torch.html#transformers4rec.torch.Trainer) extends the `Trainer` class from HF transformers and manages the model training and evaluation.\n",
    "\n",
    "You can check the [full documentation](https://nvidia-merlin.github.io/Transformers4Rec/) of Transformers4Rec if needed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "173d5b3e",
   "metadata": {},
   "source": [
    "Figure 1 illustrates Transformers4Rec meta-architecture and how each module/block interacts with each other."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f17f66e",
   "metadata": {},
   "source": [
    "![tf4rec_meta](images/tf4rec_meta2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "599efc90",
   "metadata": {},
   "source": [
    "### Import required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "3ba89970",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\"\n",
    "\n",
    "import glob\n",
    "import torch \n",
    "\n",
    "from transformers4rec import torch as tr\n",
    "from transformers4rec.torch.ranking_metric import NDCGAt, AvgPrecisionAt, RecallAt\n",
    "from transformers4rec.torch.utils.examples_utils import wipe_memory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aea2a0c5",
   "metadata": {},
   "source": [
    "Transformers4Rec library relies on a schema object to automatically build all necessary layers to represent, normalize and aggregate input features. As you can see below, `schema.pb` is a protobuf file that contains metadata including statistics about features such as cardinality, min and max values and also tags features based on their characteristics and dtypes (e.g., categorical, continuous, list, integer)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a510b6ef",
   "metadata": {},
   "source": [
    "### Set the schema object"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30a0518a-eb01-4ac4-9c6d-36b328985765",
   "metadata": {},
   "source": [
    "We create the schema object by reading the processed train parquet file generated by NVTabular pipeline in the previous, `01-ETL-with-NVTabular`, notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "c2fcfcaa-f228-4ee6-808c-1ec03f481b79",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "INPUT_DATA_DIR = os.environ.get(\"INPUT_DATA_DIR\", \"/workspace/data\")\n",
    "OUTPUT_DIR = os.environ.get(\"OUTPUT_DIR\", f\"{INPUT_DATA_DIR}/sessions_by_day\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "9d1299fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/workspace/merlin/core/merlin/schema/tags.py:149: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from merlin.schema import Schema\n",
    "from merlin.io import Dataset\n",
    "\n",
    "train = Dataset(os.path.join(INPUT_DATA_DIR, \"processed_nvt/part_0.parquet\"))\n",
    "schema = train.schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "b4f426f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can select a subset of features for training\n",
    "schema = schema.select_by_name(['item_id-list', \n",
    "                                'category-list', \n",
    "                                'weekday_sin-list',\n",
    "                                'age_days-list'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a7ba2ad-784b-433a-976e-6c5ba26d8b18",
   "metadata": {},
   "source": [
    "Let's print out the schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "e5034fad-bda3-48dc-91c6-a3c8076595e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>tags</th>\n",
       "      <th>dtype</th>\n",
       "      <th>is_list</th>\n",
       "      <th>is_ragged</th>\n",
       "      <th>properties.embedding_sizes.dimension</th>\n",
       "      <th>properties.embedding_sizes.cardinality</th>\n",
       "      <th>properties.num_buckets</th>\n",
       "      <th>properties.start_index</th>\n",
       "      <th>properties.max_size</th>\n",
       "      <th>properties.cat_path</th>\n",
       "      <th>properties.freq_threshold</th>\n",
       "      <th>properties.domain.min</th>\n",
       "      <th>properties.domain.max</th>\n",
       "      <th>properties.domain.name</th>\n",
       "      <th>properties.value_count.min</th>\n",
       "      <th>properties.value_count.max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>item_id-list</td>\n",
       "      <td>(Tags.CATEGORICAL, Tags.ITEM_ID, Tags.ITEM, Ta...</td>\n",
       "      <td>DType(name='int64', element_type=&lt;ElementType....</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>52.0</td>\n",
       "      <td>493.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>.//categories/unique.item_id.parquet</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>492.0</td>\n",
       "      <td>item_id</td>\n",
       "      <td>2</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>category-list</td>\n",
       "      <td>(Tags.LIST, Tags.CATEGORICAL)</td>\n",
       "      <td>DType(name='int64', element_type=&lt;ElementType....</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>29.0</td>\n",
       "      <td>173.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>.//categories/unique.category.parquet</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>172.0</td>\n",
       "      <td>category</td>\n",
       "      <td>2</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>weekday_sin-list</td>\n",
       "      <td>(Tags.LIST, Tags.CONTINUOUS)</td>\n",
       "      <td>DType(name='float32', element_type=&lt;ElementTyp...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>age_days-list</td>\n",
       "      <td>(Tags.LIST, Tags.CONTINUOUS)</td>\n",
       "      <td>DType(name='float32', element_type=&lt;ElementTyp...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "[{'name': 'item_id-list', 'tags': {<Tags.CATEGORICAL: 'categorical'>, <Tags.ITEM_ID: 'item_id'>, <Tags.ITEM: 'item'>, <Tags.LIST: 'list'>, <Tags.ID: 'id'>}, 'properties': {'embedding_sizes': {'dimension': 52.0, 'cardinality': 493.0}, 'num_buckets': None, 'start_index': 0.0, 'max_size': 0.0, 'cat_path': './/categories/unique.item_id.parquet', 'freq_threshold': 0.0, 'domain': {'min': 0, 'max': 492, 'name': 'item_id'}, 'value_count': {'min': 2, 'max': 16}}, 'dtype': DType(name='int64', element_type=<ElementType.Int: 'int'>, element_size=64, element_unit=None, signed=True, shape=Shape(dims=(Dimension(min=0, max=None), Dimension(min=2, max=16)))), 'is_list': True, 'is_ragged': True}, {'name': 'category-list', 'tags': {<Tags.LIST: 'list'>, <Tags.CATEGORICAL: 'categorical'>}, 'properties': {'cat_path': './/categories/unique.category.parquet', 'max_size': 0.0, 'num_buckets': None, 'start_index': 0.0, 'freq_threshold': 0.0, 'embedding_sizes': {'dimension': 29.0, 'cardinality': 173.0}, 'domain': {'min': 0, 'max': 172, 'name': 'category'}, 'value_count': {'min': 2, 'max': 16}}, 'dtype': DType(name='int64', element_type=<ElementType.Int: 'int'>, element_size=64, element_unit=None, signed=True, shape=Shape(dims=(Dimension(min=0, max=None), Dimension(min=2, max=16)))), 'is_list': True, 'is_ragged': True}, {'name': 'weekday_sin-list', 'tags': {<Tags.LIST: 'list'>, <Tags.CONTINUOUS: 'continuous'>}, 'properties': {'value_count': {'min': 2, 'max': 16}}, 'dtype': DType(name='float32', element_type=<ElementType.Float: 'float'>, element_size=32, element_unit=None, signed=True, shape=Shape(dims=(Dimension(min=0, max=None), Dimension(min=2, max=16)))), 'is_list': True, 'is_ragged': True}, {'name': 'age_days-list', 'tags': {<Tags.LIST: 'list'>, <Tags.CONTINUOUS: 'continuous'>}, 'properties': {'value_count': {'min': 2, 'max': 16}}, 'dtype': DType(name='float32', element_type=<ElementType.Float: 'float'>, element_size=32, element_unit=None, signed=True, shape=Shape(dims=(Dimension(min=0, max=None), Dimension(min=2, max=16)))), 'is_list': True, 'is_ragged': True}]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06cacefa",
   "metadata": {},
   "source": [
    "### Define the sequential input module"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c203795d",
   "metadata": {},
   "source": [
    "Below we define our `input` block using the `TabularSequenceFeatures` [class](https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/stable/transformers4rec/torch/features/sequence.py#L97). The `from_schema()` method processes the schema and creates the necessary layers to represent features and aggregate them. It keeps only features tagged as `categorical` and `continuous` and supports data aggregation methods like `concat` and `elementwise-sum`. It also supports data augmentation techniques like stochastic swap noise. It outputs an interaction representation after combining all features and also the input mask according to the training task (more on this later).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a46bc24a",
   "metadata": {},
   "source": [
    "The `max_sequence_length` argument defines the maximum sequence length of our sequential input, and if `continuous_projection` argument is set, all numerical features are concatenated and projected by an MLP block so that continuous features are represented by a vector of size defined by user, which is `64` in this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "b38d30d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tr.TabularSequenceFeatures.from_schema(\n",
    "        schema,\n",
    "        max_sequence_length=20,\n",
    "        continuous_projection=64,\n",
    "        masking=\"mlm\",\n",
    "        d_output=100,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3feb554",
   "metadata": {},
   "source": [
    "The output of the `TabularSequenceFeatures` module is the sequence of interactions embedding vectors defined in the following steps:\n",
    "- 1. Create sequence inputs: If the schema contains non sequential features, expand each feature to a sequence by repeating the value as many times as the `max_sequence_length` value.  \n",
    "- 2. Get a representation vector of categorical features: Project each sequential categorical feature using the related embedding table. The resulting tensor is of shape (bs, max_sequence_length, embed_dim).\n",
    "- 3. Project scalar values if `continuous_projection` is set : Apply an MLP layer with hidden size equal to `continuous_projection` vector size value. The resulting tensor is of shape (batch_size, max_sequence_length, continuous_projection).\n",
    "- 4. Aggregate the list of features vectors to represent each interaction in the sequence with one vector: For example, `concat` will concat all vectors based on the last dimension `-1` and the resulting tensor will be of shape (batch_size, max_sequence_length, D) where D is the sum over all embedding dimensions and the value of continuous_projection. \n",
    "- 5. If masking schema is set (needed only for the `NextItemPredictionTask` training), the masked labels are derived from the sequence of raw item-ids and the sequence of interactions embeddings are processed to mask information about the masked positions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "831d656a",
   "metadata": {},
   "source": [
    "### Define the Transformer block"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "635cd0f3",
   "metadata": {},
   "source": [
    "In the next cell, the whole model is build with a few lines of code. \n",
    "Here is a brief explanation of the main classes:  \n",
    "- [XLNetConfig](https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/stable/transformers4rec/config/transformer.py#L261) - We have injected in the HF transformers config classes like `XLNetConfig`the `build()` method that provides default configuration to Transformer architectures for session-based recommendation. Here we use it to instantiate and configure an XLNET architecture.  \n",
    "- [TransformerBlock](https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/stable/transformers4rec/torch/block/transformer.py#L57) class integrates with HF Transformers, which are made available as a sequence processing module for session-based and sequential-based recommendation models.  \n",
    "- [NextItemPredictionTask](https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/stable/transformers4rec/torch/model/prediction_task.py#L110) supports the next-item prediction task. We also support other predictions [tasks](https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/stable/transformers4rec/torch/model/prediction_task.py), like classification and regression for the whole sequence. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "ed749ca8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define XLNetConfig class and set default parameters for HF XLNet config  \n",
    "transformer_config = tr.XLNetConfig.build(\n",
    "    d_model=64, n_head=4, n_layer=2, total_seq_length=20\n",
    ")\n",
    "# Define the model block including: inputs, masking, projection and transformer block.\n",
    "body = tr.SequentialBlock(\n",
    "    inputs, tr.MLPBlock([64]), tr.TransformerBlock(transformer_config, masking=inputs.masking)\n",
    ")\n",
    "\n",
    "# Define the evaluation top-N metrics and the cut-offs\n",
    "metrics = [NDCGAt(top_ks=[20, 40], labels_onehot=True),  \n",
    "           RecallAt(top_ks=[20, 40], labels_onehot=True)]\n",
    "\n",
    "# Define a head related to next item prediction task \n",
    "head = tr.Head(\n",
    "    body,\n",
    "    tr.NextItemPredictionTask(weight_tying=True, \n",
    "                              metrics=metrics),\n",
    "    inputs=inputs,\n",
    ")\n",
    "\n",
    "# Get the end-to-end Model class \n",
    "model = tr.Model(head)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a57335ff",
   "metadata": {},
   "source": [
    "Note that we can easily define an RNN-based model inside the `SequentialBlock` instead of a Transformer-based model. You can explore this [tutorial](https://github.com/NVIDIA-Merlin/Transformers4Rec/tree/stable/examples/tutorial) for a GRU-based model example."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16d51e39",
   "metadata": {},
   "source": [
    "### Train the model "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f26d7aec",
   "metadata": {},
   "source": [
    "We use the Merlin Dataloader's PyTorch Dataloader for optimized loading of multiple features from input parquet files. You can learn more about this data loader [here](https://nvidia-merlin.github.io/dataloader)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02fd4c22",
   "metadata": {},
   "source": [
    "### **Set Training arguments**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "99ba4286-3808-4688-953e-f93f663219fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "per_device_train_batch_size = int(os.environ.get(\n",
    "    \"per_device_train_batch_size\", \n",
    "    '128'\n",
    "))\n",
    "\n",
    "per_device_eval_batch_size = int(os.environ.get(\n",
    "    \"per_device_eval_batch_size\", \n",
    "    '32'\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "693974df",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "PyTorch: setting up devices\n"
     ]
    }
   ],
   "source": [
    "from transformers4rec.config.trainer import T4RecTrainingArguments\n",
    "from transformers4rec.torch import Trainer\n",
    "# Set hyperparameters for training \n",
    "train_args = T4RecTrainingArguments(data_loader_engine='merlin', \n",
    "                                    dataloader_drop_last = True,\n",
    "                                    gradient_accumulation_steps = 1,\n",
    "                                    per_device_train_batch_size = per_device_train_batch_size, \n",
    "                                    per_device_eval_batch_size = per_device_eval_batch_size,\n",
    "                                    output_dir = \"./tmp\", \n",
    "                                    learning_rate=0.0005,\n",
    "                                    lr_scheduler_type='cosine', \n",
    "                                    learning_rate_num_cosine_cycles_by_epoch=1.5,\n",
    "                                    num_train_epochs=5,\n",
    "                                    max_sequence_length=20, \n",
    "                                    report_to = [],\n",
    "                                    logging_steps=50,\n",
    "                                    no_cuda=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "445ece64",
   "metadata": {},
   "source": [
    "Note that we add an argument `data_loader_engine='merlin'` to automatically load the features needed for training using the schema. The default value is `merlin` for optimized GPU-based data-loading. Optionally a `PyarrowDataLoader` (pyarrow) can also be used as a basic option, but it is slower and works only for small datasets, as the full data is loaded to CPU memory."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32554ea0",
   "metadata": {},
   "source": [
    "## Daily Fine-Tuning: Training over a time window"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef883061",
   "metadata": {},
   "source": [
    "Here we do daily fine-tuning meaning that we use the first day to train and second day to evaluate, then we use the second day data to train the model by resuming from the first step, and evaluate on the third day, so on and so forth."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f452d09",
   "metadata": {},
   "source": [
    "We have extended the HuggingFace transformers `Trainer` class (PyTorch only) to support evaluation of RecSys metrics. In this example, the evaluation of the session-based recommendation model is performed using traditional Top-N ranking metrics such as Normalized Discounted Cumulative Gain (NDCG@20) and Hit Rate (HR@20). NDCG accounts for rank of the relevant item in the recommendation list and is a more fine-grained metric than HR, which only verifies whether the relevant item is among the top-n items. HR@n is equivalent to Recall@n when there is only one relevant item in the recommendation list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "2283f788",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate the T4Rec Trainer, which manages training and evaluation for the PyTorch API\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=train_args,\n",
    "    schema=schema,\n",
    "    compute_metrics=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d515127c",
   "metadata": {},
   "source": [
    "Define the output folder of the processed parquet files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "ae313150",
   "metadata": {},
   "outputs": [],
   "source": [
    "start_window_index = int(os.environ.get(\n",
    "    \"start_window_index\", \n",
    "    '1'\n",
    "))\n",
    "\n",
    "final_window_index = int(os.environ.get(\n",
    "    \"final_window_index\", \n",
    "    '8'\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "8ae51de0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/venv/lib/python3.8/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "***** Running training *****\n",
      "  Num examples = 1664\n",
      "  Num Epochs = 5\n",
      "  Instantaneous batch size per device = 128\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 128\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 65\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['/workspace/data/sessions_by_day/1/train.parquet']\n",
      "********************\n",
      "Launch training for day 1 are:\n",
      "********************\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='65' max='65' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [65/65 00:01, Epoch 5/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>5.878700</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finished\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='48' max='6' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [6/6 00:14]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "********************\n",
      "Eval results for day 2 are:\t\n",
      "\n",
      "********************\n",
      "\n",
      " eval_/loss = 5.100889682769775\n",
      " eval_/next-item/ndcg_at_20 = 0.18669554591178894\n",
      " eval_/next-item/ndcg_at_40 = 0.2293076366186142\n",
      " eval_/next-item/recall_at_20 = 0.5052083730697632\n",
      " eval_/next-item/recall_at_40 = 0.7135416865348816\n",
      " eval_runtime = 0.1324\n",
      " eval_samples_per_second = 1449.793\n",
      " eval_steps_per_second = 45.306\n",
      "['/workspace/data/sessions_by_day/2/train.parquet']\n",
      "********************\n",
      "Launch training for day 2 are:\n",
      "********************\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running training *****\n",
      "  Num examples = 1536\n",
      "  Num Epochs = 5\n",
      "  Instantaneous batch size per device = 128\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 128\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 60\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='60' max='60' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [60/60 00:01, Epoch 5/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>4.931100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finished\n",
      "********************\n",
      "Eval results for day 3 are:\t\n",
      "\n",
      "********************\n",
      "\n",
      " eval_/loss = 4.713772296905518\n",
      " eval_/next-item/ndcg_at_20 = 0.1680772304534912\n",
      " eval_/next-item/ndcg_at_40 = 0.21330341696739197\n",
      " eval_/next-item/recall_at_20 = 0.46875\n",
      " eval_/next-item/recall_at_40 = 0.6875\n",
      " eval_runtime = 0.1286\n",
      " eval_samples_per_second = 1492.732\n",
      " eval_steps_per_second = 46.648\n",
      "['/workspace/data/sessions_by_day/3/train.parquet']\n",
      "********************\n",
      "Launch training for day 3 are:\n",
      "********************\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running training *****\n",
      "  Num examples = 1536\n",
      "  Num Epochs = 5\n",
      "  Instantaneous batch size per device = 128\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 128\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 60\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='60' max='60' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [60/60 00:01, Epoch 5/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>4.586300</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finished\n",
      "********************\n",
      "Eval results for day 4 are:\t\n",
      "\n",
      "********************\n",
      "\n",
      " eval_/loss = 4.556211948394775\n",
      " eval_/next-item/ndcg_at_20 = 0.18394140899181366\n",
      " eval_/next-item/ndcg_at_40 = 0.2390844076871872\n",
      " eval_/next-item/recall_at_20 = 0.484375\n",
      " eval_/next-item/recall_at_40 = 0.7552083730697632\n",
      " eval_runtime = 0.1311\n",
      " eval_samples_per_second = 1464.947\n",
      " eval_steps_per_second = 45.78\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running training *****\n",
      "  Num examples = 1664\n",
      "  Num Epochs = 5\n",
      "  Instantaneous batch size per device = 128\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 128\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 65\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['/workspace/data/sessions_by_day/4/train.parquet']\n",
      "********************\n",
      "Launch training for day 4 are:\n",
      "********************\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='65' max='65' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [65/65 00:01, Epoch 5/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>4.518600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finished\n",
      "********************\n",
      "Eval results for day 5 are:\t\n",
      "\n",
      "********************\n",
      "\n",
      " eval_/loss = 4.407809734344482\n",
      " eval_/next-item/ndcg_at_20 = 0.20426493883132935\n",
      " eval_/next-item/ndcg_at_40 = 0.24358350038528442\n",
      " eval_/next-item/recall_at_20 = 0.5885416865348816\n",
      " eval_/next-item/recall_at_40 = 0.78125\n",
      " eval_runtime = 0.1327\n",
      " eval_samples_per_second = 1447.378\n",
      " eval_steps_per_second = 45.231\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running training *****\n",
      "  Num examples = 1664\n",
      "  Num Epochs = 5\n",
      "  Instantaneous batch size per device = 128\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 128\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 65\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['/workspace/data/sessions_by_day/5/train.parquet']\n",
      "********************\n",
      "Launch training for day 5 are:\n",
      "********************\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='65' max='65' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [65/65 00:01, Epoch 5/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>4.488800</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finished\n",
      "********************\n",
      "Eval results for day 6 are:\t\n",
      "\n",
      "********************\n",
      "\n",
      " eval_/loss = 4.5086541175842285\n",
      " eval_/next-item/ndcg_at_20 = 0.19850534200668335\n",
      " eval_/next-item/ndcg_at_40 = 0.2486129105091095\n",
      " eval_/next-item/recall_at_20 = 0.5052083730697632\n",
      " eval_/next-item/recall_at_40 = 0.75\n",
      " eval_runtime = 0.1258\n",
      " eval_samples_per_second = 1526.789\n",
      " eval_steps_per_second = 47.712\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running training *****\n",
      "  Num examples = 1664\n",
      "  Num Epochs = 5\n",
      "  Instantaneous batch size per device = 128\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 128\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 65\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['/workspace/data/sessions_by_day/6/train.parquet']\n",
      "********************\n",
      "Launch training for day 6 are:\n",
      "********************\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='65' max='65' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [65/65 00:01, Epoch 5/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>4.496300</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finished\n",
      "********************\n",
      "Eval results for day 7 are:\t\n",
      "\n",
      "********************\n",
      "\n",
      " eval_/loss = 4.442317962646484\n",
      " eval_/next-item/ndcg_at_20 = 0.1833297461271286\n",
      " eval_/next-item/ndcg_at_40 = 0.23786288499832153\n",
      " eval_/next-item/recall_at_20 = 0.5052083730697632\n",
      " eval_/next-item/recall_at_40 = 0.7760416865348816\n",
      " eval_runtime = 0.143\n",
      " eval_samples_per_second = 1342.609\n",
      " eval_steps_per_second = 41.957\n",
      "['/workspace/data/sessions_by_day/7/train.parquet']\n",
      "********************\n",
      "Launch training for day 7 are:\n",
      "********************\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running training *****\n",
      "  Num examples = 1792\n",
      "  Num Epochs = 5\n",
      "  Instantaneous batch size per device = 128\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 128\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 70\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='70' max='70' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [70/70 00:02, Epoch 5/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>4.489300</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finished\n",
      "********************\n",
      "Eval results for day 8 are:\t\n",
      "\n",
      "********************\n",
      "\n",
      " eval_/loss = 4.538415431976318\n",
      " eval_/next-item/ndcg_at_20 = 0.17900079488754272\n",
      " eval_/next-item/ndcg_at_40 = 0.22817844152450562\n",
      " eval_/next-item/recall_at_20 = 0.5052083730697632\n",
      " eval_/next-item/recall_at_40 = 0.7447916865348816\n",
      " eval_runtime = 0.1381\n",
      " eval_samples_per_second = 1390.084\n",
      " eval_steps_per_second = 43.44\n"
     ]
    }
   ],
   "source": [
    "start_time_window_index = start_window_index\n",
    "final_time_window_index = final_window_index\n",
    "#Iterating over days of one week\n",
    "for time_index in range(start_time_window_index, final_time_window_index):\n",
    "    # Set data \n",
    "    time_index_train = time_index\n",
    "    time_index_eval = time_index + 1\n",
    "    train_paths = glob.glob(os.path.join(OUTPUT_DIR, f\"{time_index_train}/train.parquet\"))\n",
    "    eval_paths = glob.glob(os.path.join(OUTPUT_DIR, f\"{time_index_eval}/valid.parquet\"))\n",
    "    print(train_paths)\n",
    "    \n",
    "    # Train on day related to time_index \n",
    "    print('*'*20)\n",
    "    print(\"Launch training for day %s are:\" %time_index)\n",
    "    print('*'*20 + '\\n')\n",
    "    trainer.train_dataset_or_path = train_paths\n",
    "    trainer.reset_lr_scheduler()\n",
    "    trainer.train()\n",
    "    trainer.state.global_step +=1\n",
    "    print('finished')\n",
    "    \n",
    "    # Evaluate on the following day\n",
    "    trainer.eval_dataset_or_path = eval_paths\n",
    "    train_metrics = trainer.evaluate(metric_key_prefix='eval')\n",
    "    print('*'*20)\n",
    "    print(\"Eval results for day %s are:\\t\" %time_index_eval)\n",
    "    print('\\n' + '*'*20 + '\\n')\n",
    "    for key in sorted(train_metrics.keys()):\n",
    "        print(\" %s = %s\" % (key, str(train_metrics[key]))) \n",
    "    wipe_memory()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a8d7bd8",
   "metadata": {},
   "source": [
    "### Re-compute evaluation metrics of the validation data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "3a34be66",
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_data_paths = glob.glob(os.path.join(OUTPUT_DIR, f\"{time_index_eval}/valid.parquet\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "2c6867d8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  eval_/loss = 4.538415431976318\n",
      "  eval_/next-item/ndcg_at_20 = 0.17900079488754272\n",
      "  eval_/next-item/ndcg_at_40 = 0.22817844152450562\n",
      "  eval_/next-item/recall_at_20 = 0.5052083730697632\n",
      "  eval_/next-item/recall_at_40 = 0.7447916865348816\n",
      "  eval_runtime = 0.1281\n",
      "  eval_samples_per_second = 1498.788\n",
      "  eval_steps_per_second = 46.837\n"
     ]
    }
   ],
   "source": [
    "# set new data from day 7\n",
    "eval_metrics = trainer.evaluate(eval_dataset=eval_data_paths, metric_key_prefix='eval')\n",
    "for key in sorted(eval_metrics.keys()):\n",
    "    print(\"  %s = %s\" % (key, str(eval_metrics[key])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78b520bd-f6e3-460e-b06d-1bf480cae3cf",
   "metadata": {},
   "source": [
    "### Save the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "766cd46b-25e7-4d89-9675-152e02103f33",
   "metadata": {},
   "source": [
    "Let's save the model to be able to load it back at inference step. Using `model.save()`, we save the model as a pkl file in the given path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "d9aebd79-e6e5-4309-a692-917b7b0283d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_path= os.environ.get(\"OUTPUT_DIR\", f\"{INPUT_DATA_DIR}/saved_model\")\n",
    "model.save(model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a26a649",
   "metadata": {},
   "source": [
    "That's it! You have just trained your session-based recommendation model using Transformers4Rec. Now you can move on to the next notebook `03-serving-session-based-model-torch-backend`. Please shut down this kernel to free the GPU memory before you start the next one."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6af8ecee",
   "metadata": {},
   "source": [
    "Tip: We can easily log and visualize model training and evaluation on [Weights & Biases (W&B)](https://wandb.ai/home), [TensorBoard](https://www.tensorflow.org/tensorboard), or [NVIDIA DLLogger](https://github.com/NVIDIA/dllogger). By default, the HuggingFace transformers `Trainer` (which we extend) uses Weights & Biases (W&B) to log training and evaluation metrics, which provides nice visualization results and comparison between different runs."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  },
  "vscode": {
   "interpreter": {
    "hash": "d795d7ca5d3ec3bd6293cc80853205a74ce23d484a2b8f537732a716747107c8"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}