recommendations/deep-recommender-transformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "exceptional-exhibition",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Deep Recommender: Behavior Sequence Transformer (BST)\n",
    "\n",
    "This notebook demonstrates how recommendations can be made based on interaction sequences using transformer-based model.\n",
    "\n",
    "### Use Case\n",
    "We assume hybrid data that can include user features like age, item features like category, and interaction event sequences for certain user-item pairs. Each interaction event can be associated with additional features such as rating and timestamp. Out goal is to build a model that, for each user, predicts the item this user is likely ot interact with.\n",
    "\n",
    "### Prototype: Approach and Data\n",
    "The prototype is based on Behavior Sequence Transformer (BST) design [1] and Keras implementation created by Khalid Salama's [2]. We use `MovieLens 1M` dataset. See `datasets.md` for details.\n",
    "\n",
    "### Usage and Productization\n",
    "This prototype can be used to evaluate the feasibility of the transformer based approach for a specific dataset. Its performance can also be compared with other prototypes and optimal approach can be selected.\n",
    "\n",
    "### References:\n",
    "1. Chen Q., Zhao H., Li W., Huang P., and Ou W. -- Behavior Sequence Transformer for E-commerce Recommendation in Alibaba, 2019\n",
    "2. Salama K. -- A Transformer-based recommendation system, 2020"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "editorial-tyler",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keras version 2.6.0\n",
      "Tensorflow version 2.6.0\n"
     ]
    }
   ],
   "source": [
    "#\n",
    "# Imports and settings\n",
    "#\n",
    "import math\n",
    "import warnings\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "from tensorflow.keras.layers import StringLookup\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tabulate import tabulate\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams.update({'pdf.fonttype': 'truetype'})\n",
    "warnings.simplefilter(\"ignore\")\n",
    "\n",
    "print(\"Keras version \" + tf.keras.__version__)\n",
    "print(\"Tensorflow version \" + tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d5421479-a208-4c3c-84f2-71a1e7f40f3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Helper functions\n",
    "#\n",
    "def print_df(df, col_width = 10, rows = 10, max_cols = 10):\n",
    "    def short_srt(x):\n",
    "        return x if len(x) < col_width else x[:col_width-3] + \"...\"\n",
    "    df_short = df.head(rows).applymap(lambda x: short_srt(str(x)))\n",
    "    \n",
    "    if len(df_short.columns) > max_cols:\n",
    "        df_short = df_short.iloc[:, 0:max_cols-1]\n",
    "        df_short['...'] = '...'\n",
    "    \n",
    "    print(tabulate(df_short, headers='keys', tablefmt='psql'))\n",
    "    print(f'{len(df)} rows x {len(df.columns)} columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "regulated-device",
   "metadata": {},
   "source": [
    "# Data Loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "continuous-auditor",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_moviewlens_1m():\n",
    "    base_data_path = '~/ALGO/tensor-house-data-large-unpacked/movielens-1m/'\n",
    "\n",
    "    users = pd.read_csv(\n",
    "        base_data_path + \"users.dat\",\n",
    "        sep=\"::\",\n",
    "        names=[\"user_id\", \"sex\", \"age_group\", \"occupation\", \"zip_code\"],\n",
    "        encoding='ISO-8859-1'\n",
    "    )\n",
    "\n",
    "    ratings = pd.read_csv(\n",
    "        base_data_path + \"ratings.dat\",\n",
    "        sep=\"::\",\n",
    "        names=[\"user_id\", \"movie_id\", \"rating\", \"unix_timestamp\"],\n",
    "        encoding='ISO-8859-1'\n",
    "    )\n",
    "\n",
    "    movies = pd.read_csv(\n",
    "        base_data_path + \"movies.dat\", \n",
    "        sep=\"::\", \n",
    "        names=[\"movie_id\", \"title\", \"genres\"],\n",
    "        encoding='ISO-8859-1'\n",
    "    )\n",
    "    \n",
    "    return users, ratings, movies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8015cd93-b6d9-4012-b968-6b07955da829",
   "metadata": {},
   "source": [
    "# Baseline: Neural Collaborating Filtering\n",
    "\n",
    "We start with implementing a baseline model that uses only the rating data. We use a basic factorization network that maps user and item IDs to embeddings and estimates the rating as a dot product."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4369e207-3638-4616-928f-f08f6b8a3bcd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Load data\n",
    "#\n",
    "users, ratings, movies = load_moviewlens_1m()\n",
    "\n",
    "#\n",
    "# Train/test split\n",
    "#\n",
    "n = ratings.shape[0]\n",
    "train_idx, test_idx, _, _ = train_test_split(np.arange(n), np.arange(n), test_size=0.20, random_state=42)\n",
    "x_train, x_test = ratings[['user_id', 'movie_id']].values[train_idx], ratings[['user_id', 'movie_id']].values[test_idx]\n",
    "y_train, y_test = ratings['rating'].values[train_idx], ratings['rating'].values[test_idx]\n",
    "\n",
    "max_user_id, max_item_id = max(ratings['user_id']), max(ratings['movie_id'])\n",
    "\n",
    "#\n",
    "# Model specification\n",
    "#\n",
    "embedding_dim = 8\n",
    "\n",
    "input_user = layers.Input(shape=(1,))\n",
    "input_item = layers.Input(shape=(1,))\n",
    "\n",
    "embedding_user = layers.Embedding(input_dim=max_user_id + 1, output_dim=embedding_dim)(input_user)\n",
    "embedding_item = layers.Embedding(input_dim=max_item_id + 1, output_dim=embedding_dim)(input_item)\n",
    "\n",
    "combined = layers.Dot(axes=2)([embedding_user, embedding_item])\n",
    "\n",
    "score = layers.Dense(1, activation='linear')(combined)\n",
    "\n",
    "model_sgd = keras.models.Model(inputs=[input_user, input_item], outputs=score)\n",
    "\n",
    "#\n",
    "# Model training\n",
    "#\n",
    "opt = keras.optimizers.Adam(learning_rate=0.01)\n",
    "model_sgd.compile(optimizer=opt, loss='mse', metrics=[keras.metrics.MeanAbsoluteError()]) \n",
    "history_sgd = model_sgd.fit([x_train[:, 0], x_train[:, 1]], y_train, \n",
    "                    batch_size=256, epochs=8, verbose=1, \n",
    "                    validation_data=([x_test[:, 0], x_test[:, 1]], y_test)) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0187282a-f64e-49d3-b4cb-2f90bd34b155",
   "metadata": {},
   "source": [
    "# BST: Data Preparation\n",
    "\n",
    "We do the initial data preparation, split the dataset into train/test sets, and save them into separate files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1e580294-594c-4abd-bbd3-4563b731f531",
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Load data\n",
    "#\n",
    "users, ratings, movies = load_moviewlens_1m()\n",
    "\n",
    "users[\"user_id\"] = users[\"user_id\"].apply(lambda x: f\"user_{x}\")\n",
    "users[\"age_group\"] = users[\"age_group\"].apply(lambda x: f\"group_{x}\")\n",
    "users[\"occupation\"] = users[\"occupation\"].apply(lambda x: f\"occupation_{x}\")\n",
    "\n",
    "movies[\"movie_id\"] = movies[\"movie_id\"].apply(lambda x: f\"movie_{x}\")\n",
    "\n",
    "ratings[\"movie_id\"] = ratings[\"movie_id\"].apply(lambda x: f\"movie_{x}\")\n",
    "ratings[\"user_id\"] = ratings[\"user_id\"].apply(lambda x: f\"user_{x}\")\n",
    "ratings[\"rating\"] = ratings[\"rating\"].apply(lambda x: float(x))\n",
    "\n",
    "genres = [\n",
    "    \"Action\",\n",
    "    \"Adventure\",\n",
    "    \"Animation\",\n",
    "    \"Children's\",\n",
    "    \"Comedy\",\n",
    "    \"Crime\",\n",
    "    \"Documentary\",\n",
    "    \"Drama\",\n",
    "    \"Fantasy\",\n",
    "    \"Film-Noir\",\n",
    "    \"Horror\",\n",
    "    \"Musical\",\n",
    "    \"Mystery\",\n",
    "    \"Romance\",\n",
    "    \"Sci-Fi\",\n",
    "    \"Thriller\",\n",
    "    \"War\",\n",
    "    \"Western\",\n",
    "]\n",
    "\n",
    "for genre in genres:\n",
    "    movies[genre] = movies[\"genres\"].apply(\n",
    "        lambda values: int(genre in values.split(\"|\"))\n",
    "    )\n",
    "    \n",
    "#\n",
    "# Transform the movie ratings data into sequences\n",
    "#\n",
    "ratings_group = ratings.sort_values(by=[\"unix_timestamp\"]).groupby(\"user_id\")\n",
    "\n",
    "ratings_data = pd.DataFrame(\n",
    "    data={\n",
    "        \"user_id\": list(ratings_group.groups.keys()),\n",
    "        \"movie_ids\": list(ratings_group.movie_id.apply(list)),\n",
    "        \"ratings\": list(ratings_group.rating.apply(list)),\n",
    "        \"timestamps\": list(ratings_group.unix_timestamp.apply(list)),\n",
    "    }\n",
    ")\n",
    "\n",
    "sequence_length = 8\n",
    "step_size = 1\n",
    "\n",
    "def create_sequences(values, window_size, step_size):\n",
    "    sequences = []\n",
    "    start_index = 0\n",
    "    while True:\n",
    "        end_index = start_index + window_size\n",
    "        seq = values[start_index:end_index]\n",
    "        if len(seq) < window_size:\n",
    "            seq = values[-window_size:]\n",
    "            if len(seq) == window_size:\n",
    "                sequences.append(seq)\n",
    "            break\n",
    "        sequences.append(seq)\n",
    "        start_index += step_size\n",
    "    return sequences\n",
    "\n",
    "\n",
    "ratings_data.movie_ids = ratings_data.movie_ids.apply(\n",
    "    lambda ids: create_sequences(ids, sequence_length, step_size)\n",
    ")\n",
    "\n",
    "ratings_data.ratings = ratings_data.ratings.apply(\n",
    "    lambda ids: create_sequences(ids, sequence_length, step_size)\n",
    ")\n",
    "\n",
    "del ratings_data[\"timestamps\"]\n",
    "\n",
    "ratings_data_movies = ratings_data[[\"user_id\", \"movie_ids\"]].explode(\n",
    "    \"movie_ids\", ignore_index=True\n",
    ")\n",
    "ratings_data_rating = ratings_data[[\"ratings\"]].explode(\"ratings\", ignore_index=True)\n",
    "ratings_data_transformed = pd.concat([ratings_data_movies, ratings_data_rating], axis=1)\n",
    "ratings_data_transformed = ratings_data_transformed.join(\n",
    "    users.set_index(\"user_id\"), on=\"user_id\"\n",
    ")\n",
    "ratings_data_transformed.movie_ids = ratings_data_transformed.movie_ids.apply(\n",
    "    lambda x: \",\".join(x)\n",
    ")\n",
    "ratings_data_transformed.ratings = ratings_data_transformed.ratings.apply(\n",
    "    lambda x: \",\".join([str(v) for v in x])\n",
    ")\n",
    "\n",
    "del ratings_data_transformed[\"zip_code\"]\n",
    "\n",
    "ratings_data_transformed.rename(\n",
    "    columns={\"movie_ids\": \"sequence_movie_ids\", \"ratings\": \"sequence_ratings\"},\n",
    "    inplace=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d7462dd8-393e-4e98-99dd-9868df34ae67",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+-----------+-------------------------------------+---------------------------------+-------+-------------+---------------+\n",
      "|    | user_id   | sequence_movie_ids                  | sequence_ratings                | sex   | age_group   | occupation    |\n",
      "|----+-----------+-------------------------------------+---------------------------------+-------+-------------+---------------|\n",
      "|  0 | user_1    | movie_3186,movie_1721,movie_1270... | 4.0,4.0,5.0,5.0,3.0,5.0,4.0,4.0 | F     | group_1     | occupation_10 |\n",
      "|  1 | user_1    | movie_1721,movie_1270,movie_1022... | 4.0,5.0,5.0,3.0,5.0,4.0,4.0,5.0 | F     | group_1     | occupation_10 |\n",
      "|  2 | user_1    | movie_1270,movie_1022,movie_2340... | 5.0,5.0,3.0,5.0,4.0,4.0,5.0,4.0 | F     | group_1     | occupation_10 |\n",
      "|  3 | user_1    | movie_1022,movie_2340,movie_1836... | 5.0,3.0,5.0,4.0,4.0,5.0,4.0,3.0 | F     | group_1     | occupation_10 |\n",
      "|  4 | user_1    | movie_2340,movie_1836,movie_3408... | 3.0,5.0,4.0,4.0,5.0,4.0,3.0,5.0 | F     | group_1     | occupation_10 |\n",
      "|  5 | user_1    | movie_1836,movie_3408,movie_1207... | 5.0,4.0,4.0,5.0,4.0,3.0,5.0,4.0 | F     | group_1     | occupation_10 |\n",
      "|  6 | user_1    | movie_3408,movie_1207,movie_2804... | 4.0,4.0,5.0,4.0,3.0,5.0,4.0,4.0 | F     | group_1     | occupation_10 |\n",
      "|  7 | user_1    | movie_1207,movie_2804,movie_260,... | 4.0,5.0,4.0,3.0,5.0,4.0,4.0,4.0 | F     | group_1     | occupation_10 |\n",
      "|  8 | user_1    | movie_2804,movie_260,movie_720,m... | 5.0,4.0,3.0,5.0,4.0,4.0,4.0,5.0 | F     | group_1     | occupation_10 |\n",
      "|  9 | user_1    | movie_260,movie_720,movie_1193,m... | 4.0,3.0,5.0,4.0,4.0,4.0,5.0,5.0 | F     | group_1     | occupation_10 |\n",
      "+----+-----------+-------------------------------------+---------------------------------+-------+-------------+---------------+\n",
      "963969 rows x 6 columns\n"
     ]
    }
   ],
   "source": [
    "print_df(ratings_data_transformed, col_width=35)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3c96cd53-1623-4b1c-a6ef-abceba1f432e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Save train/test datasets to files\n",
    "#\n",
    "random_selection = np.random.rand(len(ratings_data_transformed.index)) <= 0.80\n",
    "train_data = ratings_data_transformed[random_selection]\n",
    "test_data = ratings_data_transformed[~random_selection]\n",
    "\n",
    "train_data.to_csv(\"train_data.csv\", index=False, sep=\"|\", header=False)\n",
    "test_data.to_csv(\"test_data.csv\", index=False, sep=\"|\", header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "443abd55-fa99-4d89-9de7-825e4bc42461",
   "metadata": {},
   "source": [
    "# BST: Feature Encoding \n",
    "\n",
    "Next, we map movie IDs and associated genres using embedding layers to prepare the input for the transformer. \n",
    "\n",
    "User features are mapped separately and will further be concatenated with the output of the transformer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b9bf3b07-afba-47ef-9ffb-8f0a93d50a11",
   "metadata": {},
   "outputs": [],
   "source": [
    "CSV_HEADER = list(ratings_data_transformed.columns)\n",
    "USER_FEATURES = [\"sex\", \"age_group\", \"occupation\"]\n",
    "MOVIE_FEATURES = [\"genres\"]\n",
    "CATEGORICAL_FEATURES_WITH_VOCABULARY = {\n",
    "    \"user_id\": list(users.user_id.unique()),\n",
    "    \"movie_id\": list(movies.movie_id.unique()),\n",
    "    \"sex\": list(users.sex.unique()),\n",
    "    \"age_group\": list(users.age_group.unique()),\n",
    "    \"occupation\": list(users.occupation.unique()),\n",
    "}\n",
    "\n",
    "\n",
    "def get_dataset_from_csv(csv_file_path, shuffle=False, batch_size=128):\n",
    "    def process(features):\n",
    "        movie_ids_string = features[\"sequence_movie_ids\"]\n",
    "        sequence_movie_ids = tf.strings.split(movie_ids_string, \",\").to_tensor()\n",
    "\n",
    "        # The last movie id in the sequence is the target movie\n",
    "        features[\"target_movie_id\"] = sequence_movie_ids[:, -1]\n",
    "        features[\"sequence_movie_ids\"] = sequence_movie_ids[:, :-1]\n",
    "\n",
    "        ratings_string = features[\"sequence_ratings\"]\n",
    "        sequence_ratings = tf.strings.to_number(\n",
    "            tf.strings.split(ratings_string, \",\"), tf.dtypes.float32\n",
    "        ).to_tensor()\n",
    "\n",
    "        # The last rating in the sequence is the target for the model to predict\n",
    "        target = sequence_ratings[:, -1]\n",
    "        features[\"sequence_ratings\"] = sequence_ratings[:, :-1]\n",
    "\n",
    "        return features, target\n",
    "\n",
    "    dataset = tf.data.experimental.make_csv_dataset(\n",
    "        csv_file_path,\n",
    "        batch_size=batch_size,\n",
    "        column_names=CSV_HEADER,\n",
    "        num_epochs=1,\n",
    "        header=False,\n",
    "        field_delim=\"|\",\n",
    "        shuffle=shuffle,\n",
    "    ).map(process)\n",
    "\n",
    "    return dataset\n",
    "\n",
    "\n",
    "def create_model_inputs():\n",
    "    return {\n",
    "        \"user_id\": layers.Input(name=\"user_id\", shape=(1,), dtype=tf.string),\n",
    "        \"sequence_movie_ids\": layers.Input(\n",
    "            name=\"sequence_movie_ids\", shape=(sequence_length - 1,), dtype=tf.string\n",
    "        ),\n",
    "        \"target_movie_id\": layers.Input(\n",
    "            name=\"target_movie_id\", shape=(1,), dtype=tf.string\n",
    "        ),\n",
    "        \"sequence_ratings\": layers.Input(\n",
    "            name=\"sequence_ratings\", shape=(sequence_length - 1,), dtype=tf.float32\n",
    "        ),\n",
    "        \"sex\": layers.Input(name=\"sex\", shape=(1,), dtype=tf.string),\n",
    "        \"age_group\": layers.Input(name=\"age_group\", shape=(1,), dtype=tf.string),\n",
    "        \"occupation\": layers.Input(name=\"occupation\", shape=(1,), dtype=tf.string),\n",
    "    }\n",
    "\n",
    "\n",
    "def encode_input_features(\n",
    "    inputs,\n",
    "    include_user_id=True,\n",
    "    include_user_features=True,\n",
    "    include_movie_features=True,\n",
    "):\n",
    "\n",
    "    encoded_transformer_features = []\n",
    "    encoded_other_features = []\n",
    "\n",
    "    other_feature_names = []\n",
    "    if include_user_id:\n",
    "        other_feature_names.append(\"user_id\")\n",
    "    if include_user_features:\n",
    "        other_feature_names.extend(USER_FEATURES)\n",
    "\n",
    "    ## Encode user features\n",
    "    for feature_name in other_feature_names:\n",
    "        # Convert the string input values into integer indices\n",
    "        vocabulary = CATEGORICAL_FEATURES_WITH_VOCABULARY[feature_name]\n",
    "        idx = StringLookup(vocabulary=vocabulary, mask_token=None, num_oov_indices=0)(\n",
    "            inputs[feature_name]\n",
    "        )\n",
    "        # Compute embedding dimensions\n",
    "        embedding_dims = int(math.sqrt(len(vocabulary)))\n",
    "        # Create an embedding layer with the specified dimensions\n",
    "        embedding_encoder = layers.Embedding(\n",
    "            input_dim=len(vocabulary),\n",
    "            output_dim=embedding_dims,\n",
    "            name=f\"{feature_name}_embedding\",\n",
    "        )\n",
    "        # Convert the index values to embedding representations\n",
    "        encoded_other_features.append(embedding_encoder(idx))\n",
    "\n",
    "    ## Create a single embedding vector for the user features\n",
    "    if len(encoded_other_features) > 1:\n",
    "        encoded_other_features = layers.concatenate(encoded_other_features)\n",
    "    elif len(encoded_other_features) == 1:\n",
    "        encoded_other_features = encoded_other_features[0]\n",
    "    else:\n",
    "        encoded_other_features = None\n",
    "\n",
    "    ## Create a movie embedding encoder\n",
    "    movie_vocabulary = CATEGORICAL_FEATURES_WITH_VOCABULARY[\"movie_id\"]\n",
    "    movie_embedding_dims = int(math.sqrt(len(movie_vocabulary)))\n",
    "    # Create a lookup to convert string values to integer indices\n",
    "    movie_index_lookup = StringLookup(\n",
    "        vocabulary=movie_vocabulary,\n",
    "        mask_token=None,\n",
    "        num_oov_indices=0,\n",
    "        name=\"movie_index_lookup\",\n",
    "    )\n",
    "    # Create an embedding layer with the specified dimensions\n",
    "    movie_embedding_encoder = layers.Embedding(\n",
    "        input_dim=len(movie_vocabulary),\n",
    "        output_dim=movie_embedding_dims,\n",
    "        name=f\"movie_embedding\",\n",
    "    )\n",
    "    # Create a vector lookup for movie genres\n",
    "    genre_vectors = movies[genres].to_numpy()\n",
    "    movie_genres_lookup = layers.Embedding(\n",
    "        input_dim=genre_vectors.shape[0],\n",
    "        output_dim=genre_vectors.shape[1],\n",
    "        embeddings_initializer=tf.keras.initializers.Constant(genre_vectors),\n",
    "        trainable=False,\n",
    "        name=\"genres_vector\",\n",
    "    )\n",
    "    # Create a processing layer for genres\n",
    "    movie_embedding_processor = layers.Dense(\n",
    "        units=movie_embedding_dims,\n",
    "        activation=\"relu\",\n",
    "        name=\"process_movie_embedding_with_genres\",\n",
    "    )\n",
    "\n",
    "    ## Define a function to encode a given movie id.\n",
    "    def encode_movie(movie_id):\n",
    "        # Convert the string input values into integer indices.\n",
    "        movie_idx = movie_index_lookup(movie_id)\n",
    "        movie_embedding = movie_embedding_encoder(movie_idx)\n",
    "        encoded_movie = movie_embedding\n",
    "        if include_movie_features:\n",
    "            movie_genres_vector = movie_genres_lookup(movie_idx)\n",
    "            encoded_movie = movie_embedding_processor(\n",
    "                layers.concatenate([movie_embedding, movie_genres_vector])\n",
    "            )\n",
    "        return encoded_movie\n",
    "\n",
    "    ## Encoding target_movie_id\n",
    "    target_movie_id = inputs[\"target_movie_id\"]\n",
    "    encoded_target_movie = encode_movie(target_movie_id)\n",
    "\n",
    "    ## Encoding sequence movie_ids\n",
    "    sequence_movies_ids = inputs[\"sequence_movie_ids\"]\n",
    "    encoded_sequence_movies = encode_movie(sequence_movies_ids)\n",
    "    # Create positional embedding.\n",
    "    position_embedding_encoder = layers.Embedding(\n",
    "        input_dim=sequence_length,\n",
    "        output_dim=movie_embedding_dims,\n",
    "        name=\"position_embedding\",\n",
    "    )\n",
    "    positions = tf.range(start=0, limit=sequence_length - 1, delta=1)\n",
    "    encodded_positions = position_embedding_encoder(positions)\n",
    "    # Retrieve sequence ratings to incorporate them into the encoding of the movie.\n",
    "    sequence_ratings = tf.expand_dims(inputs[\"sequence_ratings\"], -1)\n",
    "    # Add the positional encoding to the movie encodings and multiply them by rating.\n",
    "    encoded_sequence_movies_with_poistion_and_rating = layers.Multiply()(\n",
    "        [(encoded_sequence_movies + encodded_positions), sequence_ratings]\n",
    "    )\n",
    "\n",
    "    # Construct the transformer inputs\n",
    "    for encoded_movie in tf.unstack(\n",
    "        encoded_sequence_movies_with_poistion_and_rating, axis=1\n",
    "    ):\n",
    "        encoded_transformer_features.append(tf.expand_dims(encoded_movie, 1))\n",
    "    \n",
    "    encoded_transformer_features.append(encoded_target_movie)\n",
    "    encoded_transformer_features = layers.concatenate(encoded_transformer_features, axis=1)\n",
    "\n",
    "    return encoded_transformer_features, encoded_other_features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c6bcb5e-5c46-43ab-a0df-199613b5b54b",
   "metadata": {},
   "source": [
    "# BST: Model Specification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8ca0d908-8c6a-4089-b8e5-ab53a9b58f69",
   "metadata": {},
   "outputs": [],
   "source": [
    "include_user_id = False\n",
    "include_user_features = False\n",
    "include_movie_features = False\n",
    "\n",
    "hidden_units = [256, 128]\n",
    "dropout_rate = 0.1\n",
    "num_heads = 3\n",
    "\n",
    "\n",
    "def create_model():\n",
    "    inputs = create_model_inputs()\n",
    "    transformer_features, other_features = encode_input_features(\n",
    "        inputs, include_user_id, include_user_features, include_movie_features\n",
    "    )\n",
    "\n",
    "    # Create a multi-headed attention layer\n",
    "    attention_output = layers.MultiHeadAttention(\n",
    "        num_heads=num_heads, key_dim=transformer_features.shape[2], dropout=dropout_rate\n",
    "    )(transformer_features, transformer_features)\n",
    "\n",
    "    # Transformer block\n",
    "    attention_output = layers.Dropout(dropout_rate)(attention_output)\n",
    "    x1 = layers.Add()([transformer_features, attention_output])\n",
    "    x1 = layers.LayerNormalization()(x1)\n",
    "    x2 = layers.LeakyReLU()(x1)\n",
    "    x2 = layers.Dense(units=x2.shape[-1])(x2)\n",
    "    x2 = layers.Dropout(dropout_rate)(x2)\n",
    "    transformer_features = layers.Add()([x1, x2])\n",
    "    transformer_features = layers.LayerNormalization()(transformer_features)\n",
    "    features = layers.Flatten()(transformer_features)\n",
    "\n",
    "    # Included the other features\n",
    "    if other_features is not None:\n",
    "        features = layers.concatenate(\n",
    "            [features, layers.Reshape([other_features.shape[-1]])(other_features)]\n",
    "        )\n",
    "\n",
    "    # Fully-connected layers\n",
    "    for num_units in hidden_units:\n",
    "        features = layers.Dense(num_units)(features)\n",
    "        features = layers.BatchNormalization()(features)\n",
    "        features = layers.LeakyReLU()(features)\n",
    "        features = layers.Dropout(dropout_rate)(features)\n",
    "\n",
    "    outputs = layers.Dense(units=1)(features)\n",
    "    model = keras.Model(inputs=inputs, outputs=outputs)\n",
    "    return model\n",
    "\n",
    "\n",
    "model = create_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1100f246-9518-495c-99d2-5e7abe56e6d7",
   "metadata": {},
   "source": [
    "# BST: Training and Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "449f1119-0142-4cf9-aa80-0f60e758e951",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/16\n",
      "2911/2911 [==============================] - 82s 28ms/step - loss: 1.1600 - mean_absolute_error: 0.8543\n",
      "Epoch 2/16\n",
      "2911/2911 [==============================] - 88s 30ms/step - loss: 0.9421 - mean_absolute_error: 0.7715\n",
      "Epoch 3/16\n",
      "2911/2911 [==============================] - 88s 30ms/step - loss: 0.8972 - mean_absolute_error: 0.7524\n",
      "Epoch 4/16\n",
      "2911/2911 [==============================] - 96s 33ms/step - loss: 0.8742 - mean_absolute_error: 0.7423\n",
      "Epoch 5/16\n",
      "2911/2911 [==============================] - 119s 41ms/step - loss: 0.8610 - mean_absolute_error: 0.7362\n",
      "Epoch 6/16\n",
      "2911/2911 [==============================] - 150s 52ms/step - loss: 0.8497 - mean_absolute_error: 0.7310\n",
      "Epoch 7/16\n",
      "2911/2911 [==============================] - 239s 82ms/step - loss: 0.8405 - mean_absolute_error: 0.7269\n",
      "Epoch 8/16\n",
      "2911/2911 [==============================] - 218s 75ms/step - loss: 0.8344 - mean_absolute_error: 0.7247\n",
      "Epoch 9/16\n",
      "2911/2911 [==============================] - 159s 55ms/step - loss: 0.8288 - mean_absolute_error: 0.7218\n",
      "Epoch 10/16\n",
      "2911/2911 [==============================] - 155s 53ms/step - loss: 0.8238 - mean_absolute_error: 0.7194\n",
      "Epoch 11/16\n",
      "2911/2911 [==============================] - 134s 46ms/step - loss: 0.8186 - mean_absolute_error: 0.7173\n",
      "Epoch 12/16\n",
      "2911/2911 [==============================] - 151s 52ms/step - loss: 0.8144 - mean_absolute_error: 0.7156\n",
      "Epoch 13/16\n",
      "2911/2911 [==============================] - 122s 42ms/step - loss: 0.8110 - mean_absolute_error: 0.7139\n",
      "Epoch 14/16\n",
      "2911/2911 [==============================] - 118s 40ms/step - loss: 0.8073 - mean_absolute_error: 0.7121\n",
      "Epoch 15/16\n",
      "2911/2911 [==============================] - 154s 53ms/step - loss: 0.8027 - mean_absolute_error: 0.7099\n",
      "Epoch 16/16\n",
      "2911/2911 [==============================] - 133s 46ms/step - loss: 0.8001 - mean_absolute_error: 0.7089\n",
      "Test MAE: 0.735\n"
     ]
    }
   ],
   "source": [
    "# Compile the model.\n",
    "model.compile(\n",
    "    optimizer=keras.optimizers.Adagrad(learning_rate=0.01),\n",
    "    loss=keras.losses.MeanSquaredError(),\n",
    "    metrics=[keras.metrics.MeanAbsoluteError()],\n",
    ")\n",
    "\n",
    "# Read the training data.\n",
    "train_dataset = get_dataset_from_csv(\"train_data.csv\", shuffle=True, batch_size=265)\n",
    "\n",
    "# Fit the model with the training data.\n",
    "model.fit(train_dataset, epochs=16)\n",
    "\n",
    "# Read the test data.\n",
    "test_dataset = get_dataset_from_csv(\"test_data.csv\", batch_size=265)\n",
    "\n",
    "# Evaluate the model on the test data.\n",
    "_, mae = model.evaluate(test_dataset, verbose=0)\n",
    "print(f\"Test MAE: {round(mae, 3)}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py310",
   "language": "python",
   "name": "py310"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}