kred_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This repository proposes two extensions to the implementation of [KRED: Knowledge-Aware Document Representation for News Recommendations](https://arxiv.org/abs/1910.11494) [1]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "KRED is a knowledge enhanced framework which enhance a document embedding with knowledge information for multiple news recommendation tasks. The framework mainly contains two part: representation enhancement part(left) and multi-task training part(right)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](./Architecture_new.PNG)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two extensions to this model have been implemented:\n",
    "- replacing the attention layer over the entities with a multi-head attention layer;\n",
    "- adding to the context embedding layer the embedding of the news category which consists of a first general category and a second more specific category, the two categories are embedded separately.\n",
    "\n",
    "The multi-head attention, the embedding of the first news category and the embedding of the second news embedding can be enabled/disabled using the config."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization\n",
    "\n",
    "If you download this notebook and run it on colab, you need to run the following commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clone repo\n",
    "!git clone https://github.com/lolloberga/Recommendation_System_KRED.git\n",
    "%cd Recommendation_System_KRED"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install required libraries\n",
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from utils.util import *\n",
    "from train_test import *\n",
    "import sys\n",
    "import os\n",
    "import argparse\n",
    "from parse_config import ConfigParser"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset description and download\n",
    "\n",
    "MIND dataset [2] is a large-scale English news dataset. It was collected from anonymized behavior logs of Microsoft News website. MIND contains 1,000,000 users, 161,013 news articles and 15,777,377 impression logs. Every news article contains rich textual content including title, abstract, body, category and entities. Each impression log contains the click events, non-clicked events and historical news click behaviors of this user before this impression.\n",
    "\n",
    "For quicker training and evaluaiton, we use MINDsmall dataset of 50k users from MINDlarge dataset. The MINDsmall dataset has the same file format as MINDlarge.\n",
    "\n",
    "MINDsmall_train is used for training, and MINDsmall_dev is used for evaluation. Training data and evaluation data are composed of a news file and a behaviors file. You can find more detailed data description in [MIND repo](https://github.com/msnews/msnews.github.io/blob/master/assets/doc/introduction.md)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Options: demo, small, large\n",
    "MIND_type = 'small'\n",
    "data_path = \"./data/\"\n",
    "\n",
    "train_news_file = os.path.join(data_path, 'train', r'news.tsv')\n",
    "train_behaviors_file = os.path.join(data_path, 'train', r'behaviors.tsv')\n",
    "valid_news_file = os.path.join(data_path, 'valid', r'news.tsv')\n",
    "valid_behaviors_file = os.path.join(data_path, 'valid', r'behaviors.tsv')\n",
    "knowledge_graph_file = os.path.join(data_path, 'kg/wikidata-graph', r'wikidata-graph.tsv')\n",
    "entity_embedding_file = os.path.join(data_path, 'kg/wikidata-graph', r'entity2vecd100.vec')\n",
    "relation_embedding_file = os.path.join(data_path, 'kg/wikidata-graph', r'relation2vecd100.vec')\n",
    "\n",
    "mind_url, mind_train_dataset, mind_dev_dataset, _ = get_mind_data_set(MIND_type)\n",
    "\n",
    "kg_url = \"https://kredkg.blob.core.windows.net/wikidatakg/\"\n",
    "\n",
    "if not os.path.exists(train_news_file):\n",
    "    download_deeprec_resources(mind_url, os.path.join(data_path, 'train'), mind_train_dataset)\n",
    "    \n",
    "if not os.path.exists(valid_news_file):\n",
    "    download_deeprec_resources(mind_url, \\\n",
    "                               os.path.join(data_path, 'valid'), mind_dev_dataset)\n",
    "\n",
    "if not os.path.exists(knowledge_graph_file):\n",
    "    download_deeprec_resources(kg_url, \\\n",
    "                               os.path.join(data_path, 'kg'), \"kg.zip\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sys.path.append('')\n",
    "sys.argv = [''] # added by me, solved problem in this cell\n",
    "\n",
    "parser = argparse.ArgumentParser(description='KRED')\n",
    "\n",
    "\n",
    "parser.add_argument('-c', '--config', default=\"./config.json\", type=str,\n",
    "                    help='config file path (default: None)')\n",
    "parser.add_argument('-r', '--resume', default=None, type=str,\n",
    "                    help='path to latest checkpoint (default: None)')\n",
    "parser.add_argument('-d', '--device', default=None, type=str,\n",
    "                    help='indices of GPUs to enable (default: all)')\n",
    "\n",
    "config = ConfigParser.from_args(parser)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create hyper-parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "epochs = 10\n",
    "batch_size = 64\n",
    "train_type = \"single_task\"\n",
    "task = \"user2item\" # task should be within: user2item, item2item, vert_classify, pop_predict\n",
    "\n",
    "config['trainer']['epochs'] = epochs\n",
    "config['data_loader']['batch_size'] = batch_size\n",
    "config['trainer']['training_type'] = train_type\n",
    "config['trainer']['task'] = task\n",
    "config['trainer']['save_period'] = epochs/2\n",
    "# The following parameters define which of the extensions are used, \n",
    "# by setting them to False the original KRED model is executed \n",
    "config['model']['use_mh_attention'] = False\n",
    "config['model']['mh_number_of_heads'] = 12\n",
    "config['data']['use_entity_category'] = True\n",
    "config['data']['use_second_entity_category'] = False"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Process dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To speed up the execution, you can save the sentence embeddings on Google drive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive', force_remount=False)\n",
    "# WARNING: the following folder must exist, otherwise it will raise errors when\n",
    "# saving sentence embeddings\n",
    "sentence_embedding_folder = \"/content/drive/MyDrive/Dataset/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell only once to save the news embeddings in the drive folder\n",
    "save_embedding_news(\"train\", sentence_embedding_folder)\n",
    "save_embedding_news(\"valid\", sentence_embedding_folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    data = load_data_mind(config, sentence_embedding_folder)\n",
    "except NameError:\n",
    "    data = load_data_mind(config)\n",
    "test_data = data[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Limit the user2item validation dataset, since otherwise the validation during training takes too long to run\n",
    "def limit_user2item_validation_data(data, size):\n",
    "    test_data = data[-1]\n",
    "    test_data_reduced = {key: test_data[key][:size] for key in test_data.keys()}\n",
    "    # Concatenate the old tuple with the updated validation data\n",
    "    return data[:-1] + (test_data_reduced,)\n",
    "\n",
    "data = limit_user2item_validation_data(data, 10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the KRED model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if train_type == \"single_task\":\n",
    "    single_task_training(config, data)\n",
    "else:\n",
    "    multi_task_training(config, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the KRED model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "testing(test_data, config)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performance of user2item on MINDsmall\n",
    "\n",
    "We tested the performance for the user-to-item task using the MINDsmall dataset, for your reference:\n",
    "\n",
    "| Models | AUC | NDCG@10 |\n",
    "| :------- | :------- | :------- |\n",
    "| KRED(single task training) | 0.6231 | 0.3628 |\n",
    "| KRED + multi-head attention (6 heads) |  0.5844 | 0.3452|\n",
    "| KRED + multi-head attention (12 heads) |  0.5845 | 0.3470|\n",
    "| KRED + news first category embedding |  0.6421 | 0.3837|\n",
    "| KRED + news second category embedding |  0.6218 | 0.3576|\n",
    "| KRED + news first and second categories embedding |  0.6397 | 0.3777|\n",
    "| KRED + multi-head attention (12 heads) + news first category embedding | 0.6074 | 0.3615|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[1] Liu, Danyang, et al. \"KRED: Knowledge-Aware Document Representation for News Recommendations.\" Fourteenth ACM Conference on Recommender Systems. 2020.\n",
    "\n",
    "[2] Wu, Fangzhao, et al. \"MIND: A Large-scale Dataset for News Recommendation\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}