examples/00_quick_start/xdeepfm_criteo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# xDeepFM : the eXtreme Deep Factorization Machine \n",
    "This notebook will give you a quick example of how to train an [xDeepFM model](https://arxiv.org/abs/1803.05170). \n",
    "xDeepFM \\[1\\] is a deep learning-based model aims at capturing both lower- and higher-order feature interactions for precise recommender systems. Thus it can learn feature interactions more effectively and manual feature engineering effort can be substantially reduced. To summarize, xDeepFM has the following key properties:\n",
    "* It contains a component, named CIN, that learns feature interactions in an explicit fashion and in vector-wise level;\n",
    "* It contains a traditional DNN component that learns feature interactions in an implicit fashion and in bit-wise level.\n",
    "* The implementation makes this model quite configurable. We can enable different subsets of components by setting hyperparameters like `use_Linear_part`, `use_FM_part`, `use_CIN_part`, and `use_DNN_part`. For example, by enabling only the `use_Linear_part` and `use_FM_part`, we can get a classical FM model.\n",
    "\n",
    "In this notebook, we test xDeepFM on two datasets: 1) a small synthetic dataset and 2) [Criteo dataset](http://labs.criteo.com/category/dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Global Settings and Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.11 | packaged by conda-forge | (default, Aug  5 2020, 20:09:42) \n",
      "[GCC 7.5.0]\n",
      "Tensorflow version: 1.15.2\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import os\n",
    "import scrapbook as sb\n",
    "from tempfile import TemporaryDirectory\n",
    "import tensorflow as tf\n",
    "tf.get_logger().setLevel('ERROR') # only show error messages\n",
    "\n",
    "from recommenders.utils.constants import SEED\n",
    "from recommenders.models.deeprec.deeprec_utils import (\n",
    "    download_deeprec_resources, prepare_hparams\n",
    ")\n",
    "from recommenders.models.deeprec.models.xDeepFM import XDeepFMModel\n",
    "from recommenders.models.deeprec.io.iterator import FFMTextIterator\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Tensorflow version: {}\".format(tf.__version__))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "EPOCHS_FOR_SYNTHETIC_RUN = 15\n",
    "EPOCHS_FOR_CRITEO_RUN = 10\n",
    "BATCH_SIZE_SYNTHETIC = 128\n",
    "BATCH_SIZE_CRITEO = 4096\n",
    "RANDOM_SEED = SEED  # Set to None for non-deterministic result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "xDeepFM uses the FFM format as data input: `<label> <field_id>:<feature_id>:<feature_value>`  \n",
    "Each line represents an instance, `<label>` is a binary value with 1 meaning positive instance and 0 meaning negative instance. \n",
    "Features are divided into fields. For example, user's gender is a field, it contains three possible values, i.e. male, female and unknown. Occupation can be another field, which contains many more possible values than the gender field. Both field index and feature index are starting from 1. <br>\n",
    "\n",
    "## 1. Synthetic data\n",
    "Now let's start with a small synthetic dataset. In this dataset, there are 10 fields, 1000 fefatures, and label is generated according to the result of a set of preset pair-wise feature interactions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10.3k/10.3k [00:01<00:00, 5.24kKB/s]\n"
     ]
    }
   ],
   "source": [
    "tmpdir = TemporaryDirectory()\n",
    "data_path = tmpdir.name\n",
    "yaml_file = os.path.join(data_path, r'xDeepFM.yaml')\n",
    "train_file = os.path.join(data_path, r'synthetic_part_0')\n",
    "valid_file = os.path.join(data_path, r'synthetic_part_1')\n",
    "test_file = os.path.join(data_path, r'synthetic_part_2')\n",
    "output_file = os.path.join(data_path, r'output.txt')\n",
    "\n",
    "if not os.path.exists(yaml_file):\n",
    "    download_deeprec_resources(r'https://recodatasets.z20.web.core.windows.net/deeprec/', data_path, 'xdeepfmresources.zip')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1 Prepare hyper-parameters\n",
    "prepare_hparams() will create a full set of hyper-parameters for model training, such as learning rate, feature number, and dropout ratio. We can put those parameters in a yaml file, or pass parameters as the function's parameters (which will overwrite yaml settings)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kg_file=None,user_clicks=None,FEATURE_COUNT=1000,FIELD_COUNT=10,data_format=ffm,PAIR_NUM=None,DNN_FIELD_NUM=None,n_user=None,n_item=None,n_user_attr=None,n_item_attr=None,iterator_type=None,SUMMARIES_DIR=None,MODEL_DIR=None,wordEmb_file=None,entityEmb_file=None,contextEmb_file=None,news_feature_file=None,user_history_file=None,use_entity=True,use_context=True,doc_size=None,history_size=None,word_size=None,entity_size=None,entity_dim=None,entity_embedding_method=None,transform=None,train_ratio=None,dim=10,layer_sizes=[100, 100],cross_layer_sizes=[1],cross_layers=None,activation=['relu', 'relu'],cross_activation=identity,user_dropout=False,dropout=[0.0, 0.0],attention_layer_sizes=None,attention_activation=None,attention_dropout=0.0,model_type=xDeepFM,method=classification,load_saved_model=False,load_model_name=you model path,filter_sizes=None,num_filters=None,mu=None,fast_CIN_d=0,use_Linear_part=False,use_FM_part=False,use_CIN_part=True,use_DNN_part=False,init_method=tnormal,init_value=0.3,embed_l2=0.0001,embed_l1=0.0,layer_l2=0.0001,layer_l1=0.0,cross_l2=0.0001,cross_l1=0.0,reg_kg=0.0,learning_rate=0.001,lr_rs=1,lr_kg=0.5,kg_training_interval=5,max_grad_norm=2,is_clip_norm=0,dtype=32,loss=log_loss,optimizer=adam,epochs=15,batch_size=128,enable_BN=False,show_step=200000,save_model=False,save_epoch=2,metrics=['auc', 'logloss'],write_tfevents=False,item_embedding_dim=None,cate_embedding_dim=None,user_embedding_dim=None,train_num_ngs=4,need_sample=True,embedding_dropout=0.0,user_vocab=None,item_vocab=None,cate_vocab=None,pairwise_metrics=None,EARLY_STOP=100,max_seq_length=None,hidden_size=None,L=None,T=None,n_v=None,n_h=None,min_seq_length=1,attention_size=None,att_fcn_layer_sizes=None,dilations=None,kernel_size=None,embed_size=None,n_layers=None,decay=None,eval_epoch=None,top_k=None\n"
     ]
    }
   ],
   "source": [
    "hparams = prepare_hparams(yaml_file, \n",
    "                          FEATURE_COUNT=1000, \n",
    "                          FIELD_COUNT=10, \n",
    "                          cross_l2=0.0001, \n",
    "                          embed_l2=0.0001, \n",
    "                          learning_rate=0.001, \n",
    "                          epochs=EPOCHS_FOR_SYNTHETIC_RUN,\n",
    "                          batch_size=BATCH_SIZE_SYNTHETIC)\n",
    "print(hparams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2 Create data loader\n",
    "Designate a data iterator for the model. xDeepFM uses FFMTextIterator. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_creator = FFMTextIterator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3 Create model\n",
    "When both hyper-parameters and data iterator are ready, we can create a model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Add CIN part.\n"
     ]
    }
   ],
   "source": [
    "model = XDeepFMModel(hparams, input_creator, seed=RANDOM_SEED)\n",
    "\n",
    "## sometimes we don't want to train a model from scratch\n",
    "## then we can load a pre-trained model like this: \n",
    "#model.load_model(r'your_model_path')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see what is the model's performance at this point (without starting training):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'auc': 0.5043, 'logloss': 0.7515}\n"
     ]
    }
   ],
   "source": [
    "print(model.run_eval(test_file))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AUC=0.5 is a state of random guess. We can see that before training, the model behaves like random guessing.\n",
    "\n",
    "#### 1.4 Train model\n",
    "Next we want to train the model on a training set, and check the performance on a validation dataset. Training the model is as simple as a function call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "at epoch 1\n",
      "train info: logloss loss:0.7556826068773302\n",
      "eval info: auc:0.504, logloss:0.7042\n",
      "at epoch 1 , train time: 4.3 eval time: 0.6\n",
      "at epoch 2\n",
      "train info: logloss loss:0.7263523231666932\n",
      "eval info: auc:0.5066, logloss:0.6973\n",
      "at epoch 2 , train time: 4.3 eval time: 0.8\n",
      "at epoch 3\n",
      "train info: logloss loss:0.7177084291104189\n",
      "eval info: auc:0.5099, logloss:0.6953\n",
      "at epoch 3 , train time: 3.8 eval time: 0.7\n",
      "at epoch 4\n",
      "train info: logloss loss:0.7118660186983875\n",
      "eval info: auc:0.5147, logloss:0.6946\n",
      "at epoch 4 , train time: 3.7 eval time: 0.8\n",
      "at epoch 5\n",
      "train info: logloss loss:0.7055103289302682\n",
      "eval info: auc:0.523, logloss:0.6941\n",
      "at epoch 5 , train time: 3.8 eval time: 0.7\n",
      "at epoch 6\n",
      "train info: logloss loss:0.6954095556154284\n",
      "eval info: auc:0.5416, logloss:0.6929\n",
      "at epoch 6 , train time: 3.6 eval time: 0.8\n",
      "at epoch 7\n",
      "train info: logloss loss:0.6723950118133702\n",
      "eval info: auc:0.5916, logloss:0.6831\n",
      "at epoch 7 , train time: 4.0 eval time: 0.7\n",
      "at epoch 8\n",
      "train info: logloss loss:0.6119807973964927\n",
      "eval info: auc:0.7024, logloss:0.6288\n",
      "at epoch 8 , train time: 4.2 eval time: 0.8\n",
      "at epoch 9\n",
      "train info: logloss loss:0.5020270644594303\n",
      "eval info: auc:0.8154, logloss:0.5257\n",
      "at epoch 9 , train time: 4.3 eval time: 0.8\n",
      "at epoch 10\n",
      "train info: logloss loss:0.38994721433346213\n",
      "eval info: auc:0.8826, logloss:0.4315\n",
      "at epoch 10 , train time: 4.4 eval time: 0.7\n",
      "at epoch 11\n",
      "train info: logloss loss:0.30144522643785704\n",
      "eval info: auc:0.9205, logloss:0.3605\n",
      "at epoch 11 , train time: 4.4 eval time: 0.6\n",
      "at epoch 12\n",
      "train info: logloss loss:0.23429049589892023\n",
      "eval info: auc:0.9431, logloss:0.3082\n",
      "at epoch 12 , train time: 4.2 eval time: 0.6\n",
      "at epoch 13\n",
      "train info: logloss loss:0.18283964168677216\n",
      "eval info: auc:0.9577, logloss:0.2682\n",
      "at epoch 13 , train time: 3.9 eval time: 0.7\n",
      "at epoch 14\n",
      "train info: logloss loss:0.14280925843467826\n",
      "eval info: auc:0.9676, logloss:0.2369\n",
      "at epoch 14 , train time: 3.9 eval time: 0.7\n",
      "at epoch 15\n",
      "train info: logloss loss:0.11175788457655825\n",
      "eval info: auc:0.9745, logloss:0.2128\n",
      "at epoch 15 , train time: 3.7 eval time: 0.7\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<recommenders.models.deeprec.models.xDeepFM.XDeepFMModel at 0x7f9d74f7ff60>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_file, valid_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.5 Evaluate model\n",
    "\n",
    "Again, let's see what is the model's performance now (after training):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'auc': 0.9716, 'logloss': 0.2278}\n"
     ]
    }
   ],
   "source": [
    "res_syn = model.run_eval(test_file)\n",
    "print(res_syn)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/scrapbook.scrap.json+json": {
       "data": {
        "auc": 0.9716,
        "logloss": 0.2278
       },
       "encoder": "json",
       "name": "res_syn",
       "version": 1
      }
     },
     "metadata": {
      "scrapbook": {
       "data": true,
       "display": false,
       "name": "res_syn"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sb.glue(\"res_syn\", res_syn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to get the full prediction scores rather than evaluation metrics, we can do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<recommenders.models.deeprec.models.xDeepFM.XDeepFMModel at 0x7f9d74f7ff60>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(test_file, output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Criteo data \n",
    "\n",
    "Now we have successfully launched an experiment on a synthetic dataset. Next let's try something on a real world dataset, which is a small sample from [Criteo dataset](http://labs.criteo.com/category/dataset). Criteo dataset is a well known industry benchmarking dataset for developing CTR prediction models and it's frequently adopted as evaluation dataset by research papers. The original dataset is too large for a lightweight demo, so we sample a small portion from it as a demo dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "demo with Criteo dataset\n"
     ]
    }
   ],
   "source": [
    "print('demo with Criteo dataset')\n",
    "hparams = prepare_hparams(yaml_file, \n",
    "                          FEATURE_COUNT=2300000, \n",
    "                          FIELD_COUNT=39, \n",
    "                          cross_l2=0.01, \n",
    "                          embed_l2=0.01, \n",
    "                          layer_l2=0.01,\n",
    "                          learning_rate=0.002, \n",
    "                          batch_size=BATCH_SIZE_CRITEO, \n",
    "                          epochs=EPOCHS_FOR_CRITEO_RUN, \n",
    "                          cross_layer_sizes=[20, 10], \n",
    "                          init_value=0.1, \n",
    "                          layer_sizes=[20,20],\n",
    "                          use_Linear_part=True, \n",
    "                          use_CIN_part=True, \n",
    "                          use_DNN_part=True)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_file = os.path.join(data_path, r'cretio_tiny_train')\n",
    "valid_file = os.path.join(data_path, r'cretio_tiny_valid')\n",
    "test_file = os.path.join(data_path, r'cretio_tiny_test')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Add linear part.\n",
      "Add CIN part.\n",
      "Add DNN part.\n"
     ]
    }
   ],
   "source": [
    "model = XDeepFMModel(hparams, FFMTextIterator, seed=RANDOM_SEED)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'auc': 0.4728, 'logloss': 0.7113}\n"
     ]
    }
   ],
   "source": [
    "# check the predictive performance before the model is trained\n",
    "print(model.run_eval(test_file)) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "at epoch 1\n",
      "train info: logloss loss:744.3602027893066\n",
      "eval info: auc:0.6637, logloss:0.5342\n",
      "at epoch 1 , train time: 21.7 eval time: 4.3\n",
      "at epoch 2\n",
      "train info: logloss loss:385.66927337646484\n",
      "eval info: auc:0.7137, logloss:0.5109\n",
      "at epoch 2 , train time: 21.4 eval time: 4.3\n",
      "at epoch 3\n",
      "train info: logloss loss:191.50830841064453\n",
      "eval info: auc:0.7283, logloss:0.5037\n",
      "at epoch 3 , train time: 21.4 eval time: 4.2\n",
      "at epoch 4\n",
      "train info: logloss loss:92.20774269104004\n",
      "eval info: auc:0.7359, logloss:0.4991\n",
      "at epoch 4 , train time: 21.6 eval time: 4.4\n",
      "at epoch 5\n",
      "train info: logloss loss:43.159456968307495\n",
      "eval info: auc:0.74, logloss:0.4963\n",
      "at epoch 5 , train time: 21.6 eval time: 4.3\n",
      "at epoch 6\n",
      "train info: logloss loss:19.656921446323395\n",
      "eval info: auc:0.7426, logloss:0.4946\n",
      "at epoch 6 , train time: 21.3 eval time: 4.2\n",
      "at epoch 7\n",
      "train info: logloss loss:8.77035716176033\n",
      "eval info: auc:0.7441, logloss:0.4934\n",
      "at epoch 7 , train time: 21.5 eval time: 4.3\n",
      "at epoch 8\n",
      "train info: logloss loss:3.9227354377508163\n",
      "eval info: auc:0.7453, logloss:0.4925\n",
      "at epoch 8 , train time: 21.7 eval time: 4.3\n",
      "at epoch 9\n",
      "train info: logloss loss:1.8598770573735237\n",
      "eval info: auc:0.7462, logloss:0.4917\n",
      "at epoch 9 , train time: 21.3 eval time: 4.2\n",
      "at epoch 10\n",
      "train info: logloss loss:1.0249397158622742\n",
      "eval info: auc:0.747, logloss:0.491\n",
      "at epoch 10 , train time: 21.5 eval time: 4.3\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<recommenders.models.deeprec.models.xDeepFM.XDeepFMModel at 0x7f9d64b4a2e8>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_file, valid_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'auc': 0.7356, 'logloss': 0.5017}\n"
     ]
    }
   ],
   "source": [
    "# check the predictive performance after the model is trained\n",
    "res_real = model.run_eval(test_file)\n",
    "print(res_real)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/scrapbook.scrap.json+json": {
       "data": {
        "auc": 0.7356,
        "logloss": 0.5017
       },
       "encoder": "json",
       "name": "res_real",
       "version": 1
      }
     },
     "metadata": {
      "scrapbook": {
       "data": true,
       "display": false,
       "name": "res_real"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sb.glue(\"res_real\", res_real)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cleanup\n",
    "tmpdir.cleanup()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "\\[1\\] Lian, J., Zhou, X., Zhang, F., Chen, Z., Xie, X., & Sun, G. (2018). xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery \\& Data Mining, KDD 2018, London, UK, August 19-23, 2018.<br>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python (reco_gpu)",
   "language": "python",
   "name": "reco_gpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}