README.md

Model

• Model code are all in the ./Model/ folder
• These include models such as..
  • UserKNN
  • ItemKNN
  • RP3beta
  • Mult-VAE
• Check Parser.py for the list of hyperparameters available for each model

Acknowledgement

• We are using the implicit library for Weighted Matrix Factorization (WMF)
• UserKNN, ItemKNN, RP3beta, and all their associated code are from the RecSys 2019 Deep Learning Evaluation GitHub repository
• Mult-VAE is written in Tensorflow, by referencing the notebook provided by the paper's authors

Thank you!! :)

Running the Models

1. Option 1 (Run using specific hyperparameter settings and/or your own grid search)

   • For UserKNN, ItemKNN, RP3beta, and WMF: train.py

     • E.g. python3 train.py -d "ML-100K" -m "UserKNNCF"

   • For Mult-VAE: MultiVAE_train.py

     • E.g. python3 MultiVAE_train.py -d "ML-100K" -n_epochs 200 -num_hidden 1 -beta 0.1
     • For Mult-VAE, we used the following shell scripts for the experiments in our paper:
       • ML-100K: script_ML100K_MultiVAE.sh
       • Amazon (Video Games): script_AmazonVideoGames_MultiVAE.sh

2. Option 2 (Bayesian Hyperparameter Optimization, using scikit-optimize)

   • We used this option to automatically tune the hyperparameters for UserKNN, ItemKNN, RP3beta, and WMF in our paper
   • The code is in hyperOpt_train.py
   • Examples
     • ML-100K: script_ML100K.sh
     • Amazon (Video Games): script_AmazonVideoGames.sh

Experimental Results

• The results are stored in the ./logs/ folder
  • Results for a particular dataset and model can be found in ./logs/{dataset}/{model}/
• There are some helper files to 'sort' and 'gather' those results, i.e. sort_results.py, gather_results.py, and utilities_results.py
• For example, if we consider the Recall @ 10 metric, there are 3 files in the ./logs/ folder:
  • ___results_summary___Rec_10.txt shows the model performance
  • ___results_summary___Rec_10__bar.png contains the bar plot
  • ___results_summary___Rec_10__table.png shows the relative performance of Model X (row) over Model Y (column)
    • E.g. for the Amazon (Electronics) dataset in Cluster 1, the value at (Row 1, Column 2) indicates the relative improvement of RP3beta over ItemKNN
    • Values in light green (also indicated with a *) are statistically significant with a p-value < 0.05
    • Values in dark green (also indicated with a **) are statistically significant with a p-value < 0.01

Experimental Results (Recall @ 10)

Experimental Results (nDCG @ 10)