Experiments of the "Widespread flaws in offline evaluation of recommender systems" paper
- Clone the repository.
- In the root folder of the repository, run the following command to initialize the submodules:
git submodule update --init --recursive
- 📝 NOTE: If the submodules would get some major update and pulling the latest versions would be necessary, run the following command:
git submodule update --remote --recursive
- 📝 NOTE: If the submodules would get some major update and pulling the latest versions would be necessary, run the following command:
-
Install NVIDIA Driver
- make sure the driver supports CUDA Toolkit version 11.3.1
- cuda-drivers is enough, but installing both cuda-drivers and cuda-toolkit is correct as well (the environemnt described in the following steps, will install cudatoolkit, and makes sure it uses the environemnt's cudatoolkit instead of a global version).
-
Install Anaconda by following the instructions here. You can choose either Anaconda or Miniconda.
-
Conda Environments
- For any jupyter-notebook (experiments, vsknn train / test, GRU4Rec train / test, etc.) you can use the environment described in
conda_eval_flaws_env.yml. To create the eval_flaws environment, run the following command:conda env create -f eval_flaws_env.yml
- GRU4Rec experiments require the gru4rec_theano_gpu environment. Training and Testing will be ran automatically when necessary at the beginning of the experiment notebooks. (Training and testing can be executed manually as well, from the gru4rec_theano_gpu environment.) To create the environment, please run:
bash conda_gru4rec_theano_gpu_install.sh
- Installation using
conda_gru4rec_theano_gpu_install.shis strongly advised, but the environment can also be directly created with conda from theconda_gru4rec_theano_gpu.yml. However the installation script makes some extra steps to ensure the environment uses the correct cudatoolkit (which is installed by the environemnt, avoiding collusion with the system cudatoolkit), in this case, these steps must be perfomed manually (refer toconda_gru4rec_theano_gpu_install.shfor the exact steps).
- Installation using
- 📝 NOTE: The installation process might take a few minutes, it is normal.
- 📝 NOTE: For the notebooks the eval_flaws environment should be used. When gru4rec training or testing scripts are executed automatically from the notebooks, it is ensured that they will be executed from the gru4rec_theano_gpu environment.
- For any jupyter-notebook (experiments, vsknn train / test, GRU4Rec train / test, etc.) you can use the environment described in
Run the notebooks
- Rees46:
- Coveo:
- Retailrocket:
- Amazon Beauty:
- MovieLens 10M:
- Steam Games:
- Yelp: