Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Post-click Feeback for Recommender Systems

This is the code repo for our RecSys 2019 paper: Leveraging Post-click Feedback for Content Recommendations. In this paper, we leverage post-click feedback, e.g. skips and completions, to improve the training and evaluation of content recommenders. Check our paper for more details.


We used OpenRec to build our recommendation algorithms. It is built on the TensorFlow framework.

To install the dependencies needed for this repo:

$ ./scripts/


We randomly sampled data from two publicly-available datasets for experiments. For pre-processing, please refer to our paper.

  • ByteDance. Contains user interactions with short videos (average 10 seconds in length), including whether or not each video was completed.
  • Spotify. Contains music listening sessions across Spotify users. A user may skip or complete listening to each song.

After processing data into train, validaion, test sets, put them under a "dataset" folder. The directory format is referred to

You can also use to start with the data preprocessed by us:

$ ./scripts/

Updated link to processed dataset:

Train and Evaluate

To train a BPR-NR model on the Bytedance dataset under the post-click-aware evaluation metric (click-complete as positive observation, click-skip as negative observation, non-click as missing observation):

$ python3 --dataset=bytedance --l2_reg=0.01 --p_n_ratio=0.4 --eval_explicit

where p_n_ratio corresponds to the hyperparameter $\lambda_p,n$ in the paper. It controls the weights put on each types of signals, and can be any float between 0 to 1.

If you want to see the separated performance on click-skip and click-complete items, add the eval_rank flag:

$ python3 --dataset=bytedance --l2_reg=0.01 --p_n_ratio=0.4 --eval_explicit --eval_rank

If you want to evaluate on the click-only metric (click as positive, non-click as negative), remove the eval_explicit flag:

$ python3 --dataset=bytedance --l2_reg=0.01 --p_n_ratio=0.4

Similarly, to train a WRMF-NR model on the Spotify dataset, we use two hyperparameters to control the weights on positive and negative samples:

$ python3 --dataset=spotify --l2_reg=0.001 --pos_ratio=0.6 --neg_ratio=0.2 --eval_explicit


To replicate the experiments in the paper, refer to ./scripts/{bpr,wrmf} Assign dataset to be one of {bytedance, spotify} when you run the script. You can also add your own experiments following the instructions in those scripts:

$ ./scripts/ bytedance 


To cite our paper:

Hongyi Wen, Longqi Yang, and Deborah Estrin. 2019. Leveraging Post-click
Feedback for Content Recommendations. In Thirteenth ACM Conference on
Recommender Systems (RecSys ’19), September 16–20, 2019, Copenhagen, Denmark.
ACM, New York, NY, USA, 9 pages.
    title={Leveraging Post-click Feedback for Content Recommendations},
    author={Wen, Hongyi and Yang, Longqi and Estrin, Deborah},
    booktitle={Proceedings of the 13th ACM Conference on Recommender Systems},


If you have questions related to this repo, feel free to raise an issue, or contact us via:


Code for RecSys'19 paper: Leveraging Post-click Feedback for Content Recommendations







No releases published


No packages published