Repository navigation
Home
In this project we examined how recommender systems work (better or worse) if we take advantage of the review texts along with the review ratings. We aimed to combine latent ratings with latent review topics and analyze the results. Our assumption was that combining the review text with ratings would help the recommender system make better predictions. Hence, we compared different models on Amazon Apps dataset and calculated RMSE.
In today’s world of ubiquitous digital content, the number of choices we have are overwhelming. In an effort to understand how recommender systems can make better predictions, we decided to explore the impact of review text on suggesting relevant items to users and helping them discover the content they are looking for. We hypothesized that using review text would produce better recommendations than traditional ratings-only based approaches to recommendation systems.
- Baseline model
- Collaborative Filtering
- Latent Dirichlet Allocation (LDA)
- Hidden Factors as Topics (HFT)
We used the “Apps for Android” 5-core Dataset which contains both product ratings and reviews. The dataset contains 752,937 entries with 87,271 users and 13,209 products (mobile apps). The data is from Amazon.com and has been collected by Julian McAuley, UCSD.
Below is the snippet of the data:
{
"reviewerID": "A1N4O8VOJZTDVB",
"asin": "B004A9SDD8",
"reviewerName": "Annette Yancey",
"helpful": [1, 1],
"reviewText": "Loves the song, so he really couldn't wait to play this. A little less interesting for him so he doesn't play long, but he is almost 3 and likes to play the older games, but really cute for a younger child.", "overall": 3.0,
"summary": "Really cute",
"unixReviewTime": 1383350400,
"reviewTime": "11 2, 2013"
}
{
"reviewerID": "A2HQWU6HUKIEC7",
"asin": "B004A9SDD8",
"reviewerName": "Audiobook lover \"Kathy\"",
"helpful": [0, 0],
"reviewText": "Oh, how my little grandson loves this app. He's always asking for \"Monkey.\" Grandma has tired of it long before he has. Finding the items on each page that he can touch and activate is endlessly entertaining for him, at least for now. Well worth the $.99.", "overall": 5.0, "summary": "2-year-old loves it",
"unixReviewTime": 1323043200,
"reviewTime": "12 5, 2011"
}
About half of the items have ratings of 5.
Average rating is 3.96.
As shown in the second graph, most of the items have ratings around or over 4.
 |
|
| On an average, an item has 57 ratings. | On an average, a user rates about 9 items. |
 |
|
Every word in a review texts belongs to one or more topics. Using LDA, we can only observe the text and words, not the topic themselves. We tried to find these hidden topics in the text reviews and below is the distribution of these topics.
As visible from the table below, using the review text in combination with the numerical ratings hailed better results in terms of both RMSE and MAE. The HFT model performed the best with an RMSE of 1.1 on the whole 750K reviews, and an RMSE of 1.02 on 20K reviews. We did not run the Collaborative Filtering model or Latent Dirichlet Allocation model on the 750K dataset. Additionally, it is interesting to note that most of our baseline methods (using only ratings) performed either better or roughly the same on the larger dataset, whereas the HFT model performed slightly worse.
| Approach 0 (Global Mean) | Approach 1 (Biases) | Approach 2 (CF) | Approach 3 (SVD) | Approach 4 (LDA) | Approach 5 (HFT) | |
|---|---|---|---|---|---|---|
| RMSE | 1.32 | 1.23 | Mexico | 1.18 | Francisco Chang | 1.1 |
| MAE | 1.06 | 0.93 | Mexico | 0.9127 | Francisco Chang | 0.85 |
| Approach 0 (Global Mean) | Approach 1 (Biases) | Approach 2 (CF) | Approach 3 (SVD) | Approach 4 (LDA) | Approach 5 (HFT) | |
|---|---|---|---|---|---|---|
| RMSE | 1.26 | 1.37 | 1.33 | 1.18 | 1.4 | 1.02 |
| MAE | 0.98 | 1.00 | 0.93 | 0.91 | 1.03 | 0.76 |
Using review topics in conjunction with ratings gives better recommendations than traditional methods
It is difficult to make accurate predictions by relying on review text and biases alone (LDA)
HFT is dataset dependent – some categories of items have reviews that better express the subjective tastes of the user/properties of the item
Beyond a certain value, increasing the number of topics gives negligible improvements in RMSE
App reviews are short and to-the-point, and mostly positive
- Creating a test set such that all items & users are in the train set too.
- Creating an automated Amazon ASIN lookup tool which is usually a paid service
- LDA Topics are not well defined for the dataset we chose.
- No ground truth for the predictions by using review texts only.
- Word2Vec or GloVe can be used as part of hidden topic extraction from reviews
- The HFT model may be adapted to work with implicit data instead of ratings, along with reviews.
- Recommending violent/inappropriate apps to children?
- User profiles used for personalized recommendation may be used for malicious purposes like
- Profile injection is also a possible attack to push or nuke items.
Julian McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions
with review text. In Proceedings of the 7th ACM conference on Recommender systems (RecSys ’13). ACM, New
York, NY, USA, 165-172. DOI=http://dx.doi.org/10.1145/2507157.2507163