Recommendation Algorithm

Problem Description

Our recommendation algorithm comprises of three steps:

• Seeding: select favourite movies as seeds for recommendation
• Associating: find similar movies based on the seeds
• Fitting: select based the projected score based on user history and public ratings

The latter two steps are very time consuming and take up most of the time of the recommending process for each user. Thus, the process must be optimised.

Algorithm Description

Seeding

During the seeding step, we select all the movies that are rated above 8 points by user. These movies are considered user's favourite movies, and we use these movies as the starting point to generate a pool of similar movies for further selection.

Associating

To create a pool of similar movies, we need to first calculate the similarity between each movie. By assigning different weights to data fields such as actors, directors, genres and the length of runtime, we manage to calculate a numeric value that represents similarity, ranged between 0 and 1, thereby generating the similar movie pool based on certain threshold. In our case, we used 0.5 as an arbitrary start.

Fitting

The public ratings are updated based on the following condition.

• If the vote count is smaller than 1000
• If the last updated date is 2 days ago

The first condition is to accommodate new movies and unpopular movies. They do not have a large viewer base, any new ratings might cause an observable change in the overall public ratings. The second condition is more than a routinely check for other movies.

Using the three public ratings and user ratings, we are able to create a multi-linear regression model. With this model, given the three public ratings of an unwatched movie, we are able to predict the expected score for that particular user.

Performance

A sequential process comprises of the above three steps takes roughly 5 mins to 6 mins per user. Upon examination, we discover that most of the time was spent on the calculation of similarity. Thus, we decided to optimise the process using the following solution.

Current Solutions

Associating

In order to reduce the amount of time spent calculating the similarity, we create a sparse matrix in database. A separate script will run continuously to calculate the similarities between user viewing histories and the entire movie databases, starting from the newest.

Fitting

Future Solutions