Here is the repository for our course project at Vanderbilt: CS 3891 Special Topics (2020 Spring). We applied machine learning techniques in this project and worked on rating tags prediction and rating visualization on radar chart based on ratemyprofessors.com.
For rating tags prediction, we employ multi-label classification techniques such as [TODO @Amy].
For radar chart visualization, we first perform topic modelling to gain insight into the data and choose dimensions of the radar chart. Then we manually label data for each category and apply BERT embedding to train a multi-class classification model. Finally, we calculate the score based on normalization and display the visualization on a simple web server.
[TODO @ Ao] Sample Image
Link to final presentation slides
We wrote a web scraper that connects to the site's public API to obtain data for our training. Check out the Jupyter notebook demo for a sample output. Also, check out a simple analysis on the sample data.
We scraped the top 100 universities of 2021 Best U.S. University Rankings.
- School folder contains all professors' information of each university, such as professor's name and overall rating score.
- Rating folder has all the review details for every professor of each university, such as comments, review date, or help count.
- In the above two folders, the file name corresponds to the university's I.D. number on the site, and this sheet pairs each university's name with its I.D.
Rate My Professors is a popular site among college students, allowing students to rate their professors and campuses. It was founded in May 1999 by a Menlo Park, CA-based software engineer named John Swapceinski. It is currently the largest professor rating site, boasting over 1.7 million professor reviews in over 7500 campuses across the U.S., U.K., and Canada. [1] It is beneficial as it provides references to students and feedback to professors. For example, first-year college students can view the professors' performances on the campus, which aids them to choose courses.
Professor ratings are closely related to college student choices. Given the site's enormous data, we focus on what insight we may get and how to improve the user experience by applying machine learning techniques. Through discussion of several ideas, we chose the following two to implement in the course:
Goal: Automate the process of assigning tags to reviews based on their content. In other words, classify content into one or more classes.
Motivation: Many reviews on the Ratemyprofessor site don't have tags because the site introduces tagging feature afterwards. Therefore, the summary statistics on each professor's Ratemyprofessor web page may be misleading due to incomplete tagging.
Expected Outcome: The model performance will be evaluated using metrics that we commonly use for classification tasks such as cross-entropy or F1-score.
Goal: Summarize the ratings into numeric values on multiple dimensions and visualize the summarization through Radar Chart.
Motivation: Currently, on the site, we have tags associated with each professor. However, we wonder if there is a more intuitive way of summarizing the data? We propose translating each professor's ratings into numeric values of multiple dimensions through visualization tools and machine learning. For example, we can have a Radar Chart with six angles, which stands for the different dimensions. One dimension might be the 'amount of homework.' A higher number indicates there's much homework. We assigned scores to each of them and highlight the area enclosed by the shape. We believe that through visualization techniques like this, students obtain information more efficiently.
Expected Outcome: Since we don't have a ground truth for the task, we might manually label the data for classification purposes. Besides, to test if our tool helps increase efficiency, we may invite students to use the product and hear their feedback. Moreover, it might be helpful to have a tooltip integrated with the site to display our visualization.
-
Data Prep: Since we will mainly focused on the review and tags, we seperate these columns from the whole dataset. After we remove all the comments with no tags, we follow the standard preprocessing process:
- Tokenization: splitting every comments into tokens.
- Nomalization: transverting every words/tokens to their original forms
- Removal of stop words: remove all the stop words(words that is too frequently used to be meaningful) from comments
-
Data Visualization:
We have finish the visualization of all the 20 tags and get a clear distribution. We can see that the number of comments for each tags varies a lot, which need to be balanced for a better modeling experience.
We also check the number of tags for every comments. Since the maximum number of tags in RateMyProfessor website is three, we can see only three tag numbers in thhis chart. Most comments includes three tags, while some of them includes one or two. The numbers remain a ratio about 1: 2: 7.
- Word Embedding:
Here we are using our self trained word2vec model for process the dataset. Basically, Word2Vec represents each distinct word with different vectors. Due to the certain limitation of our dataset, like 'A' means a grade but not a determiner, we decided to train the word2vec model ourselves.
First of all, we need to understand what is multi-label classification. There are three kinds of classification tasks:
- Binary classification: two exclusive classes
- Multi-class classification: more than two exclusive classes
- Multi-label classification: non-exclusive classes
Rating tags prediction, in our case, would be a sample of multi-label classfication. To constructing a multi-label classification model we exxperienced with several choices:
- Simple Machine Learning Algorithms: Logistic regression
- Deep Learning Methods: LSTM, BiLSTM, CNN
Our base model uses TFIDF together with Logisitic regression and reached an average validation accuracy of 0.7899 with an average f1-score of 0.3584.
We picked self-trained Word2Vec word embedding together with BiLSTM with CNN using binary cross entropy as loss function as our final model. It reached an average validation accuracy of 0.8275, recall of 0.7702, and an average f1-score of 0.7795.
Please refer here 😊 for detailed information.
[1] Rate My Professors. (2020). Google Sites.