Analysis Work Flow

Clean up Method

• No lemmatization or stemming
• Remove all subtitle formatting
• Remove documents that are near duplicates (https://github.com/seomoz/simhash-py)
• Remove punctuation

Word2Vec Method

• Use gensim
• Minimum frequency is 5
• Number of dimensions: 50, 100, 200, 300, and 500
• Negative sampling k was 10
• Sub-sampling parameter to 1e-5
• Window size was 3 to 13 (i.e. 1 word before and after to 10 words before and after)
• CBOW and Skip Gram

Analysis Method

• See example of how to get out word vector for words by dimensions (https://towardsdatascience.com/using-word2vec-to-analyze-news-headlines-and-predict-article-success-cdeda5f14751)
• Grab words by dimensions
• Use linear regression to predict Y (varies by language) given the word2vec dimensions
• Potentially also consider using ridge regression to control for multicollinearity of the words by dimensions matrix
• Pick model with the highest R2 values (DECISION HERE: what is the best model? What about ties?)

Provide Output

• Include code to recreate these models
• Include word by dimension output as a data object
• Include functions to calculate cosine/other similarity measures on the word by dimension data object (if python, we could us the many measures in scipy.spatial.distance)
• Note that gensim's most_similar function provides the highest cosine on word by dimension data object