This project is deprecated and not maintained. It exists for historical reasons only.
This is the code I wrote for my Ph.D. dissertation at the University of Maryland (2012). It is primarily large-scale text reuse detection with basic sentiment analysis. It was used to construct a large data set containing press releases and related news coverage and analyze the relationship between these two groups. The results of the data analysis were used to challenge an existing social scientific theory.
Some of the the data collection and processing steps included:
- crawl a predetermined set of websites and extract relevant data from online newsrooms
- extract news articles from LexisNexis bulk downloads
- tokenize the corpus; identify instances of matching sequences of n tokens appearing in a press release and one or more news articles
- eliminate bad discriminators (non-unique token sequences appearing in more than one press release - i.e., corporate boilerplate doesn't help identify a relationship between a release and an article)
- run a sentence tokenizer + part-of-speech tagger; then attempt to match each token against a subjectivity lexicon; calculate document subjectivity and polarity scores
- ...and many more routine and ad-hoc data cleaning and processing tasks
- My dissertation is available here.
- A browsable HTML version of the constructed dataset (including all identified instances of text reuse + extracted quotes) is available here. (Here's a sample)
- My defense slide deck is a brief overview of the dissertation.
- Here's another slide deck from a follow-up study on a larger data set.
This is dissertation research code: it is neither elegant, nor reusable. It doesn't follow PEP 8 very closely. It is messy, suboptimal, and occasionally incomprehensible. It was written for Python 2, and it won't run under Python 3. More importantly, web scrapers are short-lived by definition; I doubt any of the scrapers in this codebase are still usable on their respective target websites.