If you came here from the CHUG, you might want to see the slide deck here #Introduction This is an illustrative project intended to demonstrate a simple, yet not quite as simple as Word Count, NLP algorithm implemented in Hadoop Map Reduce. This is also intended to demonstrate my opinions on unit testing Map Reduce jobs.
#Description The purpose of the project is to take a dataset of around 1400 speeches from Senators along with an ideal-point mapping onto the number line of the politicians based around the work of Simon Jackman to do the following:
- Find partition points separating Conservative, Moderate and Liberal senators by dividing the probability density into thirds (see histogram)
- Use this partition to construct three corpuses associated with Conservative, Moderate and Liberal senators
- Preprocess using a Porter Stemmer and rank the tokenized terms by Inverse Document Frequency
- Take the ranked terms and output the terms that are important in the corpus associated with one political orientation but not the others.
##Partition The partition of the probability space is split into even thirds as can be seen here:
##Output The results, for the curious, are here:
| LIBERAL | MODERATE | CONSERVATIVE |
| market | produce | discuss |
| power | commission | past |
| institutions | strengthening | politics |
| initiate | provisions | pointed |
| implement | subcommittee | direct |
| establish | impact | different |
| history | north | account |
| available | managers | failed |
| school | sure | debate |
| according | hard | instead |
| risk | sent | rates |
| measures | defense | reason |
| children | fiscal | congressional |
| expand | honorable | budgeting |
| trained | ability | question |
#Current Caveats
- A naive partition based on an even partition of the probability space may be unsuitable. I didn't know of a better, politically agnostic way to do this.
- An analysis based on n-grams would be more appropriate or at least a more intelligent chunking algorithm so that words like "north korea" are not split.
- This is not as efficient as it could be. I could have used term IDs instead of the actual terms, but I wanted the Map Reduce job to be as clear as possible
This code is mostly an intellectual lark and a demonstration of NLP done using Map Reduce that does not require the baggage of Mahout.
#Usage ##Prerequisites To execute this, you must have:
- A JDK installed
- Maven 2 or higher installed
To generate the histogram, you need to have R. To generate the presentation from the Cleveland CHUG, you need latex
##To generate the full example From the command line, in the political-nlp-analysis directory, execute
mvn integration-test
You can then use the political-nlp-analysis/generate_top_words.sh script to generate the top words for a particular political orientation thusly:
generate_top_words.sh {LIBERAL, MODERATE, CONSERVATIVE} <number of terms>
You can reconstruct the density plot using the political-nlp-analysis/src/main/R/generate_histogram.sh script:
generate_histogram.sh <path to ideal_points.csv>
#Contact
If anyone has comments, concerns or criticisms, please let me hear about it at cestella@gmail.com