Skip to content
This repository
branch: master
Octocat-spinner-32 PyML-0.7.9 final paper tex
Octocat-spinner-32 egpaper_final Added web version of paper
Octocat-spinner-32 neg Classifier works with ~65% accuracy in 3-fold validation
Octocat-spinner-32 neg_adj refiltered adjectives, was missing a tag before
Octocat-spinner-32 neg_position adding position tagged data, along with script to generate the data
Octocat-spinner-32 neg_tagged adding tagged data (using qtag)
Octocat-spinner-32 paper Merge branch 'master' of github.com:cathywu/Sentiment-Analysis
Octocat-spinner-32 pos Classifier works with ~65% accuracy in 3-fold validation
Octocat-spinner-32 pos_adj refiltered adjectives, was missing a tag before
Octocat-spinner-32 pos_position adding position tagged data, along with script to generate the data
Octocat-spinner-32 pos_tagged adding tagged data (using qtag)
Octocat-spinner-32 pysvmlight refactoring, ability to run svm in different configurations; + modifi…
Octocat-spinner-32 subjectivity Started working on majority voting
Octocat-spinner-32 .gitignore directory for latex report files
Octocat-spinner-32 COPYING Added license info
Octocat-spinner-32 Indexes.py built k-fold validation testing into bayes classification, and the re…
Octocat-spinner-32 LICENSE Added license info
Octocat-spinner-32 README added preprocessing for yelp data, filtering for verbs,docs for data …
Octocat-spinner-32 adjectives_filter.py added preprocessing for yelp data, filtering for verbs,docs for data …
Octocat-spinner-32 classifier.py Added binary flag to other classifiers
Octocat-spinner-32 data.db Added sqlite database of results from testing
Octocat-spinner-32 data.py Merge branch 'master' of github.com:cathywu/Sentiment-Analysis
Octocat-spinner-32 human.py script for generating human verification
Octocat-spinner-32 me.py maxent tweaking, adding presence vs frequency flag for bayes classifi…
Octocat-spinner-32 movie.py Merge branch 'master' of github.com:cathywu/Sentiment-Analysis
Octocat-spinner-32 ngrams.py Fixed underscores for tags in ngrams
Octocat-spinner-32 position_tagger.py removed yelp data from repo, modified position tagging script and adj…
Octocat-spinner-32 preprocess_yelp.py added preprocessing for yelp data, filtering for verbs,docs for data …
Octocat-spinner-32 svm.py refactoring, ability to run svm in different configurations; + modifi…
Octocat-spinner-32 svm2.py ported svm stuff from pysvmlight to pyml, running into memory problem…
Octocat-spinner-32 validate.py Merge branch 'master' of github.com:cathywu/Sentiment-Analysis
Octocat-spinner-32 verb_filter.py added preprocessing for yelp data, filtering for verbs,docs for data …
Octocat-spinner-32 yelp.py Added term frequency/inverse document frequency to ngrams
README
==============================================================================
Linking to Yelp dataset (via a symlink)
==============================================================================
ln -s $HOME/Dropbox/sentiment-data/yelp/ yelp

==============================================================================
Toolkits
==============================================================================
Oliver Mason's Qtag program [http://phrasys.net/uob/om/software]

==============================================================================
For setting up Maximum Entropy Modeling Toolkit for Python and C++
==============================================================================
Main page [http://homepages.inf.ed.ac.uk/lzhang10/maxent_toolkit.html]
Source [https://github.com/lzhang10/maxent]
Wonderful documentation, except for the missing Python API reference [http://homepages.inf.ed.ac.uk/lzhang10/software/maxent/manual.pdf]

DEPENDENCIES
zlib [http://www.techsww.com/tutorials/libraries/zlib/installation/installing_zlib_on_ubuntu_linux.php]
libboost [apt-get]
jam [apt-get]

Important points
* L-BFGS is the default parameter estimating method in this toolkit.

==============================================================================
Preprocess movie data
==============================================================================

Use Qtag with the "underscore" and "process all files in directory" options
$ java -jar qtag.jar

Move the POS tagged data out to its own directory, for further processing
$ mv pos/tagged/ pos_tagged
$ mv neg/tagged/ neg_tagged

Tag data with position
$ python position_tagger.py -d pos 
$ python position_tagger.py -d neg 

Filter out for only adjectives
$ python adjectives_filter.py -d neg
$ python adjectives_filter.py -d pos

Filter out for only verbs
$ python verb_filter.py -d pos
$ python verb_filter.py -d neg

==============================================================================
Preprocess Yelp data
==============================================================================

Make yelp data look like movie data in terms of formatting, and limit to 1000 
per star rating
$ python preprocess_yelp.py -d yelp/default/1star_limited
$ python preprocess_yelp.py -d yelp/default/2star_limited
$ python preprocess_yelp.py -d yelp/default/3star_limited
$ python preprocess_yelp.py -d yelp/default/4star_limited
$ python preprocess_yelp.py -d yelp/default/5star_limited

Use Qtag with the "underscore" and "process all files in directory" options
$ java -jar qtag.jar

Move the POS tagged data out to its own directory, for further processing
$ mv 1star_limited/tagged/ 1star_limited_tagged
$ mv 2star_limited/tagged/ 2star_limited_tagged
$ mv 3star_limited/tagged/ 3star_limited_tagged
$ mv 4star_limited/tagged/ 4star_limited_tagged
$ mv 5star_limited/tagged/ 5star_limited_tagged

Tag data with position
$ python position_tagger.py -d yelp/default/1star_limited
$ python position_tagger.py -d yelp/default/2star_limited
$ python position_tagger.py -d yelp/default/3star_limited
$ python position_tagger.py -d yelp/default/4star_limited
$ python position_tagger.py -d yelp/default/5star_limited

Filter out for only adjectives
$ python adjectives_filter.py -d yelp/default/1star_limited
$ python adjectives_filter.py -d yelp/default/2star_limited
$ python adjectives_filter.py -d yelp/default/3star_limited
$ python adjectives_filter.py -d yelp/default/4star_limited
$ python adjectives_filter.py -d yelp/default/5star_limited

Filter out for only verbs
$ python verb_filter.py -d yelp/default/1star_limited
$ python verb_filter.py -d yelp/default/2star_limited
$ python verb_filter.py -d yelp/default/3star_limited
$ python verb_filter.py -d yelp/default/4star_limited
$ python verb_filter.py -d yelp/default/5star_limited
Something went wrong with that request. Please try again.