Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Train NLTK objects with zero code
Python Shell
tree: f8942d4069

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
docs
nltk_trainer
tests
.hgignore
LICENSE
README.rst
analyze_chunked_corpus.py
analyze_chunker_coverage.py
analyze_classifier_coverage.py
analyze_tagged_corpus.py
analyze_tagger_coverage.py
categorized_corpus2csv.py
classify_corpus.py
combine_classifiers.py
requirements.txt
setup.py
tag_phrases.py
train_chunker.py
train_classifier.py
train_tagger.py
translate_corpus.py

README.rst

NLTK Trainer

NLTK Trainer exists to make training and evaluating NLTK objects as easy as possible.

Requirements

You must have Python 2.6 with argparse and NLTK 2.0 installed. NumPy, SciPy, and megam are recommended for training Maxent classifiers.

Training Classifiers

Example usage with the movie_reviews corpus can be found in Training Binary Text Classifiers with NLTK Trainer.

Train a binary NaiveBayes classifier on the movie_reviews corpus, using paragraphs as the training instances::
python train_classifier.py --instances paras --classifier NaiveBayes movie_reviews
Include bigrams as features::
python train_classifier.py --instances paras --classifier NaiveBayes --ngrams 1 --ngrams 2 movie_reviews
Minimum score threshold::
python train_classifier.py --instances paras --classifier NaiveBayes --ngrams 1 --ngrams 2 --min_score 3 movie_reviews
Maximum number of features::
python train_classifier.py --instances paras --classifier NaiveBayes --ngrams 1 --ngrams 2 --max_feats 1000 movie_reviews
Use the default Maxent algorithm::
python train_classifier.py --instances paras --classifier Maxent movie_reviews
Use the MEGAM Maxent algorithm::
python train_classifier.py --instances paras --classifier MEGAM movie_reviews
Train on files instead of paragraphs::
python train_classifier.py --instances files --classifier MEGAM movie_reviews
Train on sentences::
python train_classifier.py --instances sents --classifier MEGAM movie_reviews
Evaluate the classifier by training on 3/4 of the paragraphs and testing against the remaing 1/4, without pickling::
python train_classifier.py --instances paras --classifier NaiveBayes --fraction 0.75 --no-pickle movie_reviews
For a complete list of usage options::
python train_classifier.py --help

Using a Trained Classifier

You can use a trained classifier by loading the pickle file using nltk.data.load::
>>> import nltk.data
>>> classifier = nltk.data.load("classifiers/NAME_OF_CLASSIFIER.pickle")
Or if your classifier pickle file is not in a nltk_data subdirectory, you can load it with pickle.load::
>>> import pickle
>>> classifier = pickle.load(open("/path/to/NAME_OF_CLASSIFIER.pickle"))

Either method will return an object that supports the ClassifierI interface.

Once you have a classifier object, you can use it to classify word features with the classifier.classify(feats) method, which returns a label::
>>> words = ['some', 'words', 'in', 'a', 'sentence']
>>> feats = dict([(word, True) for word in words])
>>> classifier.classify(feats)
If you used the --ngrams option with values greater than 1, you should include these ngrams in the dictionary using nltk.util.ngrams(words, n)::
>>> from nltk.util import ngrams
>>> words = ['some', 'words', 'in', 'a', 'sentence']
>>> feats = dict([(word, True) for word in words + ngrams(words, n)])
>>> classifier.classify(feats)

The list of words you use for creating the feature dictionary should be created by tokenizing the appropriate text instances: sentences, paragraphs, or files depending on the --instances option.

Training Part of Speech Taggers

The train_tagger.py script can use any corpus included with NLTK that implements a tagged_sents() method. It can also train on the timit corpus, which includes tagged sentences that are not available through the TimitCorpusReader.

Example usage can be found in Training Part of Speech Taggers with NLTK Trainer.

Train the default sequential backoff tagger on the treebank corpus::
python train_tagger.py treebank
To use a brill tagger with the default initial tagger::
python train_tagger.py treebank --brill
To train a NaiveBayes classifier based tagger, without a sequential backoff tagger::
python train_tagger.py treebank --sequential '' --classifier NaiveBayes
To train a unigram tagger::
python train_tagger.py treebank --sequential u
To train on the switchboard corpus::
python train_tagger.py switchboard
To train on a custom corpus, whose fileids end in ".pos", using a TaggedCorpusReader::
python train_tagger.py /path/to/corpus --reader nltk.corpus.reader.tagged.TaggedCorpusReader --fileids '.+\.pos'

The corpus path can be absolute, or relative to a nltk_data directory. For example, both corpora/treebank/tagged and /usr/share/nltk_data/corpora/treebank/tagged will work.

You can also restrict the files used with the --fileids option::
python train_tagger.py conll2000 --fileids train.txt
For a complete list of usage options::
python train_tagger.py --help

Using a Trained Tagger

You can use a trained tagger by loading the pickle file using nltk.data.load::
>>> import nltk.data
>>> tagger = nltk.data.load("taggers/NAME_OF_TAGGER.pickle")
Or if your tagger pickle file is not in a nltk_data subdirectory, you can load it with pickle.load::
>>> import pickle
>>> tagger = pickle.load(open("/path/to/NAME_OF_TAGGER.pickle"))

Either method will return an object that supports the TaggerI interface.

Once you have a tagger object, you can use it to tag sentences (or lists of words) with the tagger.tag(words) method::
>>> tagger.tag(['some', 'words', 'in', 'a', 'sentence'])

tagger.tag(words) will return a list of 2-tuples of the form [(word, tag)].

Analyzing Tagger Coverage

The analyze_tagger_coverage.py script will run a part-of-speech tagger on a corpus to determine how many times each tag is found. Example output can be found in Analyzing Tagged Corpora and NLTK Part of Speech Taggers.

Here's an example using the NLTK default tagger on the treebank corpus::
python analyze_tagger_coverage.py treebank

To get detailed metrics on each tag, you can use the --metrics option. This requires using a tagged corpus in order to compare actual tags against tags found by the tagger. See NLTK Default Tagger Treebank Tag Coverage and NLTK Default Tagger CoNLL2000 Tag Coverage for examples and statistics.

To analyze the coverage of a different tagger, use the --tagger option with a path to the pickled tagger::
python analyze_tagger_coverage.py treebank --tagger /path/to/tagger.pickle
To analyze coverage on a custom corpus, whose fileids end in ".pos", using a TaggedCorpusReader::
python analyze_tagger_coverage.py /path/to/corpus --reader nltk.corpus.reader.tagged.TaggedCorpusReader --fileids '.+\.pos'

The corpus path can be absolute, or relative to a nltk_data directory. For example, both corpora/treebank/tagged and /usr/share/nltk_data/corpora/treebank/tagged will work.

For a complete list of usage options::
python analyze_tagger_coverage.py --help

Analyzing a Tagged Corpus

The analyze_tagged_corpus.py script will show the following statistics about a tagged corpus:

  • total number of words
  • number of unique words
  • number of tags
  • the number of times each tag occurs

Example output can be found in Analyzing Tagged Corpora and NLTK Part of Speech Taggers.

To analyze the treebank corpus::
python analyze_tagged_corpus.py treebank
To sort the output by tag count from highest to lowest::
python analyze_tagged_corpus.py treebank --sort count --reverse
To see simplified tags, instead of standard tags::
python analyze_tagged_corpus.py treebank --simplify_tags
To analyze a custom corpus, whose fileids end in ".pos", using a TaggedCorpusReader::
python analyze_tagged_corpus.py /path/to/corpus --reader nltk.corpus.reader.tagged.TaggedCorpusReader --fileids '.+\.pos'

The corpus path can be absolute, or relative to a nltk_data directory. For example, both corpora/treebank/tagged and /usr/share/nltk_data/corpora/treebank/tagged will work.

For a complete list of usage options::
python analyze_tagged_corpus.py --help

Training IOB Chunkers

The train_chunker.py script can use any corpus included with NLTK that implements a chunked_sents() method.

Train the default sequential backoff tagger based chunker on the treebank_chunk corpus::
python train_chunker.py treebank_chunk
To train a NaiveBayes classifier based chunker::
python train_chunker.py treebank_chunk --classifier NaiveBayes
To train on the conll2000 corpus::
python train_chunker.py conll2000
To train on a custom corpus, whose fileids end in ".pos", using a ChunkedCorpusReader::
python train_chunker.py /path/to/corpus --reader nltk.corpus.reader.chunked.ChunkedCorpusReader --fileids '.+\.pos'

The corpus path can be absolute, or relative to a nltk_data directory. For example, both corpora/treebank/tagged and /usr/share/nltk_data/corpora/treebank/tagged will work.

You can also restrict the files used with the --fileids option::
python train_chunker.py conll2000 --fileids train.txt
For a complete list of usage options::
python train_chunker.py --help
Something went wrong with that request. Please try again.