nltk.pos_tag performance

[stevenbird]

NLTK's built-in POS tagger doesn't perform so well:

>>> text = word_tokenize("The quick brown fox jumps over the lazy dog")
>>> pos_tag(text)
[('The', 'DT'), ('quick', 'NN'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'NN'), ('dog', 'NN')]

We need a better built-in model.

cf. http://stackoverflow.com/questions/30821188/python-nltk-pos-tag-not-returning-the-correct-part-of-speech-tag

[alvations]

@stevenbird , I think the simplest to "reimplement" with top quality accuracy is actually honnibal's perception tagger.

Next, the simplest to implement with unknown accuracy is ngram taggers with backoff ngram taggers using existing tagger corpora in NLTK or redistributable model that one can build with LDC's corpora (something like Spaghetti tagger.

There's one hand-crafted tagger with unknown accuracy too in nltk.parse.malt. Other rule-based tagger to consider is from CLIPS' pattern.en. The major caveats of pattern's tagger is the dependence on lexicon: https://raw.githubusercontent.com/clips/pattern/master/pattern/text/en/en-lexicon.txt

My last suggestion is a pre-trained Brill tagger, pattern has a list of rules: https://github.com/clips/pattern/blob/master/pattern/text/en/en-context.txt

[stevenbird]

@alvations: thanks for those suggestions. It looks like Honnibal's tagger uses an MIT license. @syllog1sm is there any objection to including this tagger in NLTK?

[honnibal]

Sure, go ahead. You might want my transition-based parser too, which is also under MIT I think.

[stevenbird]

@honnibal – great, thanks very much

[stevenbird]

Another example of bad behaviour: http://stackoverflow.com/questions/32571486/nltk-red-not-recognised-as-an-adjective

[stevenbird]

Resolved by #1122