Interface to very large corpora

[alexrudnick]

Suggested by Mark Liberman:

A high-performance search (whether term-based or based on other things,
like dates) would yield a result denoting smallish segments of a very large
corpus, and this result could be interpreted by NLTK to delimit areas for
further processing.

(This could use the PyLucene API to interrogate Lucene; assuming a
text-only version of the annotated corpus was already indexed by Lucene.)

Migrated from http://code.google.com/p/nltk/issues/detail?id=268

[alvations]

@alexrudnick pylucene could be replaced by whoosh. There's an easy example of whoosh usage with text corpora at https://github.com/alvations/shiva-something/blob/master/ngram_context.py that we can easily port to nltk.