PTB Tokenizer deletes last token

[garfieldnate]

(apologies for accidentally submitting before finishing typing)

The MaxentTagger is deleting the last token in the following input:

List<List<HasWord>> sentences = MaxentTagger
            .tokenizeText(new StringReader("1. Narakattaramukku"));

The resulting sentences variable contains only [[1, .]].

[garfieldnate]

Here are some other failing examples. All of these are examples of using EOS punctuation before actual EOS:

So. California
So. Korea
Stop! Dimentica
Tune! FM
Mount. Annaguan
IMPACT! Zone

[manning]

Agree that there seems to be a bug here. But it seems like it's a bug with the tagger somehow. If you just run the tokenizer then it runs fine:

$ cat tokenizer-bug.txt 
So. California
$ java -cp "*" edu.stanford.nlp.process.PTBTokenizer tokenizer-bug.txt 
So
.
California
PTBTokenizer tokenized 3 tokens at 26.53 tokens per second.

[garfieldnate]

That might actually be easier to track down, then. The tokenizer code looked generated to me, and the tagger should simply be tagging every input token and outputting it, right?

[manning]

Indeed. I've worked out where the error is -- it's in the next level up class DocumentPreprocessor, which is widely used to group tokens into sentence. I'll check in a fix later today.

Thanks for finding this! I think this bug has existed for many years....

[garfieldnate]

Thanks. I will update and try it out after the holidays are over.