KeywordTokenizer does not set start/end offset of the Token it produces [LUCENE-1441]

[asfimport]

I think just adding these two lines in the next(Token) method is the right fix:

       reusableToken.setStartOffset(0);
       reusableToken.setEndOffset(upto);

I don't think this is a back compat issue because the start/end offset are now meaningless since they will inherit whatever the reusable token had previously been used for.

---

Migrated from LUCENE-1441 by Michael McCandless (@mikemccand), resolved Nov 07 2008
Attachments: LUCENE-1441.patch

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Attached patch. I plan to commit shortly.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Committed revision 712232.