WhitespaceTokenizer supports Java whitespace, should also support Unicode whitespace [LUCENE-5096]

[asfimport]

The whitespace tokenizer supports only Java whitespace as defined in http://docs.oracle.com/javase/6/docs/api/java/lang/Character.html#isWhitespace(char)

A useful improvement would be to support also Unicode whitespace as defined in the Unicode property list http://www.unicode.org/Public/UCD/latest/ucd/PropList.txt

---

Migrated from LUCENE-5096 by Jörg Prante (@jprante), resolved Dec 08 2015
Environment:

all

Linked issues:

• WhitespaceTokenizer should tokenize on NBSP [LUCENE-6874] #7932

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

The whitespace tokenizer supports only Java whitespace as defined in http://docs\.oracle\.com/javase/6/docs/api/java/lang/Character\.html#isWhitespace(char)

Not exactly: it uses isWhitespace(int)

A useful improvement would be to support also Unicode whitespace as defined in the Unicode property list http://www\.unicode\.org/Public/UCD/latest/ucd/PropList\.txt

There are only 4 codepoints listed in there that are not java whitespace:
U+0085
U+00A0
U+2007
U+202F

breaking on the last 3 would violate the intentions of isWhitespace ("but is not also a non-breaking space ('\u00A0', '\u2007', '\u202F'). ")
U+0085 is not a whitespace character (despite having this strange White_Space property, its category is a control character).

i personally think we are doing the right thing...

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Hi,
Lucene is flexible enough to make this configureable. Just subclass CharTokenizer and provide your own list of "whitespace".

I had several people that wanted to use a WhitespaceTokenizer where also things like "-" are treated as whitespace, so this was the way to go: A fast approach for many tokenchars is to make it flexible is to use a java.util.BitSet, mark all chars that are "whitespace" and then query in isTokenChar(int) the bitset. Alternatively use a chain of ifs.

An alternative way (if you are on solr) is to inject a CharFilter before the tokenizer, that maps any "special" whitespace to one of the standard ones WhitespaceTokenizer detects.