Make WordDelimiterGraphFilter a Tokenizer [LUCENE-8516]

[asfimport]

Being able to split tokens up at arbitrary points in a filter chain, in effect adding a second round of tokenization, can cause any number of problems when trying to keep tokenstreams to contract. The most common offender here is the WordDelimiterGraphFilter, which can produce broken offsets in a wide range of situations.

We should make WDGF a Tokenizer in its own right, which should preserve all the functionality we need, but make reasoning about the resulting tokenstream much simpler.

---

Migrated from LUCENE-8516 by Alan Woodward (@romseygeek), updated Oct 04 2018
Attachments: LUCENE-8516.patch

[asfimport]

Alan Woodward (@romseygeek) (migrated from JIRA)

Here's a first stab at a patch, which largely just copies existing WDGF functionality. WordDelimiterTokenizer takes a root tokenizer (so you could base it on standard, keyword or whitespace and still get the extra level of tokenization you need) and then applies its extra tokenization on top.

• I've removed the 'english possessive' option as we have an existing filter that will do that
• I've kept configuration flags, but this may be an opportunity to make the API easier to use - for example, we could make WordDelimiterIterator an abstract class with an overridable isBreak(int previous, int current) method

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

WordDelimiterTokenizer takes a root tokenizer (so you could base it on standard, keyword or whitespace and still get the extra level of tokenization you need) and then applies its extra tokenization on top.

This seems unnecessary. Its already over-configurable as far as how to break itself.

[asfimport]

Alan Woodward (@romseygeek) (migrated from JIRA)

It's needed at the moment for the concatenation parameters, in that if you're stringing terms back together again then you need to know where to stop. Then again, that's an argument to get rid of concatenization.

IME I've seen WDGF used for two purposes: searching for hyphenated or apostrophised words, and searching for IDs or manufacturing part numbers. Concentrating on the second, we could make this tokenizer something like CharTokenizer only instead of only breaking on specific characters, you can also break on transitions. For the first, a simple filter that indexes all subparts of a word without changing offsets (more like a synonym filter) might be the way forward?

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

It just doesnt seem like it will really improve anything if it takes that tokenizer parameter. Depending on what the tokenizer does, it may still be buggy just like before. still a tokenfilter in disguise. That is clear from the api being totally wrong (tokenizer taking tokenizer as input).

[asfimport]

Alan Woodward (@romseygeek) (migrated from JIRA)

Another solution would be for WordDelimiterGraphFilter to no longer amend offsets. So all token parts would be stored with the offsets of the original undelimited token.

[asfimport]

Alan Woodward (@romseygeek) (migrated from JIRA)

Comment from @msokolov:

My current usage of this filter requires it to be a filter, since I need to precede it with other filters. I think the idea of not touching offsets preserves more flexibility, and since the offsets are already unreliable, we wouldn't be losing much.

[asfimport]

Michael Sokolov (@msokolov) (migrated from JIRA)

Thanks for copy/paste, @romseygeek, I meant to reply-all, but fumble fingered on the phone. If we can get TokenFilters to stop messing with offsets, then maybe we can kill OffsetAttribute altogether. I feel like that is the vision we are groping towards? Either that or support Offsets in a first-class way in TokenFilters, but nobody seems to want to do that, Do we agree those are the choices? In the end, highlighting the entire original token (even when your query only really matches a piece of it) doesn't seem so terrible. I would advocate for fixing the API problems first by tightening the API around offsets, and then later if we want to make it possible to do more precise offsets / multiple passes of token splitting, we can maybe find a way to do that, but the "highlight a subtoken doesn't work" seems like a relatively minor problem, not really deserving of major efforts to support it.

[asfimport]

David Smiley (@dsmiley) (migrated from JIRA)

or support Offsets in a first-class way in TokenFilters, but nobody seems to want to do that,

Can you elaborate? This rings a bell but I forget.

Honestly I don't see the big deal in a token filter doing tokenization. I see it has certain challenges but don't think it's fundamentally wrong.

[asfimport]

Michael Sokolov (@msokolov) (migrated from JIRA)

Can you elaborate? This rings a bell but I forget.

#9496 has the discussion. The basic idea there was to add some methods to TokenStream that would be analogous to CharFilter.correctOffset so TokenFilters could also apply offsets correctly.