TestRandomChains failure [LUCENE-8092]

[asfimport]

https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.2/1/

ant test -Dtestcase=TestRandomChains -Dtests.method=testRandomChains -Dtests.seed=C006DAD2E1FC77AF -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/Users/romseygeek/projects/lucene-test-data/enwiki.random.lines.txt -Dtests.locale=tr -Dtests.timezone=Europe/Simferopol -Dtests.asserts=true -Dtests.file.encoding=UTF-8

Reproduces locally on 7.2

---

Migrated from LUCENE-8092 by Alan Woodward (@romseygeek), updated Jan 05 2022

[asfimport]

Alan Woodward (@romseygeek) (migrated from JIRA)

This is due to a combination of ShingleFilter and CJKBigramFilter, leading to offsets going backwards. Digging further.

[asfimport]

Alan Woodward (@romseygeek) (migrated from JIRA)

SF:ShingleFilter@2d06f345 term=ᇻᆑᅌ,bytes=[e1 87 bb e1 86 91 e1 85 8c],startOffset=0,endOffset=3,positionIncrement=1,positionLength=1,type=<HANGUL>,termFrequency=1
CJ:CJKBigramFilter@20d01b2b term=ᇻᆑ,bytes=[e1 87 bb e1 86 91],startOffset=0,endOffset=2,positionIncrement=1,positionLength=1,type=<DOUBLE>,termFrequency=1
CJ:CJKBigramFilter@20d01b2b term=ᆑᅌ,bytes=[e1 86 91 e1 85 8c],startOffset=1,endOffset=3,positionIncrement=1,positionLength=1,type=<DOUBLE>,termFrequency=1
SF:ShingleFilter@2d06f345 term=ᇻᆑᅌ IacUTe,bytes=[e1 87 bb e1 86 91 e1 85 8c 20 49 61 63 55 54 65],startOffset=0,endOffset=14,positionIncrement=0,positionLength=2,type=shingle,termFrequency=1

So this is because both ShingleFilter and CJKBigramFilter emit multiple tokens with adjusted offsets. The CJKBigramFilter is splitting the first unigram emitted by ShingleFilter, and emitting two tokens, the second of which has an increased offset. The ShingleFilter then emits a combination of the first two terms, with offset set back to 0.

I'm not sure of the best way to fix this. Maybe both ShingleFilter and CJKBigramFilter need to cache their inputs and check that the underlying TokenStream has moved on before they emit bigrams? @uschindler what do you think?

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

This is indeed a funny edge-case. Maybe the second filter could alternatively reorder the tokens.

But I am not really sure if this is really a problem in real-world scenarios. It may affect all tokenfilters (not only those 2) that inject tokens with a diffent offset and length. I think the above combination is useless in real-world, but maybe others make sense?

An alternative would be to reorder tokens as last step in the analysis chain (using a "fixing" token filter). It just reorders all tokens with same position so their offsets increase?

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

CJKBigramFilter isn't really prepared to handle an arbitrary input graph (or maybe even synonyms): its looking for a flat stream of tokens that may include some CJK.

It already has a ridiculously complex job, its like a shinglefilter but with crazy custom logic: but it does manage that to support the use-case across different tokenizer variants (StandardTokenizer, UAXURLTokenizer, ICUTokenizer).

Maybe it should throw a clear exception if it encounters posinc=0 or poslen > 1 ? It would at least make it totally clear that it won't work, rather than the user getting a more vague exception from indexwriter. Ideally this would be detected earlier though (in construction of the chain). Unfortunately its not so easy to simply require that its input is a tokenizer, see the CJKAnalyzer use-case where the width-filter comes before, because that impacts bigramming.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

CJKBigramFilter was already in the offsetsAreBroken list for this test, so it used to not fail in such cases. However, IndexWriter now always checks the offsets, so you'll get an exception from indexwriter regardless. Hence, this list doesn't work (if offsets are broken, the tokenfilter is broken). See #9239.

This doesn't fix the fact the filter is buggy, we know that. it just reflects changes made in #8677.