QueryParser should now allow leading '?' wildcards [LUCENE-2262]

[asfimport]

QueryParser currently throws an exception if a wildcard term begins with the '?' operator.

The current documentation describes why this is:

When set, * or ? are allowed as the first character of a PrefixQuery and WildcardQuery.
Note that this can produce very slow queries on big indexes. 

In the flexible indexing branch, wildcard queries with leading '?' operator are no longer slow on big indexes (they do not enumerate terms in linear fashion).
Thus, it no longer makes sense to throw a ParseException for a leading '?'

So, users should be able to perform a query of "?foo" and no longer get a ParseException from the QueryParser.

For the flexible indexing branch, wildcard queries of 'foo?', '?foo', 'f?oo', etc are all the same from a performance perspective.

---

Migrated from LUCENE-2262 by Robert Muir (@rmuir), updated May 09 2016
Attachments: LUCENE-2262_backwards.patch, LUCENE-2262.patch

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

initial patch:

• doesnt throw the exception for leading ? when matchVersion >= 3.1
• contrib's queryparser doesnt use Version, so i simply change AllowLeadingWildcardProcessor
• the backwards tests will fail because Tests using Version.LUCENE_CURRENT will produce problems in backwards branch, when development for 3.2 starts [LUCENE-2248] #3324: TestQueryParser uses LUCENE_CURRENT when it should really be using some constant that resolves to LUCENE_30: we can fix the test to use LUCENE_30 instead of LUCENE_CURRENT

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

patch for the backwards tests (not the best I admit, but uwe has fixed this in general for the future)

[asfimport]

Yonik Seeley (@yonik) (migrated from JIRA)

It doesn't seem like this type of bug is not one that people would rely on (throwing an exception for ? but not for *), hence we should just fix and not make it conditional on matchVersion?

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

It doesn't seem like this type of bug is not one that people would rely on (throwing an exception for ? but not for *

yonik, that's not what this issue is about. have you looked at the patch?

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

btw Yonik, in my opinion disallowing these queries with leading wildcards, be it * or ? or whatever, is rather silly, since we allow even slower fuzzyqueries by default.
so another option would be to do away with this 'setAllowLeadingWildcard' stuff alltogether, and if you want to disallow something, extend QueryParser yourself and throw ParseException yourself. I would actually prefer this.

i do think we should provide back compat to duplicate the previous behavior, its perfectly reasonable that someone would have special handling for these leading wildcards (e.g. Solr does).

but this issue is simply about the fact that in flex branch, leading ? is now fast, so there is no excuse to disallow it anymore, hence we should relax the queryparser to only throw a fit on leading *

[asfimport]

Yonik Seeley (@yonik) (migrated from JIRA)

yonik, that's not what this issue is about. have you looked at the patch?

Nope, I was going by the comments. Looks like I misinterpreted them. Sorry.

As an aside: In solr-land I encourage adding API examples, etc, to lower the bar for giving feedback (i.e. they don't have to read the patch to give an opinion on what the syntax of a new request should look like). That's not a criticism of your description (which was fine) but rather the implication that I should have read the patch (and I may have misread that too).

in my opinion disallowing these queries with leading wildcards, be it * or ? or whatever, is rather silly, since we allow even slower fuzzyqueries by default.

Agree.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

update description to be less terse

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

in my opinion disallowing these queries with leading wildcards, be it * or ? or whatever, is rather silly, since we allow even slower fuzzyqueries by default.

Agree.

What do you think, should we skip this step then and simply deprecate the entire setAllowLeadingWildcard concept all together, setting it to true for Version >= 3.1?

The only concern I have is that doing this might mislead someone into thinking leading * is no longer slow, which it still is (its faster than 3.0 but the computational complexity is still the same, unlike ?). Here is a comparison of 3.0 and trunk, on 10M terms, from the #2680 benchmark:

Pattern	Avg query time (ms) Lucene 3.0	Avg query time (ms) Flex
?NNNNNN	2408.5	28.8
*NNNNNN	3258.3	1048.0

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Bulk move 4.4 issues to 4.5 and 5.0

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Move issue to Lucene 4.9.