QueryParsers do not support negative boosts [LUCENE-4378]

[asfimport]

Negative query boosts have been supported at the "Query" object level for a long time (resulting in negative scores for matching documents), but evidently we never updated the QueryParsers to know about this - attempting to specify a negative boost in the query string results in a parse error.

we should probably add this to the parser grammer(s)

---

Migrated from LUCENE-4378 by Chris M. Hostetter (@hossman), 1 vote, updated Dec 02 2014
Attachments: LUCENE-4378-test.patch
Linked issues:

• SOLR-3278

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

patch includes a testcase demonstrating the basic problem. example stack traces...

[junit4:junit4]    > Throwable #1: INVALID_SYNTAX_CANNOT_PARSE: Syntax Error, cannot parse foo^-3: Lexical error at line 1, column 5.  Encountered: "-" (45), after : "" 
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([49CF3A1EAABC55F0:1AED92BFB422F3DE]:0)
[junit4:junit4]    > 	at org.apache.lucene.queryparser.flexible.standard.parser.StandardSyntaxParser.parse(StandardSyntaxParser.java:74)
[junit4:junit4]    > 	at org.apache.lucene.queryparser.flexible.core.QueryParserHelper.parse(QueryParserHelper.java:250)
[junit4:junit4]    > 	at org.apache.lucene.queryparser.flexible.standard.StandardQueryParser.parse(StandardQueryParser.java:168)
[junit4:junit4]    > 	at org.apache.lucene.queryparser.flexible.standard.TestStandardQP.parse(TestStandardQP.java:51)

[junit4:junit4]    > Throwable #1: org.apache.lucene.queryparser.classic.ParseException: Cannot parse 'foo^-3': Lexical error at line 1, column 5.  Encountered: "-" (45), after : ""
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([49CF3A1EAABC55F0:1AED92BFB422F3DE]:0)
[junit4:junit4]    > 	at org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:126)
[junit4:junit4]    > 	at org.apache.lucene.queryparser.classic.TestQueryParser.getQuery(TestQueryParser.java:80)

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

I don't agree with this. Negative boosts to terms are generally undesirable, this is really esoteric and trappy.

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

I don't agree with this. Negative boosts to terms are generally undesirable, this is really esoteric and trappy.

i agree with you somewhat, but it is a disconnect between what is possible in the Query objects, and what the parses allows so i opened an issue.

in some cases it is actually desirable, and somewhat straight forward to use when programaticly combined with other Query objects, so we should consider adding it, or at least adding an option for it that doesn't involve editing the grammer – ie: perhaps allow it in the grammer(s), but then make the java code reject it in a way that people can override in subclasses?

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

As I said before, I just don't agree.

I don't think it would be good to add to the grammar: its significantly trappy and I can't think of any actual use cases,
instead lots of confusion where people really want a small boost instead of a negative one.

So I think its a good thing that its not supported and throws an exception (and documented as such in the syntax today).

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

I can't think of any actual use cases,

bq. instead lots of confusion where people really want a small boost instead of a negative one.

Hmmm.. all of my experiences are the generally the opposite – people want to penalize the scores of documents that match some query, and if they try using a "small boost" it still increases the score of the documents they are trying to penalize, so they wind up needing to do something like...

  (thing_they_want_to_match (*:* -thing_they_want_to_penalize)^100)

when what they ideally want to be able to write is..

  (thing_they_want_to_match thing_they_want_to_penalize^-100)

...but i'm not passionate enough about this to really fight for it – i just wanted to point out the disconnect.

(Wondering if there are any other opinions)

[asfimport]

Yonik Seeley (@yonik) (migrated from JIRA)

I don't think it would be good to add to the grammar: its significantly trappy and I can't think of any actual use cases,

We're back to the general problem of prohibiting things that some people don't see use cases for. We should really try to avoid this.

Can you expand on what you mean by "trappy"? I don't see anyone accidentally using a negative boost when they didn't mean to.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Right my problem with that is I think often people that want this don't really understand the scoring system?

To me it basically boils down to the old problem of scores not really being comparable.

http://find.searchhub.org/document/f00241f4af0134f8
http://find.searchhub.org/document/2e05678ca2fa9c27
http://find.searchhub.org/document/1461c1221f4c55b0#19b755925b57f641

I'm not going to exclude the possibility that these days you could write a similarity where something like this
could actually make sense, but I don't know what it would be :)

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

We're back to the general problem of prohibiting things that some people don't see use cases for. We should really try to avoid this.

I don't look at it that way. I want to make lucene easy to use to get good search results.

Dismissing my opinion isn't productive.

[asfimport]

Yonik Seeley (@yonik) (migrated from JIRA)

Dismissing my opinion isn't productive.

If you don't like negative boosts, then don't use them.
You're again trying to enforce your opinion on everyone else. Allowing negative boosts gives people a choice.

In my opinion, we should add negative boosts to the grammar and give people the choice. I've seen plenty of people ask for them, and I don't think they were all misguided. If they are normally better avoided, that can be handled with documentation.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

I'm not forcing anything: you can write your own queryparser to do this, the queries won't fail or anything.

I'm saying I disagree with them being in the general purpose QP that we ship as "QueryParser".

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

Dismissing my opinion isn't productive.

I don't see any one dismissing your opinion, i see yonik asking you to explain what you mean by trappy since you seem to object to a feature on principle, when i've already listed an example of a specific usecase where the people have asked for this to feature and functionally it does work at the Query/Searcher/Explaination level – it's only at the QueryParser level that you can't use it.

Right my problem with that is I think often people that want this don't really understand the scoring system?

there are lots of things about the query parser that may not make sense to you if you don't understand the scoring system – that doesn't mean we don't support them, it's up to the user to understand them if they want to use them.

As i said: i'm perfectly fine with the idea that something like this should not be supported by default, and you should have to go out of your way to enable it in the QueryParser – but making people edit the grammer seems like forcing them to go too far out of their way.

[asfimport]

Yonik Seeley (@yonik) (migrated from JIRA)

I'm saying I disagree with them being in the general purpose QP that we ship as "QueryParser".

What about those of us who want it? Will you veto a patch that implements this optional functionality because you believe that those who want it are misguided?

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

I just said I disagreed, nothing more. I don't think its a good idea for a general-purpose queryparser.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

In my opinion, a better approach is something like the BoostingQuery in lucene/queries for this demoting of terms.

As far as the implementation, I don't particularly like that it goes thru coord(), but we could improve that.

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

In my opinion, a better approach is something like the BoostingQuery in lucene/queries for this demoting of terms.

My confusion is that your argument seems like it confluates the concept/syntax with an implementation – you're saying you don't like a particular implementation, therefore you don't think QueryParser should allow for the concept.

I'm suggesting we let the syntax in the grammer allow for the concept of negative boosts, and then let the implementations of QueryParser decide what it means – the default implementation could be "error". Some users could choose to have a subclass with "simple" implementation (literally using the negative boost on the query) if it makes sense for them, and other users could do other things.

For example: suppose i wanted to use BoostingQuery like you suggest, but i want to be able to use it in some way with the QueryParser syntax. If the grammer allowed for "foo bar^-100" to do something then by subclassing the QueryParser i could impelement demoting items using this syntax with something like...

public MyParser extends QueryParser {
  `@Override`
  public BooleanQuery getBooleanQuery(List<BooleanClause> clauses,
                               boolean disableCoord) throws ParseException {
     BooleanQuery q = super.getBooleanQuery(clauses, disableCoord);
     Query result = q;
     for (clause : q.getClauses(){ 
       if (clause.getBoose() < 0) {
         q.removeClause(clause);
         penalty = -1/clause.getBoost();
         clause.setBoost(1);
         result = new BoostingQuery(result, clause, penalty)
     }
     return result;
  }
  `@Override`
  public void checkBoost(Query q, float boost) {
    // override super which will throw an exception if boost is negative
    // :NOOP:
  }
}