simple_query_string - operator not working with default_operator OR

[ksaritek]

I tried to negates a keyword at query but it already comes in result

simple query is
"query": {
"simple_query_string": {
"query": ""This repository" -removed",
"fields": [
"content",
"headline"
]
}
},

[imotov]

Could you provide a complete reproduction that would help us to reproduce the issue? See http://www.elasticsearch.org/help/ for an example.

[clintongormley]

Recreation:

PUT /t/t/1
{
  "content": "This repository has been removed"
}


GET /t/_search
{
  "query": {
    "simple_query_string": {
      "query": "+this -removed",
      "fields": [
        "content",
        "headline"
      ]
    }
  }
}

The negated term is ignored, because with default_operator: "OR", it is optional.

[dakrone]

@clintongormley not sure how this is a bug though? This is like saying "contains the term this OR does not contain removed", which the document does (it contains the term "this")

[clintongormley]

I think that a user would expect foo bar -baz (with default operator OR) to match as if they had written: +(foo bar) -baz. Otherwise, a negated clause with default operator OR is meaningless.

[dakrone]

@clintongormley I'm still not sure that this is actually a bug, the negated clause with the default OR operation is not meaningless (it has meaning, it just means" anything that does not contain this"). I think what is desired is setting default_operator to AND, which is a user-changeable setting?

[ppf2]

@dakrone @clintongormley Here is another use case from the field. Is this a bug or not currently supported (a feature)? Don't think default_operator:AND will help here since the use case is to return both documents in the example. thx

[clintongormley]

@ppf2 Actually, your example works as expected if you use the simple_query_string (you used the query_stringquery instead). As a query string query, you would need to change:

"(name:Android) OR NOT (status:approved)"

to:

"(name:Android) OR (NOT status:approved)"

The latter, when run through validate-query, shows the following:

"name:android (-status:approved +*:*)"

[ppf2]

Ah got it, thx @clintongormley for the tip!

[jdconrad]

Taking a look.

[jdconrad]

This is working as expected as @dakrone has inferred. The SimpleQueryParser should be thought of as only using AND and OR as operators. There is no concept of SHOULD and MUST other than internally to create the AND and OR queries. So when doing the query "+this -removed" the AND (+) is actually ignored as it is not thought of as a MUST. Using SimpleQueryParser this will always be the case where the query ends up being documents that either have 'this' OR not 'removed' ... Also note, that while this will return all the documents, the not 'removed' still affects scoring so it's not meaningless. Going to leave this open for now for further discussion if necessary.

[clintongormley]

While it may be working as designed, I'd argue that the syntax is surprising to most users. For example:

POST t/t/_bulk
{"index":{"_id":1}}
{"foo":"one"}
{"index":{"_id":2}}
{"foo":"two"}
{"index":{"_id":3}}
{"foo":"one two"}
{"index":{"_id":4}}
{"foo":"three"}

I would expect the following:

• "one": Return docs 1 & 3 (works)
• "-two": Return docs 1 & 3 (works)
• "one -two": Return doc 1 (returns 1, 3, & 4)
• "one three -two": Return docs 1 & 4 (returns 1, 3 & 4)

To get what I want (ie "Give me docs with one or three, but exclude anything with two") I need to write it as "one three +-two". That is not at all intuitive. If I typed "windows -microsoft" into google, I wouldn't expect Google to return all of the documents on the internet which don't contain the word microsoft.

At the very least it should be well documented but, given that this query is intended to be exposed to users who will not read documentation, I would say that the syntax could be improved.

[imotov]

If I typed "windows -microsoft" into google, I wouldn't expect Google to return all of the documents on the internet which don't contain the word microsoft.

@clintongormley that's because google (as well as most other search engines in 21st century) is using AND instead of OR as a default operator, which should be the default behavior for elasticsearch as well IMO. Having OR as a default operator is causing all sort of confusion for many new users.

[rmuir]

Not really. if you query https://www.google.com/?gws_rd=ssl#q=elasticsearch+reference+query+dsl+oracle it gladly returns high ranking hits and just tells you: Missing: oracle

Switching to AND breaks many analysis chains such as n-grams. With a good ranking algo its also not necessary, its just that DefaultSimilarity is really weak here.

[jdconrad]

I agree that this syntax is ugly -- "one three +-two" ; however, I am reluctant to special case the not operator because right now you have one OR three OR NOT two which while may be unexpected is predictable, but if I change this it becomes one OR three AND NOT two which is no longer predictable because it ignores the default operator and it loses its consistency. It is also very difficult to predict proper sub queries outside of this simple case. Take for example "one -three two" -- is this one AND not three OR two? Do I need to reorder this? I think this would end up being more confusing because of the way operator precedence works in that it's always first come first serve.

[imotov]

What google does, is some weird "fuzzy" AND (or something like should with large minimum should match) search that google turns on a long tail queries with a large number of terms. But the basic behavior with 2-3 term queries resembles AND more than OR, would you agree?

n-grams is an advanced feature, I think if a user can figure out how to enable n-gram (or configure any other custom analysis chain) they should be able to figure out how to switch from AND to OR in the query.

Anyway, I shouldn't hijack this discussion. I apologize for that. Back to the original topic. I think that my expectation would be that foo bar +baz -qux should be translated into something like this:

{
   "bool": {
        "should" : [
            {
                "term" : { "_all" : "foo" }
            }, {
                "term" : { "_all" : "bar" }
            }
        ],
        "must" : {
            "term" : { "_all" : "baz" }
        },
        "must_not" : {
            "term" : { "_all" : "qux" }
        }
   }
}

[jdconrad]

I should explain further what happens right now, each time an AND is switched to an OR or vice versa a new boolean query branch is created. So if you have a b c +d +e f the tree ends up looking like

bq( should bq( should bq( should a should b should c ) must d must e ) should f)

so changing the not operator to always use must will have an inconsistent change in boolean query branches since operator precedence is always left to right.

We could change it to be something like @imotov suggests (maybe this should be a different parser altogether in Lucene?), but then you have should, must, and must not... if you're truly a basic user I think and/or is easier to understand than should/must/must not.

[imotov]

I should explain further what happens right now, each time an AND is switched to an OR or vice versa a new boolean query branch is created.

Yes, and this is where it breaks my expectation. To me order of elements in the query shouldn't make any difference because "+" and "-" feel like unary operators but they behave in strange ways.

[jdconrad]

@imotov What you're saying makes sense to me from the point of view of someone that regularly deals with search, but for someone less technical I think and/or make more sense. Honestly, the default to OR is a bit odd to me too because if someone, say my mother, types "dog food" into the google she expects it to be anded together there at least through decent scoring (as you and @rmuir mentioned earlier). I think making a new parser with the behavior of must/should/must not makes sense depending on what our target audience wants. SimpleQueryParser2 or something.

[jdconrad]

All right after a bit more thought and discussion, I've come to agree with everyone in this issue that this behavior is unexpected for everyone. I'll work on making a Lucene patch for the SimpleQueryParser using the behavior describe by @imotov and @rmuir where the structure will be a single bq per subquery.

[clintongormley]

@jdconrad did anything ever come of this? Did you open any issue in Lucene that we can track?

[jdconrad]

@clintongormley Sorry, I must've gotten distracted by other issues before I had anytime to address this. I'll have to take a bit of time to remember what we had discussed.

[clintongormley]

Let's document and close

[dakrone]

Okay, opened a PR to document this, and then it can be closed.

[peterdm]

Just stumbled on this limitation myself. I'd like to echo @imotov 's suggestion from Oct 15, 2017. (I'll paste below).

(quote) I think that my expectation would be that foo bar +baz -qux should be translated into something like this:

{
   "bool": {
        "should" : [
            {
                "term" : { "_all" : "foo" }
            }, {
                "term" : { "_all" : "bar" }
            }
        ],
        "must" : {
            "term" : { "_all" : "baz" }
        },
        "must_not" : {
            "term" : { "_all" : "qux" }
        }
   }
}

The usecase is based on the previously-mentioned 'google expectation' (or really any major search engine at this point) that can be approximated with a default_operator: "OR" accompanied by a high minimum_should_match.

The appeal of simple_query_string is in the ability to meet the syntactical needs of users who are familiar with Google operators (without throwing parse exceptions).

I'm a little bummed that I have to roll my own parser to offer a commonly-accepted negation operator. It's not the end of the world, but adds friction to anyone looking for an otherwise extremely nice (almost turnkey) drop-in query which largely meets common syntax expectations.

@clintongormley, where is the right forum to re-open this? It looks like this was closed with a doc-comment b/c it belongs in Lucene's JIRA.