Failure to parse query with special characters via Elasticsearch search endpoints

[esatterwhite]

Describe the bug
In a json string, the escape character also has to be escaped for it to be transmitted correctly, but this creates a parsing error.

Steps to reproduce the behavior:

Query Payload

{
  "size": 0,
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "fields": [
              "program"
            ],
            "query": "vector\\:\\:sources\\:\\:kafka*",
            "lenient": true
          }
        }
      ]
    }
  }
}

Response Payload

{
  "status": 400,
  "error": {
    "caused_by": null,
    "reason": "failed to parse query: `vector\\:\\:sources\\:\\:kafka*`",
    "stack_trace": null,
    "type": null
  }
}

Expected behavior
The query should be parsed as an escaped colon : as described in the docs for escaping special characters vector\:\:sources\:\:kafka*

[trinity-1686a]

fixed in quickwit-oss/tantivy#2416, we still have to update the tantivy used in quickwit

[esatterwhite]

fixed in quickwit-oss/tantivy#2416, we still have to update the tantivy used in quickwit

Thank you!

[trinity-1686a]

hum, actually there might be a 2nd issue, but I'm not entirely sure.
The tantivy patch fixes the error you get and #5041 returning everything, but then in both case we return nothing because we consider the entire thing as a wildcard query, which doesn't get tokenized.
I've checked what ES does, and in both case it doesn't find any matches, so it seems it doesn't tokenize wildcard queries either.

Is that the result you expect?

[esatterwhite]

No, elastic definitely returns the results matching only the query.

in elastic, we have program mapped as a keyword field w/ a lower case normalizer. And in quickwit, a text field, with a raw tokenizer + lowercase normalizer.

[trinity-1686a]

oh okay. that field uses the raw tokenizer, then not tokenizing is what should happen indeed

[esatterwhite]

The documents have a field like this

  "program": "vector::sources::kafka"