[BUG] PPL search command drops wildcards (* / ?) when value contains a space

[penghuo]

Query Information

PPL Command/Query:

source=my_index name="POST /test-logs/_search*" | fields name

Expected Result:
Documents whose keyword name field starts with POST /test-logs/_search (i.e. the * is treated as a wildcard, matching zero or more characters).

Actual Result:
Zero hits. The same value without a space works as expected:

source=my_index name="*/test-logs/_search*" | fields name   -- returns matching docs
source=my_index name="POST /test-logs/_search*" | fields name -- returns 0 docs

The only difference is the leading POST (space). The * stops working the moment the value contains whitespace.

Dataset Information

Dataset/Schema Type

• OpenTelemetry (OTEL)
• Simple Schema for Observability (SS4O)
• Open Cybersecurity Schema Framework (OCSF)
• Custom (details below)

Index Mapping

{
  "mappings": {
    "properties": {
      "name": { "type": "keyword" }
    }
  }
}

Sample Data

{"name":"foo"}
{"name":"foobar"}
{"name":"food"}
{"name":"foo bar"}
{"name":"foo barbaz"}
{"name":"POST /test-logs/_search"}
{"name":"POST /test-logs/_search/xyz"}

Bug Description

Issue Summary:
When a search-command value contains both a space and a wildcard (* or ?), the parser emits a Lucene phrase query. Inside a phrase, * and ? are treated as literal characters — wildcard semantics are silently lost.

• On a keyword field: the query returns 0 hits, even when matching documents exist. Hard failure.
• On a text field: the query returns hits but the wildcard is silently ignored — you get whatever the bare phrase matches. Silent-correctness failure.

Root cause (code pointer):

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstExpressionBuilder.java:1141

return new SearchLiteral(new Literal(content, DataType.STRING), content.contains(" "));

The isPhrase flag is set to true whenever the value contains a space. That triggers SearchLiteral.toQueryString() to wrap the value in double quotes, which becomes a Lucene phrase query — and Lucene's classic query-string parser strips wildcard interpretation inside phrases.

Steps to Reproduce:

1. Create the two indices below (one keyword, one text) and bulk-load the sample docs.
2. Run each PPL query in the matrix below.
3. Compare emitted query_string (from /_plugins/_ppl/_explain) and observed hits.

Index setup:

curl -X PUT localhost:9200/matrix_keyword -H 'Content-Type: application/json' \
  -d '{"mappings":{"properties":{"name":{"type":"keyword"}}}}'
curl -X PUT localhost:9200/matrix_text -H 'Content-Type: application/json' \
  -d '{"mappings":{"properties":{"name":{"type":"text"}}}}'

# same values loaded into both, plus text variants ending with " HTTP/1.1"
# for the last two rows in matrix_text

Measured matrix

Every row was executed end-to-end. The Emitted query_string column is extracted from /_plugins/_ppl/_explain (calcite.physical → sourceBuilder.query.query_string.query); the Hits column is from /_plugins/_ppl execution.

Corpus (7 docs per index, same values in matrix_keyword and matrix_text, except text rows for D3 append " HTTP/1.1" and row 5 is "foo barbaz here" in text): foo, foobar, food, foo bar, foo barbaz, POST /test-logs/_search, POST /test-logs/_search/xyz.

Keyword field (matrix_keyword)

#	PPL query	Emitted query_string	Hits	Verdict
A-k	name="foo"	name:foo	1	✅ expected
B1-k	name="foo*"	name:foo*	5	✅ expected
B2-k	name="foo?"	name:foo?	1	✅ expected
C-k	name="foo bar"	name:"foo bar"	1	✅ expected
D1-k	name="foo bar*"	name:"foo bar*"	0	❌ bug — corpus has foo barbaz
D2-k	name="foo b?r"	name:"foo b?r"	0	❌ bug — corpus has foo bar
D3-k	name="POST /test-logs/_search*"	name:"POST \/test\-logs\/_search*"	0	❌ bug — corpus has 2 matching docs (the reported case)

Text field (matrix_text)

#	PPL query	Emitted query_string	Hits	Verdict
A-t	name="foo"	name:foo	3	✅ expected (analyzed term match)
B1-t	name="foo*"	name:foo*	5	✅ expected
B2-t	name="foo?"	name:foo?	1	✅ expected
C-t	name="foo bar"	name:"foo bar"	1	✅ expected (positional phrase)
D1-t	name="foo bar*"	name:"foo bar*"	1	⚠️ wildcard silently ignored — * did not extend match
D2-t	name="foo b?r"	name:"foo b?r"	0	⚠️ wildcard silently ignored
D3-t	name="POST /test-logs/_search*"	name:"POST \/test\-logs\/_search*"	2	⚠️ wildcard silently ignored — matched by phrase alone

Summary: 8/14 rows correct (all of A / B / C). 6/14 rows show the bug — all of case D. Keyword returns 0 hits; text returns partial/wrong hits with no error signal.

Impact:

• Any PPL user searching a keyword field with a value that contains a space cannot use wildcards at all — they get 0 hits with no error.
• The linked PPL docs (docs/user/ppl/cmd/search.md) advertise wildcards as a supported feature for keyword fields and separately advertise quoted multi-word phrases, but never mention the two are mutually exclusive. Every wildcard example in the docs uses a whitespace-free value, so a reader following the docs would reasonably expect the space-plus-wildcard case to work.

Proposed fix

The bug is confined to the AST → query_string conversion; the Lucene layer is behaving correctly for what it is asked to do. Two focused changes:

1. AstExpressionBuilder.visitSearchLiteral — narrow the isPhrase heuristic so it does not fire when the value contains a wildcard:

   // before
   return new SearchLiteral(new Literal(content, DataType.STRING), content.contains(" "));
   
   // after
   boolean hasWildcard = content.indexOf('*') >= 0 || content.indexOf('?') >= 0;
   boolean isPhrase = content.contains(" ") && !hasWildcard;
   return new SearchLiteral(new Literal(content, DataType.STRING), isPhrase);

2. SearchLiteral.toQueryString — when NOT a phrase, escape spaces so the whole value is parsed by query_string as a single term/wildcard pattern instead of being split at whitespace into separate clauses:

   // non-phrase branch
   return QueryStringUtils.escapeLuceneSpecialCharacters(str).replace(" ", "\\ ");

After the fix, Case D on keyword emits name:POST\ \/test\-logs\/_search* — an unquoted wildcard term that Lucene builds as a PrefixQuery on the raw keyword term, and the reported query returns the expected matches. Cases A, B, C are untouched.

On text fields, Case D after the fix produces an honest per-token PrefixQuery (which structurally cannot match a cross-token pattern) instead of silently dropping the wildcard. This is a Lucene limitation, not a bug — the existing docs (docs/user/ppl/cmd/search.md around lines 113 / 123 / 463) already recommend a .keyword subfield for wildcard use on text.

Environment Information

OpenSearch Version: 3.8.0-SNAPSHOT (also reproduces on main)

Additional Details:

• Reproduced via ./gradlew opensearch-sql:run local cluster.
• plugins.calcite.enabled=true, plugins.calcite.pushdown.enabled=false at cluster level. The query_string filter is still emitted regardless of pushdown setting; extracted via _plugins/_ppl/_explain.
• Docs page: docs/user/ppl/cmd/search.md.