Add DSL Regex Query Support. - #22693
Conversation
PR Reviewer Guide 🔍(Review updated until commit 01ddb04)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to 01ddb04 Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit 637f6d7
Suggestions up to commit c5f0a5f
Suggestions up to commit b38c169
Suggestions up to commit 667b8eb
Suggestions up to commit 6f4c3a2
|
|
❌ Gradle check result for 6f4c3a2: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Persistent review updated to latest commit 667b8eb |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #22693 +/- ##
============================================
+ Coverage 71.48% 71.52% +0.04%
+ Complexity 77022 76955 -67
============================================
Files 6156 6139 -17
Lines 358482 358059 -423
Branches 52246 52218 -28
============================================
- Hits 256255 256102 -153
+ Misses 81810 81578 -232
+ Partials 20417 20379 -38 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Persistent review updated to latest commit b38c169 |
b38c169 to
c5f0a5f
Compare
|
Persistent review updated to latest commit c5f0a5f |
|
❌ Gradle check result for c5f0a5f: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
c5f0a5f to
637f6d7
Compare
|
Persistent review updated to latest commit 637f6d7 |
|
❌ Gradle check result for 637f6d7: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
…try wiring Signed-off-by: Kamal <askkamal@amazon.com>
Signed-off-by: Kamal <askkamal@amazon.com>
Signed-off-by: Kamal <askkamal@amazon.com>
Signed-off-by: Kamal <askkamal@amazon.com>
Signed-off-by: Kamal <askkamal@amazon.com>
Signed-off-by: Kamal <askkamal@amazon.com>
Signed-off-by: Kamal <askkamal@amazon.com>
Signed-off-by: Kamal <askkamal@amazon.com>
Signed-off-by: Kamal <askkamal@amazon.com>
…rgences Signed-off-by: Kamal <askkamal@amazon.com>
637f6d7 to
01ddb04
Compare
|
Persistent review updated to latest commit 01ddb04 |
Signed-off-by: Kamal <askkamal@amazon.com>
Description
Adds a translation-and-delegation path for DSL
regexpqueries in the analytics engine. Without this, a DSLregexpquery crashes withIllegalStateException("Unrecognized filter operator")becauseScalarFunction.fromSqlOperatorWithFallbackreturns null forUNRESOLVED_QUERY. The pattern is delegated verbatim to Lucene'sRegexpQueryBuilderso runtime semantics match vanilla_searchby construction — non-scoring filter context only.What / Why / Where
Scope: translate → serialize → delegate DSL
regexpto Lucene, filter context only.sandbox/plugins/dsl-query-executorsandbox/plugins/analytics-backend-lucenesandbox/libs/analytics-frameworkScalarFunction.REGEXP_QUERYenum constantSupported queries
All four parameter variants with verbatim pattern passthrough:
{ "query": { "regexp": { "hostname": { "value": "web-[0-9]{2}\\.prod\\.example\\.com" } } } }Lucene-only automaton syntax (
<n-m>intervals,&intersection):{ "query": { "regexp": { "sku": { "value": "A<100-999>&[A-Z].*", "flags": "INTERVAL|INTERSECTION" } } } }Multiple optional parameters:
{ "query": { "regexp": { "email": { "value": "[a-z]+@example\\.(com|org)", "case_insensitive": true, "max_determinized_states": 20000, "rewrite": "constant_score_boolean" } } } }Cross-engine bool/filter — regexp delegates to Lucene, range evaluates in DataFusion:
{ "query": { "bool": { "filter": [ { "regexp": { "path": { "value": "/api/v[0-9]+/users/.*" } } }, { "range": { "status_code": { "gte": 400, "lt": 500 } } } ] } } }Design
Verbatim delegation, not pattern translation — the pattern reaches
RegexpQueryBuilderunchanged. A translation approach would reject or mistranslate four Lucene constructs with no RE2 equivalent (&,~,<n-m>,<id>). By delegating, zero regex-dialect gaps.New
ScalarFunction.REGEXP_QUERY— the existingScalarFunction.REGEXProutes toRegexpSerializer(PPL) which wraps patterns as.*pattern.*for substring-match semantics; reusing it would un-anchor every DSL pattern. A separateREGEXP_QUERYinCategory.FULL_TEXTkeeps the two paths isolated.Emitted RexCall shape:
Optional params at operand index 2+, matching
AbstractRelevanceSerializer.optionalParamsStartIndex(). Serializer usesConversionUtils.extractRelevanceOperandsandConversionUtils.extractOptionalParams.Flags as raw int bitmask —
RegexpFlag.resolveValue("0")throws forflags=NONE(value 0), breaking a name-based round-trip. Raw int is lossless and mirrors vanilla'sflags_valuefield.Opaque delegated leaf — no Substrait function signature, no
opensearch_scalar_functions.yamlentry, no DataFusion plugin change.Execution path
SearchActionFilterreroutes_searchto the analytics engine;RegexpQueryTranslatoremits theREGEXP_QUERYRexCall.OpenSearchFilterRuleannotates the predicate;CapabilityRegistryfindsREGEXP_QUERYin Lucene'sFULL_TEXT_OPS×FULL_TEXT_TYPES— Lucene is the sole taker.RegexpQueryDslSerializerrebuilds aRegexpQueryBuilder; the leaf becomes an opaque delegated predicate.RegexpQuery, intersects the automaton against the term dictionary, and produces a per-segmentFixedBitSetthat crosses to Rust via FFM → ParquetRowSelection.Parameter matrix
value(pattern)RegexpQueryBuildercase_insensitivefalsetrue; omitted at defaultflagsALLmax_determinized_statesrewritenullboostConversionException(HTTP 500)_namenullConversionException(HTTP 500)Rejection rationale: columnar path is filter-only (scoring has no semantics);
matched_queriesis never assembled.Field-type support
keywordtextmatch_only_textwildcardconstant_keywordversionflat_objectGaps vs vanilla
RegexpQueryBuilderbyte-for-byte unchanged; all Lucene automaton constructs (&,~,<n-m>,<id>) work.search.allow_expensive_queries=falsenot honoured — Lucene backend injects a hardcoded always-true supplier; family-wide gap, not regexp-specific.boostrejected;_name/matched_queriesunavailable.wildcard,constant_keyword,version,flat_objectare unreachable rather than semantically divergent.RegexpQueryBuilder.doToQueryon the data node propagate unmapped becauseLuceneScanInstructionHandlercatches onlyIOException. Family-wide, not regexp-specific.Warningheaders raised on the data node never reach the client —HeaderWarning.addWarningattaches to the data node'sThreadContext, not the coordinating REST thread. The warning fires and is visible in the node log. Family-wide.At parity (inherited free by delegating)
index.max_regex_length— enforced byRegexpQueryBuilder.doToQuerybefore query construction.StringFieldType.regexpQuery→indexedValueForSearch.case_insensitive— passed toRegexpQueryBuilder.caseInsensitive; Lucene applies full Unicode case folding viaCaseFolding.expand(theASCII_CASE_INSENSITIVEconstant name is legacy). Behaviour is vanilla's by construction.RegexpQueryBuilder.doToQuery.failIfNotIndexed()— non-indexed field guard still applies.Test evidence
dsl-query-executoranalytics-backend-luceneNew:
RegexpQueryTranslatorTests14,RegexpQueryDslSerializerTests19, 5 golden plan fixtures, +2 PPL-coexistence pinning tests that assertScalarFunction.REGEXPstill resolves toRegexpSerializerwhileREGEXP_QUERYresolves toRegexpQueryDslSerializer(the DSL path cannot silently reroute the PPL path).DslRegexpQueryITis@AwaitsFix-parked (4 methods) becauseSearchResponseBuilder.build()returns empty hits repo-wide — not specific to this change.Manual E2E validation (engine-mode cluster, not automated): confirmed full-string anchoring (substring pattern → zero hits);
&intersection,<n-m>interval andflagssemantics; all four optional parameters provably wired via falsifiability pairs (e.g. invalidrewrite→Failed to parse rewrite_method [not_a_method]fromQueryParsers.parseRewriteMethodviaRegexpQueryBuilder.doToQuery); OR-of-mixed-backends (INTERLEAVED) correctly includes the delegated regexp leaf, proven by three distinct result fingerprints for union / range-only / regexp-only. Differential-oracle comparison against vanilla was not possible —SearchActionFilterintercepts all_searchunconditionally so a true vanilla index cannot be created with the plugin installed; semantic equivalence relies on the code-path argument (vanillaRegexpQueryBuilderconstructs the query, so semantics are vanilla's by construction).Out of scope / follow-ups
RegexpSerializer(PPL) latent bug: unparenthesised.*wrap causesa|b→(.*a)|(b.*). Pre-existing; separate follow-up.SearchResponseBuilderempty-hits fix and page-pruning for delegated predicates — both family-wide.Related Issues
Resolves #
Part of the DSL query-translator work for optimized engine mode.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.