ESQL: Add support for Full Text Functions and Lucene Pushable Predicates for Lookup Join #136104

julian-elastic · 2025-10-07T16:04:17Z

• Added Full Text Functions Support in Lookup Join Expressions: Extended lookup join functionality to support full-text functions like MATCH() as part of join conditions, enabling text search capabilities within lookup operations.
• Enhanced Lucene Pushable Filters Support. Implemented support for any Lucene pushable filters that contain only fields from the right side of the lookup join, allowing for more complex filtering conditions. At least one condition relating the left and right side of the Lookup Join is still required.

Full text function

FROM left_table
| LOOKUP JOIN right_table ON left_id == right_id AND MATCH(right_field, "search_term")

OR of Lucene pushable filters on fields from the lookup index

FROM left_table
| LOOKUP JOIN right_table ON left_id == right_id AND (right_field like "*pattern" OR right_value > 1000)

OR and NOT of Lucene pushable filters on fields from the lookup index

FROM left_table
| LOOKUP JOIN right_table ON (MATCH(right_field, "search_term") AND right_value < 5000 OR NOT (right_field != "exclude1" AND right_field != "exclude2")) AND left_id == right_id

elasticsearchmachine · 2025-10-07T16:17:17Z

Hi @julian-elastic, I've created a changelog YAML for you.

elasticsearchmachine · 2025-10-07T16:17:17Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

carlosdelest

LGTM from the full text functions perspective 👍 :

Changes imply adding checks for lookup join conditions
IT tests have been updated with relevant tests

alex-spies

First round of comments: analysis

docs/changelog/136104.yaml

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/LogicalPlanBuilder.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

alex-spies

I have a question on the modeling of right-only filters, see below.

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/enrich/ExpressionQueryList.java

alex-spies

Productive code reviewed and looking mighty good!

Wrapping up the tests as well. Found some things that we need to have a look at there, but nothing that'll block this for long.

I think we'll want to add the new expression joins to our generative tests, no?

Also, this generally allows all Lucene-pushable filters to be put into the ON condition. Some expressions might not consider this correct and will require an update of, say, the validation (as for full text functions). Maybe we should consider a randomized test that generates a pushable expression that we then stuff into the join condition? (We can compare the result by just running the WHERE filter on the lookup index in a separate query.)

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/enrich/ExpressionQueryList.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/LogicalPlanBuilder.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/enrich/ExpressionQueryList.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/LogicalPlanBuilder.java

...rc/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/FullTextFunction.java

x-pack/plugin/esql/qa/testFixtures/src/main/resources/lookup-join-expression.csv-spec

alex-spies

Ok, all done now. This still looks very good. I found another oopsie re. the capabilities used which we should fix, but in total I think we can merge after the next iteration, most likely. Nice!

x-pack/plugin/esql/qa/testFixtures/src/main/resources/lookup-join-expression.csv-spec

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/VerifierTests.java

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/VerifierTests.java

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/StatementParserTests.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/enrich/ExpressionQueryList.java

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/ParsingTests.java

julian-elastic · 2025-10-22T14:07:55Z

@vadimkibana @stratoula This PR is not making any changes to the antler grammar, but it does allow for more complex expressions as part of the lookup join on condition. Do you need any changes on your side for this? Please have this in mind when working on elastic/kibana#236939

stratoula · 2025-10-22T14:27:32Z

Thanx @julian-elastic noted!

alex-spies

This is great, thanks @julian-elastic !

alex-spies · 2025-10-23T15:15:37Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/lookup-join-expression.csv-spec

+
+FROM airports
+| RENAME abbrev AS airport_code
+| LOOKUP JOIN airport_city_boundaries ON airport_code == abbrev AND ST_CONTAINS(TO_GEOSHAPE("POLYGON((-0.3 51.0, -0.1 51.0, -0.1 51.2, -0.3 51.2, -0.3 51.0))"), city_boundary)


@craigtaverner , you may want to see this :) woohoo

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

…tes for Lookup Join (elastic#136104) • Added Full Text Functions Support in Lookup Join Expressions: Extended lookup join functionality to support full-text functions like MATCH() as part of join conditions, enabling text search capabilities within lookup operations. • Enhanced Lucene Pushable Filters Support. Implemented support for any Lucene pushable filters that contain only fields from the right side of the lookup join, allowing for more complex filtering conditions. At least one condition relating the left and right side of the Lookup Join is still required. This commit may include content that was generated or assisted by GenAI tools Cursor, Gemini CLI and/or Github Copilot

Full Text Functions support for Lookup Join

2ffa01f

elasticsearchmachine added needs:triage Requires assignment of a team area label v9.3.0 labels Oct 7, 2025

julian-elastic self-assigned this Oct 7, 2025

julian-elastic added :Analytics/ES|QL AKA ESQL >enhancement labels Oct 7, 2025

elasticsearchmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) and removed needs:triage Requires assignment of a team area label labels Oct 7, 2025

Update docs/changelog/136104.yaml

b139388

julian-elastic added 7 commits October 10, 2025 10:06

Merge branch 'main' into rightFTF

f018cac

Add new UTs, fix failing UTs

3095216

Bugfix, add UTs for all FTFs

124f2fa

Fix failing UT

4d524e2

Merge branch 'main' into rightFTF

65351c7

Minor fixes

965d801

Bugfix

b467ea8

julian-elastic requested review from alex-spies and carlosdelest October 11, 2025 13:10

julian-elastic added 3 commits October 13, 2025 11:16

Merge branch 'main' into rightFTF

868cca1

Perf Improvements

a1c0989

Merge branch 'main' into rightFTF

c69f5bf

carlosdelest reviewed Oct 14, 2025

View reviewed changes

alex-spies reviewed Oct 16, 2025

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/enrich/ExpressionQueryList.java Show resolved Hide resolved

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/enrich/ExpressionQueryList.java Show resolved Hide resolved

julian-elastic changed the title ~~ESQL: Add support for Full Text Functions for Lookup Join~~ ESQL: Add support for Full Text Functions and Lucene Pushable Predicates for Lookup Join Oct 16, 2025

julian-elastic added 3 commits October 16, 2025 11:16

Address code review comments

4d4a049

Merge branch 'main' into rightFTF

49296aa

Add more UTs

895b948

julian-elastic added 3 commits October 16, 2025 13:55

Fix failing UT

6917da2

Add a new transport version

7ba0ed1

Merge branch 'main' into rightFTF

e7d3751

alex-spies reviewed Oct 17, 2025

View reviewed changes

julian-elastic added 5 commits October 20, 2025 11:55

Address code review comments

b2f44e4

Allow for Mutable Analyzer Context for testing purposes

8bca38a

Fix transport version

a9b1ceb

Merge branch 'main' into rightFTF

b8254c1

Fix failing UT

5b99f18

stratoula mentioned this pull request Oct 22, 2025

[ES|QL] Supports any boolean expression in lookup join ON operator elastic/kibana#236939

Closed

4 tasks

alex-spies approved these changes Oct 23, 2025

View reviewed changes

julian-elastic added 2 commits October 23, 2025 11:43

Merge branch 'main' into rightFTF

45dbf66

Merge branch 'main' into rightFTF

520fc04

julian-elastic merged commit d213909 into elastic:main Oct 23, 2025
34 checks passed

ESQL: Add support for Full Text Functions and Lucene Pushable Predicates for Lookup Join #136104

ESQL: Add support for Full Text Functions and Lucene Pushable Predicates for Lookup Join #136104

Uh oh!

Conversation

julian-elastic commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 7, 2025

Uh oh!

elasticsearchmachine commented Oct 7, 2025

Uh oh!

carlosdelest left a comment

Choose a reason for hiding this comment

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alex-spies left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

julian-elastic commented Oct 22, 2025

Uh oh!

stratoula commented Oct 22, 2025

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

alex-spies Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

julian-elastic commented Oct 7, 2025 •

edited

Loading

alex-spies left a comment •

edited

Loading