[CALCITE-7442] Getting Wrong index of Correlated variable inside Subquery after FilterJoinRule by yashlimbad · Pull Request #4840 · apache/calcite

yashlimbad · 2026-03-18T11:52:35Z

Jira Link

CALCITE-7442

Changes

Adjust offset of correlated variable inside subquery when pushing filter via FilterJoinRule.java

xiedeyantu · 2026-03-19T01:25:55Z

There is an error in the CI that needs to be resolved.

yashlimbad · 2026-03-19T03:04:57Z

Updated the Code.

sonarqubecloud · 2026-03-19T03:27:20Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
90.0% Coverage on New Code
9.3% Duplication on New Code

See analysis details on SonarQube Cloud

Copilot

Pull request overview

Fixes a decorrelation/planning correctness issue (CALCITE-7442) where a correlated variable’s field index inside a subquery can become incorrect after FilterJoinRule pushes filters.

Changes:

Add a regression test that exercises FILTER_INTO_JOIN + FILTER_SUB_QUERY_TO_CORRELATE and then decorrelates, asserting expected plans.
Extend FilterJoinRule to propagate correlation-variable sets when creating pushed-down filters (and when keeping an above-join filter).
Enhance RelOptUtil.classifyFilters shifting logic so correlated field accesses inside RexSubQuery.rel can be adjusted during filter pushdown.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
core/src/test/java/org/apache/calcite/sql2rel/RelDecorrelatorTest.java	Adds a regression test covering correlated variable index behavior through filter pushdown + decorrelation.
core/src/main/java/org/apache/calcite/rel/rules/FilterJoinRule.java	Tracks correlation variables when constructing new Filters after pushdown.
core/src/main/java/org/apache/calcite/plan/RelOptUtil.java	Extends filter-shifting to also adjust correlated field accesses inside subqueries.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

core/src/main/java/org/apache/calcite/rel/rules/FilterJoinRule.java

core/src/main/java/org/apache/calcite/plan/RelOptUtil.java

core/src/test/java/org/apache/calcite/sql2rel/RelDecorrelatorTest.java

xiedeyantu · 2026-03-19T07:54:25Z

I'm not sure if these comments will be helpful, as I'm not very familiar with this specific area. If someone more knowledgeable doesn't review this PR within the next few days, I'll give it a try myself.

yashlimbad · 2026-03-19T09:03:41Z

some comments were looking helpful so I updated code now will check by running build if it goes through and push accordingly! 😄 it's fine if someone reviews code in few days, but I feel @julianhyde would be the best to review this because I see only his commit on RelOptUtil which I updated which is called from FilterJoinRule

yashlimbad · 2026-03-19T11:31:51Z

sometimes the tests passes sometimes fails randomly with below error in CI

Execution failed for task ':arrow:test'.
> Could not resolve all dependencies for configuration ':arrow:jacocoAgent'.
   > Could not load module metadata from /home/jenkins/.gradle/caches/modules-2/metadata-2.106/descriptors/org.jacoco/org.jacoco.agent/0.8.11/26c913274550a0b2221f47a0fe2d2358/descriptor.bin

even tho gradlew build is passed
it will be good if this CI run is consistent

core/src/main/java/org/apache/calcite/plan/RelOptUtil.java

yashlimbad · 2026-03-24T04:41:51Z

Hey @xiedeyantu ,
I think no one has reviewed the PR yet.
so I request you, can you please review this PR?

xiedeyantu · 2026-03-24T05:04:21Z

Sorry, could I get back to you in a few days? I'm currently on vacation and don't have access to a computer for debugging.

yashlimbad · 2026-03-24T05:42:54Z

sure, no problem!

caicancai · 2026-03-25T02:26:04Z

@yashlimbad fix correlated variable's index inside subquery Your PR headline doesn't seem to match the Jira headline.

yashlimbad · 2026-03-25T03:57:10Z

my bad. updated @caicancai !

caicancai

Just two simple comments

caicancai · 2026-03-25T13:10:21Z

core/src/main/java/org/apache/calcite/plan/RelOptUtil.java

     * @param adjustments     the amount to adjust each field by
+     * @param offset          the amount to shift field accesses by when
+     *                        rewriting correlated subqueries
+     * @param correlateVariableChild the child relation providing the


The code comment format here is strange.

the variable name is big here, that's why it's going into it's doc. any suggestions on formatting?

Indeed, I'll take a look at other Calcite code later to see if there are any good solutions.

Perhaps changing to a shorter, more concise variable name would solve the problem. 🤔

got it, will get back on this tomorrow

using a shorter variable name to make formatting of comments nice is not a good reason.

Thanks for the reminder, I'll look into whether there's a better way to handle this tomorrow.

caicancai · 2026-03-25T13:45:07Z

core/src/main/java/org/apache/calcite/plan/RelOptUtil.java

+    @Override public RexNode visitSubQuery(RexSubQuery subQuery) {
+      boolean[] update = {false};
+      List<RexNode> clonedOperands = visitList(subQuery.operands, update);
+      if (update[0]) {


I suspect there might be an issue with the EXISTS subquery, but I'm not entirely sure. Could you add a similar test?

EXISTS fails similar to IN clause, nice catch! thanks for this. will work on it

fixed and added test

caicancai · 2026-03-25T13:45:47Z

core/src/main/java/org/apache/calcite/plan/RelOptUtil.java

+        subQuery = subQuery.clone(subQuery.getType(), clonedOperands);
+        final Set<CorrelationId> variablesSet = RelOptUtil.getVariablesUsed(subQuery.rel);
+        if (!variablesSet.isEmpty() && correlateVariableChild != null) {
+          CorrelationId id = Iterables.getOnlyElement(variablesSet);


Are you assuming there's only one correlation variable?

my bad, updating whole variable set now

caicancai · 2026-03-26T13:48:18Z

I have some questions that I might need to confirm with debugging. I will try my best to complete the review this week.

yashlimbad · 2026-03-26T13:56:52Z

Great! thank you @caicancai

yashlimbad force-pushed the correlate_subquery_fix branch 4 times, most recently from 53f3132 to fb20a82 Compare March 18, 2026 16:08

yashlimbad force-pushed the correlate_subquery_fix branch from fb20a82 to e6ebfcc Compare March 19, 2026 02:58

xiedeyantu requested a review from Copilot March 19, 2026 07:10

Copilot AI reviewed Mar 19, 2026

View reviewed changes

yashlimbad force-pushed the correlate_subquery_fix branch 3 times, most recently from 2c9af44 to 815c57d Compare March 19, 2026 11:26

xiedeyantu added the request review request a review from committers/contributors label Mar 19, 2026

mihaibudiu reviewed Mar 19, 2026

View reviewed changes

core/src/main/java/org/apache/calcite/plan/RelOptUtil.java Show resolved Hide resolved

yashlimbad force-pushed the correlate_subquery_fix branch 2 times, most recently from e35441e to 55e4b1d Compare March 20, 2026 05:01

yashlimbad changed the title ~~[CALCITE-7442] fix correlated variable's index inside subquery~~ [CALCITE-7442] Getting Wrong index of Correlated variable inside Subquery after FilterJoinRule Mar 25, 2026

caicancai reviewed Mar 25, 2026

View reviewed changes

caicancai requested changes Mar 25, 2026

View reviewed changes

[CALCITE-7442] Correlated variable has wrong index inside subquery

4f4341c

yashlimbad force-pushed the correlate_subquery_fix branch from 55e4b1d to 4f4341c Compare March 26, 2026 09:02

Conversation

yashlimbad commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jira Link

Changes

Uh oh!

xiedeyantu commented Mar 19, 2026

Uh oh!

yashlimbad commented Mar 19, 2026

Uh oh!

sonarqubecloud bot commented Mar 19, 2026

Quality Gate passed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xiedeyantu commented Mar 19, 2026

Uh oh!

yashlimbad commented Mar 19, 2026

Uh oh!

yashlimbad commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

yashlimbad commented Mar 24, 2026

Uh oh!

xiedeyantu commented Mar 24, 2026

Uh oh!

yashlimbad commented Mar 24, 2026

Uh oh!

caicancai commented Mar 25, 2026

Uh oh!

yashlimbad commented Mar 25, 2026

Uh oh!

caicancai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

caicancai commented Mar 26, 2026

Uh oh!

yashlimbad commented Mar 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

yashlimbad commented Mar 18, 2026 •

edited

Loading

yashlimbad commented Mar 19, 2026 •

edited

Loading