planner/core: fix wrong results caused by rule_project_eliminate for queries with Apply

[guo-shaoge]

What problem does this PR solve?

Issue Number: close #67138

Problem Summary:

                Apply
    |                        |
selection-1                 MaxOneRow
    |                        |
projection-2(col-9 -> col-6)  projection
    |                        |
projection-1                selection-2(using col-9, which is a correlated col reference the output of projection-1)
    |                        |
datasource-1                datasource-2(only output two columns: col-14, col-15)
(outer side).                 (inner side of Apply)

There is a replaceMap in rule_eliminate_projection: <col-9, col-6>, which means we should replace all col-9 to col-6.

But since datasource-2 only outputs col-14 and col-15, this piece of code prevents the replacement from happening.
Because it assumes that all target columns to be replaced must come from the child output schema. This is true in normal cases, but correlated columns do not actually come from the child output schema—they are instead set explicitly during the execution of Apply.
As a result, col-9 in selection-2 is not replaced with col-6, and the correlated column is not updated correctly.

                Apply
    |                        |
selection-1(output col-6)  MaxOneRow
    |                        |
    |                      projection
 datasource-1                |
                           selection-2(still using col-9, which is unexpected, so the dataum pointer will not be updated during Apply execution)
                             |
                           datasource-2

What changed and how does it work?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

planner/core: fix wrong results caused by rule_project_eliminate for queries with Apply

[pantheon-ai]

Review failed due to infrastructure/execution failure (Codex API service overloaded - 503 errors). Please re-trigger review when service is available.

_{ℹ️ Learn more details on Pantheon AI.}

[hawkingrei]

/ok-to-test

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR adjusts test sharding configuration, introduces a new test verifying CTE WITH clause behavior with Apply operations, and modifies the projection elimination logic to unconditionally apply column replacements without schema validation filtering.

Changes

Cohort / File(s)	Summary
Test Infrastructure pkg/planner/core/casetest/rule/BUILD.bazel, pkg/planner/core/casetest/rule/rule_eliminate_projection_test.go	Increased test shard count from 24 to 25, added new test TestWithApply to verify CTE WITH queries produce expected results and contain Apply operations in execution plans.
Projection Elimination Logic pkg/planner/core/rule_eliminate_projection.go	Removed schema-existence validation filtering on the replacement column map; now unconditionally applies the full replacement map via ReplaceExprColumns() instead of filtering and conditionally applying a subset.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• planner: fix window frame replacement after projection elimination #66910: Fixes ReplaceExprColumns and copy-on-write replacement logic that directly relate to how expression-column replacements are applied during projection elimination.

Suggested labels

size/L

Suggested reviewers

• hawkingrei
• qw4990
• guo-shaoge

Poem

🐰 A projection's chains once held too tight,
Schema filters blocked the rightful light,
Now replacements flow without restraint,
CTEs with ORDER BY—no more complaint!
Apply reveals what hidden plans once hid. ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: fixing incorrect query results caused by the rule_project_eliminate optimizer rule when queries involve Apply operators.
Linked Issues check	✅ Passed	The PR directly addresses issue #67138 by fixing the rule_project_eliminate logic that was causing incorrect results in CTE queries with Apply operators, matching the issue's requirement for correct behavior.
Out of Scope Changes check	✅ Passed	All changes are directly related to fixing the rule_project_eliminate bug: the test validates the fix, the BUILD.bazel adjustment supports test execution, and the core logic change implements the fix.
Description check	✅ Passed	The PR description includes a clear problem statement with code diagrams, identifies the root cause, and specifies the fix. It includes issue number, selected test type, and release notes.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

pkg/planner/core/rule_eliminate_projection.go (1)

186-186: Document why replacement is unfiltered here.

This call is critical and non-obvious (especially with Apply/correlated references). Please add a brief intent comment to prevent future regressions from “cleanup” refactors.

✍️ Suggested inline comment

-	p.ReplaceExprColumns(replace)
+	// Intentionally apply the full replacement map (no child-schema filtering).
+	// Apply/correlated plans may need replacements that are not visible in a single child schema.
+	p.ReplaceExprColumns(replace)

As per coding guidelines: "Comments SHOULD explain non-obvious intent, constraints, invariants, concurrency guarantees, SQL/compatibility contracts, or important performance trade-offs, and SHOULD NOT restate what the code already makes clear."

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@pkg/planner/core/rule_eliminate_projection.go` at line 186, Add a brief
intent comment immediately above the p.ReplaceExprColumns(replace) call
explaining why the replacement is intentionally unfiltered: note that
replacements must be applied globally here (including into Apply/correlated
references) to preserve correct binding of correlated columns and avoid breaking
Apply-based plans or leaving stale references; mention the invariant that
filtering these replacements would prevent proper propagation of column
substitutions for correlated expressions.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@pkg/planner/core/rule_eliminate_projection.go`:
- Line 186: Add a brief intent comment immediately above the
p.ReplaceExprColumns(replace) call explaining why the replacement is
intentionally unfiltered: note that replacements must be applied globally here
(including into Apply/correlated references) to preserve correct binding of
correlated columns and avoid breaking Apply-based plans or leaving stale
references; mention the invariant that filtering these replacements would
prevent proper propagation of column substitutions for correlated expressions.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: bdc7c789-f4bc-43fe-a96e-caccbb2b2b0e

📥 Commits

Reviewing files that changed from the base of the PR and between 362aeba and 1aacbcc.

📒 Files selected for processing (3)

• pkg/planner/core/casetest/rule/BUILD.bazel
• pkg/planner/core/casetest/rule/rule_eliminate_projection_test.go
• pkg/planner/core/rule_eliminate_projection.go

[guo-shaoge]

/retest

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.6780%. Comparing base (0763fed) to head (1aacbcc).
⚠️ Report is 52 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #67231        +/-   ##
================================================
- Coverage   77.6966%   77.6780%   -0.0187%     
================================================
  Files          2013       1943        -70     
  Lines        551311     556218      +4907     
================================================
+ Hits         428350     432059      +3709     
- Misses       121229     124094      +2865     
+ Partials       1732         65      -1667     

Flag	Coverage Δ
integration	41.7708% <100.0000%> (-6.3847%)	⬇️
unit	77.0266% <100.0000%> (+0.8259%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	61.5065% <ø> (+4.7091%)	⬆️
parser	∅ <ø> (∅)
br	47.2882% <ø> (-13.5762%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hawkingrei, winoros

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• pkg/planner/OWNERS [hawkingrei,winoros]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-03-23 10:40:54.2018111 +0000 UTC m=+178450.237881361: ☑️ agreed by winoros.
• 2026-03-23 13:41:03.710605334 +0000 UTC m=+189259.746675594: ☑️ agreed by hawkingrei.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-8.5: #67246.
But this PR has conflicts, please resolve them!