Convert `dedup` pushdown to composite + top_hits #4844

LantaoJin · 2025-11-21T15:31:40Z

Description

Convert dedup pushdown to composite + top_hits
Main changes:

Add a rule DedupPushdownRule to convert the dedup plan pattern to Aggregate with TopHits metrics
Upgrade TopHitsParser to support parsing data from _source (fetchField cannot work for OS Object type)
Corresponding API change on MetricParser: Map<String, Object> parse(Aggregation aggregation); -> List<Map<String, Object>> parse(Aggregation aggregation);

Follow-ups: Support dedup on script (expression)

Related Issues

Resolves #4797

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <ltjin@amazon.com>

qianheng-aws · 2025-11-24T08:08:35Z

opensearch/src/main/java/org/opensearch/sql/opensearch/request/AggregateAnalyzer.java

    return project == null
        ? List.of()
-        : aggCall.getArgList().stream().map(project.getProjects()::get).toList();
+        : PlanUtils.getObjectFromLiteralAgg(aggCall) != null


Can you elaborate more on why calling this method here?

Is it for identifying LITERAL_AGG?

Yes. LITERAL_AGG passes the numberOfDedup here

opensearch/src/main/java/org/opensearch/sql/opensearch/request/AggregateAnalyzer.java

qianheng-aws · 2025-11-24T08:14:26Z

integ-test/src/test/resources/expectedOutput/calcite/explain_dedup_complex1.yaml

+            LogicalFilter(condition=[IS NOT NULL($4)])
+              CalciteLogicalIndexScan(table=[[OpenSearch, opensearch-sql_test_index_account]])
+  physical: |
+    CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch-sql_test_index_account]], PushDownContext=[[PROJECT->[account_number, firstname, address, balance, gender, city, employer, state, age, email, lastname], FILTER->IS NOT NULL($4), AGGREGATION->rel#:LogicalAggregate.NONE.[](input=LogicalProject#,group={0},agg#0=LITERAL_AGG(1)), LIMIT->10000], OpenSearchRequestBuilder(sourceBuilder={"from":0,"size":0,"timeout":"1m","query":{"exists":{"field":"gender","boost":1.0}},"_source":{"includes":["account_number","firstname","address","balance","gender","city","employer","state","age","email","lastname"],"excludes":[]},"aggregations":{"composite_buckets":{"composite":{"size":1000,"sources":[{"gender":{"terms":{"field":"gender.keyword","missing_bucket":false,"order":"asc"}}}]},"aggregations":{"$f1":{"top_hits":{"from":0,"size":1,"version":false,"seq_no_primary_term":false,"explain":false,"_source":false,"fields":[{"field":"gender"},{"field":"account_number"},{"field":"firstname"},{"field":"address"},{"field":"balance"},{"field":"city"},{"field":"employer"},{"field":"state"},{"field":"age"},{"field":"email"},{"field":"lastname"}]}}}}}}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])


We won't have FILTER->IS NOT NULL($4) push down for common aggregate push down, after this PR: #4843

Could you check if we can remove them as well for dedup push down?

qianheng-aws · 2025-11-24T08:29:39Z

opensearch/src/main/java/org/opensearch/sql/opensearch/planner/rules/DedupPushdownRule.java

+
+    // 3. Push an Aggregate
+    final List<RexNode> newDedupColumns = RexUtil.apply(mappingForDedupColumns, dedupColumns);
+    relBuilder.aggregate(relBuilder.groupKey(newDedupColumns), relBuilder.literalAgg(dedupNumer));


LITERAL_AGG in Calcite has totally different function as we use here. It seems to be tricky to implement this in this way. Do we have alternative approach? Or at least add some comments to notice that.

I don't have other alternative approach. I think it's safe since LITERAL_AGG usually used in Project removing for Aggregate. And PPL doesn't have explicit syntax to call LITERAL_AGG. I will add comment to elaborate.

...earch/src/main/java/org/opensearch/sql/opensearch/data/value/OpenSearchExprValueFactory.java

qianheng-aws · 2025-11-24T08:48:39Z

opensearch/src/main/java/org/opensearch/sql/opensearch/request/AggregateAnalyzer.java

+        }
+        Integer dedupNumber = literal.getValueAs(Integer.class);
+        TopHitsAggregationBuilder topHitsAggregationBuilder =
+            AggregationBuilders.topHits(aggFieldName).from(0).fetchSource(false).size(dedupNumber);


Can you elaborate more on why we set fetchSource(false) here? The default value is true?

Update: change to use fetchSource instead of fetchField due to fetchField cannot work on OS object type.

qianheng-aws · 2025-11-24T09:00:37Z

opensearch/src/main/java/org/opensearch/sql/opensearch/response/agg/TopHitsParser.java

+      //   LinkedHashMap["name" -> "A", "category" -> "X"],
+      //   LinkedHashMap["name" -> "A", "category" -> "X"]
+      // ]
+      List<Map<String, Object>> res =


[question] Is this the only case that agg metric parser should return a List of value for each bucket? I'm thinking Is it still appropriate to translate dedup x to aggregate? Normally speaking, aggregate should return only 1 row value for the metric in each bucket. While dedup x can return more than 1 rows and it now affects the API of agg metric parser.

[question] Is this the only case that agg metric parser should return a List of value for each bucket?

Yeah, just top_hits metric agg does. top_hits metric agg in OpenSearch can be used in any bucket agg in DSL as a sub-aggregation, it's quite different with SQL' aggregate.

Signed-off-by: Lantao Jin <ltjin@amazon.com>

yuancu · 2025-11-26T05:16:10Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

            .aggregateCall(SqlStdOperatorTable.ROW_NUMBER)
            .over()
            .partitionBy(dedupeFields)
            .orderBy(dedupeFields)


Will dedupe work without ordering by deduped fields?

em, it should work in non-pushdown case. maybe we could remove this orderBy in window.

It looks a little strange to me because the sort keys in a window should be the same (the partition key)

Yes. it was not introduced by this pr. let's fix them in followup PR since it will change the plan a lot.

yuancu · 2025-11-26T05:32:26Z

opensearch/src/main/java/org/opensearch/sql/opensearch/request/OpenSearchRequestBuilder.java


  @ToString.Exclude private final Settings settings;

+  @ToString.Exclude private boolean topHitsAgg = false;


What is this field used for? It seems it's not referred anywhere

yes. can be deleted in current impl

yuancu · 2025-11-26T05:48:42Z

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteExplainIT.java

+  @Ignore("https://github.com/opensearch-project/sql/issues/4789")
+  public void testDedupExpr() throws IOException {
+    enabledOnlyWhenPushdownIsEnabled();
+    String expected = loadExpectedPlan("explain_dedup_expr1.yaml");
+    assertYamlEqualsIgnoreId(
+        expected,
+        explainQueryYaml(
+            "source=opensearch-sql_test_index_account | eval new_gender = lower(gender) | dedup 1"
+                + " new_gender"));


Isn't such case covered by CalcitePPLDedupIT.testDedupExpr

qianheng-aws · 2025-11-26T07:26:03Z

opensearch/src/main/java/org/opensearch/sql/opensearch/request/AggregateAnalyzer.java

                  String.format("Unsupported push-down aggregator %s", aggCall.getAggregation()));
        };
      }
+      case LITERAL_AGG -> {


Can you check if we will never produce a LITERAL_AGG in RelBuilder or by some rules in planner?

Otherwise, we may push down a real LITERAL_AGG to be tophits while it shouldn't be.

One alternative in my mind is using a self-defined class extends calcite.Aggregate. It should also be able to leverage our AggregateAnalyzer.

Can you check if we will never produce a LITERAL_AGG in RelBuilder or by some rules in planner?

Otherwise, we may push down a real LITERAL_AGG to be tophits while it shouldn't be.

Checked, so far no LITERAL_AGG can be produced because PPL doesn't support literal in aggregators.
In SQL

SELECT dept_id, COUNT(*) as emp_count, 1 as constant_val FROM employees GROUP BY dept_id;

Could be rewritten to

SELECT dept_id, COUNT(*) as emp_count, LITERAL_AGG(1) as constant_val FROM employees GROUP BY dept_id;

to reduce a Project upon Aggregate.

From

Project(dept_id, emp_count, constant_val=1) Aggregate(GROUP BY dept_id, compute COUNT(*) as emp_count) Scan(employees)

To

Aggregate(GROUP BY dept_id, compute COUNT(*) as emp_count, compute LITERAL_AGG(1) as constant_val) Scan(employees)

But stats only support pre-defined aggregators. cc @qianheng-aws

I see SubQueryRemoveRule will produce LITERAL_AGG for some(we don't have) and in subquery. But I'm not sure whether it will be triggered.

And RelBuilder's literalAgg method is public, we should avoid call that method by developers then.

https://issues.apache.org/jira/browse/CALCITE-4334

opensearch-trigger-bot · 2025-11-26T08:12:24Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4844-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5ceacb6945d6bacc36e037ce4da215e1cf031b56
# Push it to GitHub
git push --set-upstream origin backport/backport-4844-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4844-to-2.19-dev.

…4844) * Enable dedup pushdown Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix doctest Signed-off-by: Lantao Jin <ltjin@amazon.com> * refactor Signed-off-by: Lantao Jin <ltjin@amazon.com> * Disable dedup expr Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix yaml test Signed-off-by: Lantao Jin <ltjin@amazon.com> * add more comments in code Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix conflicts Signed-off-by: Lantao Jin <ltjin@amazon.com> * Address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> (cherry picked from commit 5ceacb6)

…4844) (#4878)

…4844) * Enable dedup pushdown Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix doctest Signed-off-by: Lantao Jin <ltjin@amazon.com> * refactor Signed-off-by: Lantao Jin <ltjin@amazon.com> * Disable dedup expr Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix yaml test Signed-off-by: Lantao Jin <ltjin@amazon.com> * add more comments in code Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix conflicts Signed-off-by: Lantao Jin <ltjin@amazon.com> * Address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com>

LantaoJin added 6 commits November 12, 2025 16:39

Enable dedup pushdown

8a1bf11

Signed-off-by: Lantao Jin <ltjin@amazon.com>

fix doctest

d2a197f

Signed-off-by: Lantao Jin <ltjin@amazon.com>

refactor

5567fe7

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Merge remote-tracking branch 'upstream/main' into issues/4797

d1cdf7f

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Disable dedup expr

08209f1

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Merge remote-tracking branch 'upstream/main' into issues/4797

5487bc4

Signed-off-by: Lantao Jin <ltjin@amazon.com>

LantaoJin added the enhancement New feature or request label Nov 21, 2025

LantaoJin mentioned this pull request Nov 21, 2025

Re-enable dedup pushdown #4790

Closed

8 tasks

LantaoJin added 2 commits November 22, 2025 16:49

fix IT

7e35a10

Signed-off-by: Lantao Jin <ltjin@amazon.com>

fix yaml test

8bf0431

Signed-off-by: Lantao Jin <ltjin@amazon.com>

LantaoJin marked this pull request as ready for review November 22, 2025 16:09

LantaoJin requested review from GumpacG, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, seankao-az, vamsimanohar and ykmr1224 as code owners November 22, 2025 16:09

LantaoJin requested review from RyanL1997, qianheng-aws and yuancu as code owners November 22, 2025 16:09

qianheng-aws reviewed Nov 24, 2025

View reviewed changes

LantaoJin added 4 commits November 24, 2025 18:07

add more comments in code

4ec1e7a

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Merge remote-tracking branch 'upstream/main' into issues/4797

cbfc625

fix conflicts

ad8eacb

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Address comments

2307c78

Signed-off-by: Lantao Jin <ltjin@amazon.com>

yuancu reviewed Nov 26, 2025

View reviewed changes

qianheng-aws approved these changes Nov 26, 2025

View reviewed changes

qianheng-aws reviewed Nov 26, 2025

View reviewed changes

yuancu approved these changes Nov 26, 2025

View reviewed changes

LantaoJin added the backport 2.19-dev label Nov 26, 2025

LantaoJin merged commit 5ceacb6 into opensearch-project:main Nov 26, 2025
39 checks passed

opensearch-trigger-bot bot added the backport-failed label Nov 26, 2025

LantaoJin mentioned this pull request Nov 27, 2025

[Backport 2.19-dev] Convert dedup pushdown to composite + top_hits (#4844) #4878

Merged

LantaoJin added the backport-manually Filed a PR to backport manually. label Nov 27, 2025

LantaoJin added a commit that referenced this pull request Nov 28, 2025

[Backport 2.19-dev] Convert dedup pushdown to composite + top_hits (#…

eeca6ce

…4844) (#4878)

LantaoJin mentioned this pull request Dec 10, 2025

[FEATURE] Speedup Join with option max=n by converting to TopHits Aggregation #4927

Closed

LantaoJin mentioned this pull request Dec 23, 2025

Dedup pushdown (TopHits Agg) should work with Object fields #4991

Merged

8 tasks


		@ToString.Exclude private final Settings settings;

		@ToString.Exclude private boolean topHitsAgg = false;

Convert dedup pushdown to composite + top_hits #4844

Convert dedup pushdown to composite + top_hits #4844

Uh oh!

Conversation

LantaoJin commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LantaoJin Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LantaoJin Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuancu Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Nov 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Convert `dedup` pushdown to composite + top_hits #4844

Convert `dedup` pushdown to composite + top_hits #4844

LantaoJin commented Nov 21, 2025 •

edited

Loading

LantaoJin Nov 24, 2025 •

edited

Loading

LantaoJin Nov 24, 2025 •

edited

Loading

yuancu Nov 26, 2025 •

edited

Loading