Support group by span over time based column with Span UDF by songkant-aws · Pull Request #3421 · opensearch-project/sql

songkant-aws · 2025-03-13T11:25:22Z

Description

Support group by span over time based column with Span UDF

Related Issues

Resolves #3354

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Songkan Tang <songkant@amazon.com>

…ngine Signed-off-by: Songkan Tang <songkant@amazon.com>

qianheng-aws · 2025-03-14T02:43:19Z

core/src/main/java/org/opensearch/sql/calcite/udf/SpanFunction.java

+                date.atStartOfDay().atZone(ZoneOffset.UTC).toInstant().toEpochMilli(), interval);
+        return SqlFunctions.timestampToDate(dateEpochValue);
+      case SqlTypeName.TIME:
+        if (dateTimeUnit.getId() > 4) {


Could you explain why need this limitation?

Added comment inline. The TIME type usually means date format without year, month, etc like the field '17:59:59.99'

qianheng-aws · 2025-03-14T03:42:55Z

core/src/main/java/org/opensearch/sql/calcite/udf/SpanFunction.java

+ *       day, 1 month, 1 hour
+ * </ol>
+ */
+public class SpanFunction implements UserDefinedFunction {


Maybe leave a TODO here to do refactoring in future. Your previous idea of implementing a self-defined implementor seems to be a better approach.

Looks like it could be implemented by replacing ScalarFunction(we currently use) with ImplementableFunction, and move logic for handling different filed type to the implementor. In that way, this span function should be simpler with only handling timestamp type.

Then, we may have different functions like dateToTs, tsToDate, span(ForTs) and provide us more flexibility to reuse them or combine them.

…ngine

Signed-off-by: Songkan Tang <songkant@amazon.com>

LantaoJin · 2025-03-17T04:15:47Z

core/src/main/java/org/opensearch/sql/calcite/utils/BuiltinFunctionUtils.java

        return SqlLibraryOperators.DATEADD;
+        // UDF Functions
+      case "SPAN":
+        return TransferUserDefinedFunction(SpanFunction.class, "SPAN", ReturnTypes.ARG0);


ReturnTypes.ARG0 or ReturnTypes.ARG0_NULLABLE? What if the timestamp is null in span(timestamp, ...)?

I changed it to ReturnTypes.ARG0_NULLABLE but it will transform it to default long or int value like 0L or 0 in linq4j generated code. Still looking

Resolved the groupby nulls issue due to sql type creation conflicts in RexNodeBuilder.

LantaoJin · 2025-03-17T04:18:00Z

integ-test/src/test/java/org/opensearch/sql/calcite/standalone/CalcitePPLAggregationIT.java

        executeQuery(
            String.format(
-                "source=%s | stats avg(balance) by span(birthdate, 1 day) as age_balance",
+                "source=%s | stats avg(balance) by span(birthdate, 1 month) as age_balance",


we need some ITs for such as 15 minutes

Added 15 minutes IT.

Signed-off-by: Songkan Tang <songkant@amazon.com>

…ngine

Signed-off-by: Songkan Tang <songkant@amazon.com>

LantaoJin · 2025-03-18T10:57:32Z

PPLQueryDataAnonymizer and PPLQueryDataAnonymizerTest are required to update for span expression. @songkant-aws

LantaoJin · 2025-03-18T11:02:43Z

Flaky test

CalcitePPLStringBuiltinFunctionIT > testPosition FAILED
    java.lang.AssertionError: expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([EFDC39F9CD048074:F34E991E6AAE7EDB]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.opensearch.sql.util.MatcherUtils.verify(MatcherUtils.java:192)
        at org.opensearch.sql.util.MatcherUtils.verifyDataRows(MatcherUtils.java:160)
        at org.opensearch.sql.calcite.standalone.CalcitePPLStringBuiltinFunctionIT.testPosition(CalcitePPLStringBuiltinFunctionIT.java:161)

Let me trigger a re-run attempt.

cc @yuancu

Signed-off-by: Songkan Tang <songkant@amazon.com>

songkant-aws · 2025-03-19T06:22:01Z

PPLQueryDataAnonymizer and PPLQueryDataAnonymizerTest are required to update for span expression. @songkant-aws

Added corresponding change and more IT to cover span over different kinds of date formats.

songkant-aws · 2025-03-19T07:15:33Z

Looks like we have some flaky tests, this time there is one in build-windows-macos, which is irrelevant to this PR's change.

org.opensearch.sql.calcite.standalone.CalcitePPLStringBuiltinFunctionIT.testSubstring(CalcitePPLStringBuiltinFunctionIT.java:148)

LantaoJin · 2025-03-19T09:03:23Z

Failure due to flaky test, merged.

* Support group by span over time based column with Span UDF Signed-off-by: Songkan Tang <songkant@amazon.com> * Minor fix UT and IT after merge Signed-off-by: Songkan Tang <songkant@amazon.com> * Add custom minute span IT Signed-off-by: Songkan Tang <songkant@amazon.com> * Remove print Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix spotless style Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix groupby nulls issue Signed-off-by: Songkan Tang <songkant@amazon.com> * Remove unnecessary sql type creation in RexNodeBuilder Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix rounding for date and time type Signed-off-by: Songkan Tang <songkant@amazon.com> * Add more tests Signed-off-by: Songkan Tang <songkant@amazon.com> * Revert unrelated change in build.gradle Signed-off-by: Songkan Tang <songkant@amazon.com> --------- Signed-off-by: Songkan Tang <songkant@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>

Support group by span over time based column with Span UDF

5cd75c5

Signed-off-by: Songkan Tang <songkant@amazon.com>

songkant-aws requested review from GumpacG, LantaoJin, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, seankao-az and ykmr1224 as code owners March 13, 2025 11:25

Merge branch 'feature/calcite-engine' into songkant/feature/calcite-e…

f94632d

…ngine Signed-off-by: Songkan Tang <songkant@amazon.com>

qianheng-aws reviewed Mar 14, 2025

View reviewed changes

songkant-aws added 2 commits March 17, 2025 10:19

Merge branch 'feature/calcite-engine' into songkant/feature/calcite-e…

a4137a2

…ngine

Minor fix UT and IT after merge

e2ea5ea

Signed-off-by: Songkan Tang <songkant@amazon.com>

LantaoJin reviewed Mar 17, 2025

View reviewed changes

songkant-aws added 3 commits March 17, 2025 17:27

Add custom minute span IT

3a39f38

Signed-off-by: Songkan Tang <songkant@amazon.com>

Merge branch 'feature/calcite-engine' into songkant/feature/calcite-e…

8fe3727

…ngine

Remove print

7ae4dc8

Signed-off-by: Songkan Tang <songkant@amazon.com>

songkant-aws added 4 commits March 17, 2025 18:14

Fix spotless style

e81326b

Signed-off-by: Songkan Tang <songkant@amazon.com>

Fix groupby nulls issue

cac6023

Signed-off-by: Songkan Tang <songkant@amazon.com>

Remove unnecessary sql type creation in RexNodeBuilder

70afc6c

Signed-off-by: Songkan Tang <songkant@amazon.com>

Fix rounding for date and time type

92d3857

Signed-off-by: Songkan Tang <songkant@amazon.com>

LantaoJin added the calcite calcite migration releated label Mar 18, 2025

songkant-aws added 2 commits March 19, 2025 14:12

Add more tests

ee7d03f

Signed-off-by: Songkan Tang <songkant@amazon.com>

Revert unrelated change in build.gradle

f515620

Signed-off-by: Songkan Tang <songkant@amazon.com>

LantaoJin approved these changes Mar 19, 2025

View reviewed changes

LantaoJin merged commit b99130a into opensearch-project:feature/calcite-engine Mar 19, 2025
18 of 20 checks passed

songkant-aws deleted the songkant/feature/calcite-engine branch October 9, 2025 02:44

qianheng-aws mentioned this pull request Mar 18, 2026

Add songkant-aws as maintainer #5244

Open

Conversation

songkant-aws commented Mar 13, 2025 • edited by LantaoJin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

qianheng-aws Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

songkant-aws Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qianheng-aws Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

songkant-aws Mar 17, 2025

Choose a reason for hiding this comment

Uh oh!

LantaoJin Mar 17, 2025

Choose a reason for hiding this comment

Uh oh!

songkant-aws Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

songkant-aws Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

LantaoJin Mar 17, 2025

Choose a reason for hiding this comment

Uh oh!

songkant-aws Mar 17, 2025

Choose a reason for hiding this comment

Uh oh!

LantaoJin commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LantaoJin commented Mar 18, 2025

Uh oh!

songkant-aws commented Mar 19, 2025

Uh oh!

songkant-aws commented Mar 19, 2025

Uh oh!

Uh oh!

LantaoJin commented Mar 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

songkant-aws commented Mar 13, 2025 •

edited by LantaoJin

Loading

songkant-aws Mar 17, 2025 •

edited

Loading

qianheng-aws Mar 14, 2025 •

edited

Loading

songkant-aws Mar 17, 2025 •

edited

Loading

LantaoJin commented Mar 18, 2025 •

edited

Loading