Add where clause support in create statement #2366

dai-chen · 2023-10-25T18:16:28Z

Description

Uploaded latest Flint grammar with support for WHERE clause in create statement
Fixed MV OpenSearch index name bug following Flint naming convention https://github.com/opensearch-project/opensearch-spark/blob/main/docs/index.md#opensearch
a. Without fix: flint_catalog.db.mv
b. After fix: flint_catalog_db_mv

Issues Resolved

N/A

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Chen Dai <daichen@amazon.com>

codecov · 2023-10-25T18:40:39Z

Codecov Report

Merging #2366 (4ad4410) into main (a5512f5) will decrease coverage by 0.01%.
The diff coverage is 91.66%.

@@             Coverage Diff              @@
##               main    #2366      +/-   ##
============================================
- Coverage     96.47%   96.46%   -0.01%     
- Complexity     4915     4918       +3     
============================================
  Files           465      465              
  Lines         13519    13522       +3     
  Branches        910      913       +3     
============================================
+ Hits          13042    13044       +2     
  Misses          458      458              
- Partials         19       20       +1

Flag	Coverage Δ
sql-engine	`96.46% <91.66%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
.../sql/spark/dispatcher/model/IndexQueryDetails.java	`96.66% <100.00%> (-0.48%)`	⬇️
...park/dispatcher/model/FullyQualifiedTableName.java	`90.47% <87.50%> (-1.84%)`	⬇️

Signed-off-by: Chen Dai <daichen@amazon.com>

spark/src/main/java/org/opensearch/sql/spark/dispatcher/model/FullyQualifiedTableName.java

vamsi-amazon · 2023-10-25T23:02:18Z

spark/src/main/java/org/opensearch/sql/spark/dispatcher/model/FullyQualifiedTableName.java

@@ -5,6 +5,9 @@

 package org.opensearch.sql.spark.dispatcher.model;

+import static org.apache.commons.lang3.StringUtils.strip;
+import static org.opensearch.sql.spark.dispatcher.model.IndexQueryDetails.STRIP_CHARS;


Nit: Should we bring this constant to this File.

sure, will merge this first. CI took so long every commit ...

vamsi-amazon · 2023-10-25T23:05:02Z

spark/src/main/java/org/opensearch/sql/spark/dispatcher/model/IndexQueryDetails.java

        break;
      case MATERIALIZED_VIEW:
-        indexName = "flint" + "_" + StringUtils.strip(getMvName(), STRIP_CHARS).toLowerCase();
+        indexName = "flint_" + new FullyQualifiedTableName(mvName).toFlintName();


Let me know if this is the edge case still pending.

CREATE MATERIALIZED VIEW default.mv
Do we allow this without datasource name?

I think this is impossible from frontend. The reason I add check in toFlintName() is I found many UT pass such name and failed after my changes. So I make toFlintName() flexible.

But for API call, user can put any name. May need to validate this.

It can come from workbench but probably we can take that as backlog.

* Upload latest grammar with where clause support in create statement Signed-off-by: Chen Dai <daichen@amazon.com> * Fix MV index name bug Signed-off-by: Chen Dai <daichen@amazon.com> * Add UT for create index statement with where clause Signed-off-by: Chen Dai <daichen@amazon.com> * Fix spotless format Signed-off-by: Chen Dai <daichen@amazon.com> --------- Signed-off-by: Chen Dai <daichen@amazon.com> (cherry picked from commit bb82d85) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Upload latest grammar with where clause support in create statement * Fix MV index name bug * Add UT for create index statement with where clause * Fix spotless format --------- (cherry picked from commit bb82d85) Signed-off-by: Chen Dai <daichen@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ect#2366) (opensearch-project#2370)" This reverts commit b620a56. Signed-off-by: Eric <menwe@amazon.com>

…ect#2366) (opensearch-project#2370)" This reverts commit b620a56.

* Revert "Add more metrics and handle emr exception message (#2422) (#2426)" This reverts commit b57f7cc. * Revert "Block settings in sql query settings API and add more unit tests (#2407) (#2412)" This reverts commit 3024737. * Revert "Added session, statement, emrjob metrics to sql stats api (#2398) (#2400)" This reverts commit 6e17ae6. * Revert "Redefine Drop Index as logical delete (#2386) (#2397)" This reverts commit e939bb6. * Revert "add concurrent limit on datasource and sessions (#2390) (#2395)" This reverts commit deb3ccf. * Revert "Add Flint Index Purging Logic (#2372) (#2389)" This reverts commit dd48b9b. * Revert "Refactoring for tags usage in test files and also added explicit denly list setting. (#2383) (#2385)" This reverts commit 37e010f. * Revert "Enable session by default (#2373) (#2375)" This reverts commit 7d95e4c. * Revert "Create new session if client provided session is invalid (#2368) (#2371)" This reverts commit 5ab7858. * Revert "Add where clause support in create statement (#2366) (#2370)" This reverts commit b620a56. * Revert "create new session if current session not ready (#2363) (#2365)" This reverts commit 5d07281. * Revert "Handle Describe,Refresh and Show Queries Properly (#2357) (#2362)" This reverts commit 16e2f30. * Revert "Add Session limitation (#2354) (#2359)" This reverts commit 0f334f8. * Revert "Bug Fix, support cancel query in running state (#2351) (#2353)" This reverts commit 9a40591. * Revert "Fix bug, using basic instead of basicauth (#2342) (#2355)" This reverts commit e4827a5. * Revert "Add missing tags and MV support (#2336) (#2346)" This reverts commit 8791bb0. * Revert "[Backport 2.x] deprecated job-metadata-index (#2340) (#2343)" This reverts commit bea432c. * Revert "Integration with REPL Spark job (#2327) (#2338)" This reverts commit 58a5ae5. * Revert "Implement patch API for datasources (#2273) (#2329)" This reverts commit 4c151fe. * Revert "Add sessionId parameters for create async query API (#2312) (#2324)" This reverts commit 3d1a376. * Revert "Add Statement (#2294) (#2318) (#2319)" This reverts commit b3c2e94. * Revert "Upgrade json (#2307) (#2314)" This reverts commit 6c65bb4. * Revert "Minor Refactoring (#2308) (#2317)" This reverts commit 051cc4f. * Revert "add InteractiveSession and SessionManager (#2290) (#2293) (#2315)" This reverts commit 6ac197b. --------- Co-authored-by: Vamsi Manohar <reddyvam@amazon.com>

Upload latest grammar with where clause support in create statement

794b174

Signed-off-by: Chen Dai <daichen@amazon.com>

dai-chen added 3 commits October 25, 2023 15:12

Fix MV index name bug

8838013

Signed-off-by: Chen Dai <daichen@amazon.com>

Add UT for create index statement with where clause

7db15a0

Signed-off-by: Chen Dai <daichen@amazon.com>

Fix spotless format

4ad4410

Signed-off-by: Chen Dai <daichen@amazon.com>

dai-chen marked this pull request as ready for review October 25, 2023 22:52

dai-chen requested review from pjfitzgibbons, ps48, kavithacm, derek-ho, joshuali925, YANG-DB, rupal-bq, mengweieric, vamsi-amazon, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto and GumpacG as code owners October 25, 2023 22:52

Swiddis approved these changes Oct 25, 2023

View reviewed changes

vamsi-amazon reviewed Oct 25, 2023

View reviewed changes

spark/src/main/java/org/opensearch/sql/spark/dispatcher/model/FullyQualifiedTableName.java Show resolved Hide resolved

vamsi-amazon reviewed Oct 25, 2023

View reviewed changes

penghuo approved these changes Oct 25, 2023

View reviewed changes

vamsi-amazon approved these changes Oct 25, 2023

View reviewed changes

dai-chen merged commit bb82d85 into opensearch-project:main Oct 25, 2023
19 of 21 checks passed

dai-chen deleted the add-where-clause-syntax-in-create-statement branch October 25, 2023 23:20

vamsi-amazon added backport 2.x backport 2.11 labels Oct 25, 2023

opensearch-trigger-bot bot mentioned this pull request Oct 25, 2023

[Backport 2.x] Add where clause support in create statement #2369

Merged

opensearch-trigger-bot bot mentioned this pull request Oct 25, 2023

[Backport 2.11] Add where clause support in create statement #2370

Merged

dai-chen self-assigned this Nov 4, 2023

dai-chen mentioned this pull request Nov 4, 2023

Fix Flint index name and MV name bug opensearch-project/opensearch-spark#127

Merged

mengweieric added a commit to mengweieric/sql that referenced this pull request Nov 8, 2023

Revert "Add where clause support in create statement (opensearch-proj…

601b2ad

…ect#2366) (opensearch-project#2370)" This reverts commit b620a56. Signed-off-by: Eric <menwe@amazon.com>

mengweieric added a commit to mengweieric/sql that referenced this pull request Nov 8, 2023

Revert "Add where clause support in create statement (opensearch-proj…

469fab7

…ect#2366) (opensearch-project#2370)" This reverts commit b620a56. Signed-off-by: Eric <menwe@amazon.com>

vamsi-amazon added a commit to mengweieric/sql that referenced this pull request Nov 13, 2023

Revert "Add where clause support in create statement (opensearch-proj…

d049134

…ect#2366) (opensearch-project#2370)" This reverts commit b620a56.

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add where clause support in create statement #2366

Add where clause support in create statement #2366

dai-chen commented Oct 25, 2023 •

edited

Loading

codecov bot commented Oct 25, 2023 •

edited

Loading

vamsi-amazon Oct 25, 2023

dai-chen Oct 25, 2023

vamsi-amazon Oct 25, 2023

dai-chen Oct 25, 2023 •

edited

Loading

vamsi-amazon Oct 25, 2023

Add where clause support in create statement #2366

Add where clause support in create statement #2366

Conversation

dai-chen commented Oct 25, 2023 • edited Loading

Description

Issues Resolved

Check List

codecov bot commented Oct 25, 2023 • edited Loading

Codecov Report

vamsi-amazon Oct 25, 2023

Choose a reason for hiding this comment

dai-chen Oct 25, 2023

Choose a reason for hiding this comment

vamsi-amazon Oct 25, 2023

Choose a reason for hiding this comment

dai-chen Oct 25, 2023 • edited Loading

Choose a reason for hiding this comment

vamsi-amazon Oct 25, 2023

Choose a reason for hiding this comment

dai-chen commented Oct 25, 2023 •

edited

Loading

codecov bot commented Oct 25, 2023 •

edited

Loading

dai-chen Oct 25, 2023 •

edited

Loading