Add Statement #2294

penghuo · 2023-10-13T22:59:26Z

Description

add statement implementation.
refactor SessionStateStore to generic StateStore for Statement and Session.
fixed flaky test. [Backport 2.x] add InteractiveSession and SessionManager #2293

Issues Resolved

#2089

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo · 2023-10-13T23:03:58Z

spark/src/main/java/org/opensearch/sql/spark/execution/statestore/StateStore.java

+    }
+  }
+
+  /** Helper Functions */


Refactor as StateStore, Add Helper functions for Session and Statement access.

Signed-off-by: Peng Huo <penghuo@gmail.com>

codecov · 2023-10-13T23:19:19Z

Codecov Report

Merging #2294 (eafbeb4) into main (f856cb3) will decrease coverage by 0.04%.
Report is 1 commits behind head on main.
The diff coverage is 93.20%.

@@             Coverage Diff              @@
##               main    #2294      +/-   ##
============================================
- Coverage     96.61%   96.58%   -0.04%     
- Complexity     4777     4823      +46     
============================================
  Files           449      454       +5     
  Lines         12907    13134     +227     
  Branches        882      887       +5     
============================================
+ Hits          12470    12685     +215     
- Misses          429      436       +7     
- Partials          8       13       +5

Flag	Coverage Δ
sql-engine	`96.58% <93.20%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...ql/spark/execution/session/InteractiveSession.java	`100.00% <100.00%> (ø)`
...ch/sql/spark/execution/session/SessionManager.java	`100.00% <100.00%> (ø)`
...arch/sql/spark/execution/session/SessionModel.java	`98.80% <100.00%> (+0.32%)`	⬆️
...arch/sql/spark/execution/session/SessionState.java	`100.00% <100.00%> (ø)`
...earch/sql/spark/execution/statement/Statement.java	`100.00% <100.00%> (ø)`
...rch/sql/spark/execution/statement/StatementId.java	`100.00% <100.00%> (ø)`
.../sql/spark/execution/statement/StatementState.java	`100.00% <100.00%> (ø)`
...rch/sql/spark/execution/statestore/StateModel.java	`100.00% <100.00%> (ø)`
.../sql/spark/execution/statement/StatementModel.java	`97.91% <97.91%> (ø)`
...rch/sql/spark/execution/statestore/StateStore.java	`71.42% <71.42%> (ø)`

... and 1 file with indirect coverage changes

anirudha

Thanks ton!

kaituo · 2023-10-16T18:33:11Z

spark/src/main/java/org/opensearch/sql/spark/execution/session/InteractiveSession.java

+      } else {
+        String errMsg =
+            String.format(
+                "can't submit statement, session should in running state, "


should in => should be in

kaituo · 2023-10-16T18:50:00Z

spark/src/main/java/org/opensearch/sql/spark/execution/session/InteractiveSession.java

+      throw new IllegalStateException("session does not exist. " + sessionModel.getSessionId());
+    } else {
+      sessionModel = model.get();
+      if (sessionModel.getSessionState() == SessionState.RUNNING) {


Spark job sets session state to RUNNING, right? If yes, can we allow submitting statements as long it is it not dead or fail state? SQL does not know when Spark job will set the state to RUNNING. It helps unblock statements submissions without waiting for spark job.

Spark job sets session state to RUNNING, right?
Yes.

SQL does not know when Spark job will set the state to RUNNING. It helps unblock statements submissions without waiting for spark job.
Make sense.

kaituo · 2023-10-16T20:35:54Z

spark/src/main/java/org/opensearch/sql/spark/execution/statement/StatementModel.java

+  public static final String ERROR = "error";
+  public static final String UNKNOWN = "unknown";
+  public static final String STATEMENT_DOC_TYPE = "statement";
+


Can we add application id? Spark queries application id to make sure new queries will be sent to new application when the old application is terminated.

Could query session instead? applicationId is associated with session.

dai-chen

Thanks for the changes! Could you add missing Javadoc on new interface/class?

dai-chen · 2023-10-16T20:47:48Z

spark/src/main/java/org/opensearch/sql/spark/execution/statestore/StateStore.java

+              .setIfSeqNo(model.getSeqNo())
+              .setIfPrimaryTerm(model.getPrimaryTerm())
+              .doc(model.toXContent(XContentFactory.jsonBuilder(), ToXContent.EMPTY_PARAMS))
+              .fetchSource(true)


Is this required?

Do u mean OCC? Yes. Plugin move statement to cancelled state, it could be multiple jvm.

Sorry, only this line .fetchSource(true)

Signed-off-by: Peng Huo <penghuo@gmail.com>

* add InteractiveSession and SessionManager Signed-off-by: Peng Huo <penghuo@gmail.com> * add statement Signed-off-by: Peng Huo <penghuo@gmail.com> * add statement Signed-off-by: Peng Huo <penghuo@gmail.com> * fix format Signed-off-by: Peng Huo <penghuo@gmail.com> * address comments Signed-off-by: Peng Huo <penghuo@gmail.com> --------- Signed-off-by: Peng Huo <penghuo@gmail.com> (cherry picked from commit 297e26f) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

opensearch-trigger-bot · 2023-10-17T16:21:05Z

The backport to 2.11 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.11 2.11
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.11
# Create a new branch
git switch --create backport/backport-2294-to-2.11
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 297e26f9622e66cf01c777d229e9b13dbc19525d
# Push it to GitHub
git push --set-upstream origin backport/backport-2294-to-2.11
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.11

Then, create a pull request where the base branch is 2.11 and the compare/head branch is backport/backport-2294-to-2.11.

* add InteractiveSession and SessionManager * add statement * add statement * fix format * address comments --------- (cherry picked from commit 297e26f) Signed-off-by: Peng Huo <penghuo@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* add InteractiveSession and SessionManager * add statement * add statement * fix format * address comments --------- (cherry picked from commit 297e26f) Signed-off-by: Peng Huo <penghuo@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> (cherry picked from commit 7aa3cab) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* add InteractiveSession and SessionManager * add statement * add statement * fix format * address comments --------- (cherry picked from commit 297e26f) (cherry picked from commit 7aa3cab) Signed-off-by: Peng Huo <penghuo@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

) (opensearch-project#2319)" This reverts commit b3c2e94. Signed-off-by: Eric <menwe@amazon.com>

) (opensearch-project#2319)" This reverts commit b3c2e94.

* Revert "Add more metrics and handle emr exception message (#2422) (#2426)" This reverts commit b57f7cc. * Revert "Block settings in sql query settings API and add more unit tests (#2407) (#2412)" This reverts commit 3024737. * Revert "Added session, statement, emrjob metrics to sql stats api (#2398) (#2400)" This reverts commit 6e17ae6. * Revert "Redefine Drop Index as logical delete (#2386) (#2397)" This reverts commit e939bb6. * Revert "add concurrent limit on datasource and sessions (#2390) (#2395)" This reverts commit deb3ccf. * Revert "Add Flint Index Purging Logic (#2372) (#2389)" This reverts commit dd48b9b. * Revert "Refactoring for tags usage in test files and also added explicit denly list setting. (#2383) (#2385)" This reverts commit 37e010f. * Revert "Enable session by default (#2373) (#2375)" This reverts commit 7d95e4c. * Revert "Create new session if client provided session is invalid (#2368) (#2371)" This reverts commit 5ab7858. * Revert "Add where clause support in create statement (#2366) (#2370)" This reverts commit b620a56. * Revert "create new session if current session not ready (#2363) (#2365)" This reverts commit 5d07281. * Revert "Handle Describe,Refresh and Show Queries Properly (#2357) (#2362)" This reverts commit 16e2f30. * Revert "Add Session limitation (#2354) (#2359)" This reverts commit 0f334f8. * Revert "Bug Fix, support cancel query in running state (#2351) (#2353)" This reverts commit 9a40591. * Revert "Fix bug, using basic instead of basicauth (#2342) (#2355)" This reverts commit e4827a5. * Revert "Add missing tags and MV support (#2336) (#2346)" This reverts commit 8791bb0. * Revert "[Backport 2.x] deprecated job-metadata-index (#2340) (#2343)" This reverts commit bea432c. * Revert "Integration with REPL Spark job (#2327) (#2338)" This reverts commit 58a5ae5. * Revert "Implement patch API for datasources (#2273) (#2329)" This reverts commit 4c151fe. * Revert "Add sessionId parameters for create async query API (#2312) (#2324)" This reverts commit 3d1a376. * Revert "Add Statement (#2294) (#2318) (#2319)" This reverts commit b3c2e94. * Revert "Upgrade json (#2307) (#2314)" This reverts commit 6c65bb4. * Revert "Minor Refactoring (#2308) (#2317)" This reverts commit 051cc4f. * Revert "add InteractiveSession and SessionManager (#2290) (#2293) (#2315)" This reverts commit 6ac197b. --------- Co-authored-by: Vamsi Manohar <reddyvam@amazon.com>

penghuo added 4 commits October 12, 2023 13:34

add InteractiveSession and SessionManager

78acb1a

Signed-off-by: Peng Huo <penghuo@gmail.com>

add statement

b7b3d77

Signed-off-by: Peng Huo <penghuo@gmail.com>

add statement

aab6651

Signed-off-by: Peng Huo <penghuo@gmail.com>

update

8d4dc76

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo requested review from pjfitzgibbons, ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, rupal-bq, mengweieric, vamsi-amazon, Swiddis, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto and GumpacG as code owners October 13, 2023 22:59

penghuo changed the title ~~Feature/repl/statement pr~~ Add Statement Oct 13, 2023

penghuo mentioned this pull request Oct 13, 2023

[FEATURE] OpenSearch and EMR-Serverless Integration #2089

Open

46 tasks

penghuo commented Oct 13, 2023

View reviewed changes

fix format

ff02f28

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo self-assigned this Oct 13, 2023

penghuo added Flint backport 2.x labels Oct 13, 2023

anirudha previously approved these changes Oct 14, 2023

View reviewed changes

kaituo reviewed Oct 16, 2023

View reviewed changes

dai-chen previously approved these changes Oct 16, 2023

View reviewed changes

address comments

eafbeb4

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo dismissed stale reviews from dai-chen and anirudha via eafbeb4 October 17, 2023 00:49

kaituo approved these changes Oct 17, 2023

View reviewed changes

dai-chen approved these changes Oct 17, 2023

View reviewed changes

vamsi-amazon approved these changes Oct 17, 2023

View reviewed changes

penghuo merged commit 297e26f into opensearch-project:main Oct 17, 2023
19 of 21 checks passed

penghuo added the backport 2.11 label Oct 17, 2023

opensearch-trigger-bot bot mentioned this pull request Oct 17, 2023

[Backport 2.x] Add Statement #2318

Merged

opensearch-trigger-bot bot added the backport-failed label Oct 17, 2023

joshuali925 removed the backport-failed label Oct 30, 2023

mengweieric added a commit to mengweieric/sql that referenced this pull request Nov 8, 2023

Revert "Add Statement (opensearch-project#2294) (opensearch-project#2318

3128fac

) (opensearch-project#2319)" This reverts commit b3c2e94. Signed-off-by: Eric <menwe@amazon.com>

mengweieric added a commit to mengweieric/sql that referenced this pull request Nov 8, 2023

Revert "Add Statement (opensearch-project#2294) (opensearch-project#2318

ed8c9f0

) (opensearch-project#2319)" This reverts commit b3c2e94. Signed-off-by: Eric <menwe@amazon.com>

vamsi-amazon added a commit to mengweieric/sql that referenced this pull request Nov 13, 2023

Revert "Add Statement (opensearch-project#2294) (opensearch-project#2318

77af17c

) (opensearch-project#2319)" This reverts commit b3c2e94.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Statement #2294

Add Statement #2294

penghuo commented Oct 13, 2023 •

edited

Loading

penghuo Oct 13, 2023

codecov bot commented Oct 13, 2023 •

edited

Loading

anirudha left a comment

kaituo Oct 16, 2023

kaituo Oct 16, 2023

penghuo Oct 16, 2023

kaituo Oct 16, 2023

penghuo Oct 16, 2023

penghuo Oct 17, 2023

dai-chen left a comment

dai-chen Oct 16, 2023

penghuo Oct 17, 2023

dai-chen Oct 17, 2023

opensearch-trigger-bot bot commented Oct 17, 2023

Add Statement #2294

Add Statement #2294

Conversation

penghuo commented Oct 13, 2023 • edited Loading

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

codecov bot commented Oct 13, 2023 • edited Loading

Codecov Report

anirudha left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dai-chen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Oct 17, 2023

penghuo commented Oct 13, 2023 •

edited

Loading

codecov bot commented Oct 13, 2023 •

edited

Loading