[Backport to 2.x]Catalog to Datasource changes (#1027) #1049

vamsi-amazon · 2022-11-08T22:41:01Z

Signed-off-by: vamsi-amazon reddyvam@amazon.com
(cherry picked from commit 3e30379)

Description

[Describe what this change achieves]

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: vamsi-amazon <reddyvam@amazon.com> (cherry picked from commit 3e30379)

joshuali925

should 2.4 be merged into 2.x?

vamsi-amazon · 2022-11-08T22:50:55Z

should 2.4 be merged into 2.x?

Yes, But I am not sure if there are any pending PRs which are not merged into 2.x

vamsi-amazon · 2022-11-08T22:51:12Z

should 2.4 be merged into 2.x?

Yes, But I am not sure if there are any pending PRs which are not merged into 2.x

This is the only one I could find.

codecov-commenter · 2022-11-08T22:58:07Z

Codecov Report

Merging #1049 (8a05fe7) into 2.x (1a52511) will decrease coverage by 2.41%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##                2.x    #1049      +/-   ##
============================================
- Coverage     98.21%   95.80%   -2.42%     
- Complexity     3268     3410     +142     
============================================
  Files           314      336      +22     
  Lines          8197     9326    +1129     
  Branches        533      700     +167     
============================================
+ Hits           8051     8935     +884     
- Misses          142      334     +192     
- Partials          4       57      +53

Flag	Coverage Δ
query-workbench	`62.76% <ø> (?)`
sql-engine	`98.31% <100.00%> (+0.09%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...rch/sql/planner/physical/catalog/CatalogTable.java	`100.00% <ø> (ø)`
...ensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java	`100.00% <ø> (ø)`
...metheus/storage/PrometheusMetricDefaultSchema.java	`100.00% <ø> (ø)`
...ain/java/org/opensearch/sql/analysis/Analyzer.java	`100.00% <100.00%> (ø)`
...sql/planner/physical/catalog/CatalogTableScan.java	`100.00% <100.00%> (ø)`
...l/planner/physical/catalog/CatalogTableSchema.java	`100.00% <100.00%> (ø)`
...sql/opensearch/response/agg/SingleValueParser.java	`100.00% <100.00%> (ø)`
...earch/sql/opensearch/response/agg/StatsParser.java	`100.00% <100.00%> (ø)`
.../opensearch/sql/opensearch/response/agg/Utils.java	`100.00% <100.00%> (ø)`
...java/org/opensearch/sql/ppl/parser/AstBuilder.java	`100.00% <100.00%> (ø)`
... and 41 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

joshuali925 · 2022-11-08T22:59:46Z

ok i checked the other missing one is #1050, should be good

YANG-DB · 2022-11-08T23:18:25Z

docs/user/general/identifiers.rst

@@ -231,7 +231,7 @@ tableName = ``http_requests_total``.
 2. ``logs.12.13.1``


-catalogName = ``@opensearch`` [Resolves to default @opensearch connector since [my_prometheus] is the only catalog configured name.]
+datasourceName = ``@opensearch`` [Resolves to default @opensearch connector since [my_prometheus] is the only catalog configured name.]


missed one here ...

This is a backport PR can't do much. This is a good catch will fix it in coming PR.
The idea of this PR is to remove only public references[Documentation and show catlalogs command only] as it was done on the last day of release.

All the references in the implementation will be removed in a new PR.

YANG-DB · 2022-11-08T23:18:43Z

docs/user/general/identifiers.rst


 schemaName = ``default`` [No supported schema found, so default to `default`].

 tableName = ``prometheus.http_requests_total``.

 5. ``prometheus.default.http_requests_total.1.2.3``

-catalogName = ``@opensearch`` [Resolves to default @opensearch connector since [my_prometheus] is the only catalog configured name.]
+datasourceName = ``@opensearch`` [Resolves to default @opensearch connector since [my_prometheus] is the only catalog configured name.]


missed one here

YANG-DB · 2022-11-08T23:20:33Z

doctest/build.gradle

@@ -104,7 +104,7 @@ String mlCommonsPlugin = 'opensearch-ml'

 testClusters {
    docTestCluster {
-        keystore 'plugins.query.federation.catalog.config', new File("$projectDir/catalog", 'catalog.json')
+        keystore 'plugins.query.federation.datasources.config', new File("$projectDir/catalog", 'catalog.json')


why not change the json file name also into datasource.json ?

YANG-DB · 2022-11-08T23:21:04Z

integ-test/src/test/java/org/opensearch/sql/ppl/ShowCatalogsCommandIT.java

@@ -20,25 +20,25 @@ public class ShowCatalogsCommandIT extends PPLIntegTestCase {

  @Test
  public void testShowCatalogsCommands() throws IOException {


should be testShowDatasourcesCommands()...

YANG-DB · 2022-11-08T23:21:30Z

integ-test/src/test/java/org/opensearch/sql/ppl/ShowCatalogsCommandIT.java

    verifyDataRows(result,
        rows("my_prometheus", "PROMETHEUS"),
        rows("@opensearch", "OPENSEARCH"));
    verifyColumn(
        result,
-        columnName("CATALOG_NAME"),
+        columnName("DATASOURCE_NAME"),
        columnName("CONNECTOR_TYPE")
    );
  }

  @Test
  public void testShowCatalogsCommandsWithWhereClause() throws IOException {


change test name here also...

YANG-DB · 2022-11-08T23:22:46Z

integ-test/build.gradle

@@ -111,7 +111,7 @@ testClusters.all {

 testClusters.integTest {
    plugin ":opensearch-sql-plugin"
-    keystore 'plugins.query.federation.catalog.config', new File("$projectDir/src/test/resources/catalog/", 'catalog.json')
+    keystore 'plugins.query.federation.datasources.config', new File("$projectDir/src/test/resources/catalog/", 'catalog.json')


can the catalog.json also be changed into datasource.json ?

* Fix `FLOAT` -> `DOUBLE` cast. (#1025) Signed-off-by: Yury-Fridlyand <yuryf@bitquilltech.com> * Fix error messaging from prometheus. (#1029) (#1037) Signed-off-by: vamsi-amazon <reddyvam@amazon.com> (cherry picked from commit 4a9cef3) Co-authored-by: vamsi-amazon <reddyvam@amazon.com> * Add `query` function as alternate syntax to `query_string` function (#1010) This maintains backwards compatibility with the v1 engine. * Update DATE and TIME functions to parse string input as datetime (#991) Add option to accept datetime like string in both TIME and DATE (eg. accept "1999-01-02 12:12:12" for both TIME and DATE. Strict check on date for testing for valid dates (eg. Don't accept Feb 30th as a valid date) and throws a SemanticCheckException. Co-authored-by: Yury-Fridlyand <yuryf@bitquilltech.com> Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com> * back quote fix (#1041) (#1050) Signed-off-by: vamsi-amazon <reddyvam@amazon.com> (cherry picked from commit d3bb902) Co-authored-by: vamsi-amazon <reddyvam@amazon.com> * Catalog to Datasource changes (#1027) (#1049) Signed-off-by: vamsi-amazon <reddyvam@amazon.com> (cherry picked from commit 3e30379) * Bump jackson to 2.14.0 (#1058) Signed-off-by: Joshua Li <joshuali925@gmail.com> (cherry picked from commit 5a1adb2) * Add valueOf() to Expression (#1055) Signed-off-by: Joshua Li <joshuali925@gmail.com> Signed-off-by: Yury-Fridlyand <yuryf@bitquilltech.com> Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com> Signed-off-by: Joshua Li <joshuali925@gmail.com> Signed-off-by: Chen Dai <daichen@amazon.com> Co-authored-by: Yury-Fridlyand <yuryf@bitquilltech.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: vamsi-amazon <reddyvam@amazon.com> Co-authored-by: GabeFernandez310 <64448015+GabeFernandez310@users.noreply.github.com> Co-authored-by: MitchellGale-BitQuill <104795536+MitchellGale-BitQuill@users.noreply.github.com> Co-authored-by: Joshua Li <joshuali925@gmail.com>

Catalog to Datasource changes (#1027)

8a05fe7

Signed-off-by: vamsi-amazon <reddyvam@amazon.com> (cherry picked from commit 3e30379)

vamsi-amazon requested a review from a team as a code owner November 8, 2022 22:41

vamsi-amazon changed the title ~~Catalog to Datasource changes (#1027)~~ [Backport to 2.x]Catalog to Datasource changes (#1027) Nov 8, 2022

joshuali925 approved these changes Nov 8, 2022

View reviewed changes

ps48 approved these changes Nov 8, 2022

View reviewed changes

vamsi-amazon merged commit c2ccc91 into 2.x Nov 8, 2022

github-actions bot deleted the backport/backport-1027-to-2.x branch November 8, 2022 23:23

YANG-DB reviewed Nov 8, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backport to 2.x]Catalog to Datasource changes (#1027) #1049

[Backport to 2.x]Catalog to Datasource changes (#1027) #1049

vamsi-amazon commented Nov 8, 2022

joshuali925 left a comment

vamsi-amazon commented Nov 8, 2022

vamsi-amazon commented Nov 8, 2022

codecov-commenter commented Nov 8, 2022 •

edited

joshuali925 commented Nov 8, 2022

YANG-DB Nov 8, 2022

vamsi-amazon Nov 8, 2022

YANG-DB Nov 8, 2022

YANG-DB Nov 8, 2022

YANG-DB Nov 8, 2022

YANG-DB Nov 8, 2022

YANG-DB Nov 8, 2022

		@@ -20,25 +20,25 @@ public class ShowCatalogsCommandIT extends PPLIntegTestCase {

		@Test
		public void testShowCatalogsCommands() throws IOException {

[Backport to 2.x]Catalog to Datasource changes (#1027) #1049

[Backport to 2.x]Catalog to Datasource changes (#1027) #1049

Conversation

vamsi-amazon commented Nov 8, 2022

Description

Issues Resolved

Check List

joshuali925 left a comment

Choose a reason for hiding this comment

vamsi-amazon commented Nov 8, 2022

vamsi-amazon commented Nov 8, 2022

codecov-commenter commented Nov 8, 2022 • edited

Codecov Report

joshuali925 commented Nov 8, 2022

YANG-DB Nov 8, 2022

Choose a reason for hiding this comment

vamsi-amazon Nov 8, 2022

Choose a reason for hiding this comment

YANG-DB Nov 8, 2022

Choose a reason for hiding this comment

YANG-DB Nov 8, 2022

Choose a reason for hiding this comment

YANG-DB Nov 8, 2022

Choose a reason for hiding this comment

YANG-DB Nov 8, 2022

Choose a reason for hiding this comment

YANG-DB Nov 8, 2022

Choose a reason for hiding this comment

codecov-commenter commented Nov 8, 2022 •

edited