Add support for YugabyteDB connector #5352

ebyhr · 2020-09-30T01:45:42Z

Yugabyte offers JDBC drivers, but it has BETA label for now.
https://docs.yugabyte.com/latest/reference/drivers/yugabytedb-jdbc-driver/

Another option is using Java client driver that is based on DataStax's one (similar to the current Cassandra connector).
https://docs.yugabyte.com/latest/reference/drivers/ycql-client-drivers/#java

Related to #4836 #4838

Add Yugabyte plugin (Add Yugabyte connector #5708)
Add Yugabyte plugin docs
CI integration for https://github.com/prestosql/presto-yugabyte-driver project (see Add pom.xml and Dummy.java for presto-yugabyte-driver trino-yugabyte-driver#1 (comment))
Add Yugabyte plugin product tests

Reported issue:

The text was updated successfully, but these errors were encountered:

tedyu · 2020-09-30T16:14:25Z

Suggested change for presto-cassandra-driver to connect to Yugabyte DB:

diff --git a/pom.xml b/pom.xml
index c0c71c2..ebf25d9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -41,11 +41,17 @@

     <dependencies>
         <dependency>
-            <groupId>com.datastax.cassandra</groupId>
+            <groupId>com.yugabyte</groupId>
             <artifactId>cassandra-driver-core</artifactId>
-            <version>3.6.0</version>
+            <version>3.2.0-yb-19</version>
             <scope>runtime</scope>
         </dependency>
+            <dependency>
+                <groupId>com.google.guava</groupId>
+                <artifactId>guava</artifactId>
+                <version>19.0</version>
+              <scope>runtime</scope>
+            </dependency>
     </dependencies>

     <build>
@@ -87,7 +93,7 @@
                             <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
                             <artifactSet>
                                 <includes>
-                                    <include>com.datastax.cassandra:cassandra-driver-core</include>
+                                    <include>com.yugabyte:cassandra-driver-core</include>
                                     <include>com.google.guava:guava</include>
                                     <include>org.ow2.asm:*</include>
                                     <include>com.github.jnr:*</include>

ebyhr · 2020-09-30T16:18:05Z

@tedyu I don't think we can replace the dependency. We will need to create a new connector.

tedyu · 2020-09-30T16:23:48Z

Agreed to creating new connector for Presto.

The Yugabyte JDBC driver covers both YCQL (for Cassandra) and YSQL (for Postgres).
The Beta label is for some YSQL feature to be added, not for YCQL.

For YCQL, Presto should be able to use Yugabyte JDBC driver.

tedyu · 2020-09-30T16:54:14Z

@ebyhr
Do you mind creating a new connector for Yugabyte DB (via Yugabyte JDBC driver) ?

Thanks

findepi · 2020-10-02T15:20:13Z

Do you mind creating a new connector for Yugabyte DB (via Yugabyte JDBC driver) ?

Since Yugabyte is different from Cassandra, yes we should have dedicated connector.
Does Yugabyte aims for maintaining JDBC and behavior compatibility with Cassandra (also for future Cassandra changes)
If yes, we could perhaps consider creating Yugabyte connector which extends Cassandra connector.
We should also consider copying the connector -- as we did for eg MemSQL (#1906), since MemSQL does not apparently aim at maintaining compat with MySQL.

@tedyu do you want to work on this?

@ebyhr @tedyu @electrum let's have agreement before commencing the work.

tedyu · 2020-10-02T15:55:43Z

Yugabyte does plan to maintain JDBC and behavior compatibility with Cassandra.

The suggested change in #5352 (comment) references cassandra-driver-core maintained by yugabyte.
This is for the Presto Yugabyte connector (to be created in new repo).

tedyu · 2020-10-09T18:22:53Z

@ebyhr @findepi

Given the change in:
#5352 (comment)

Can you help create repository for Presto connector ?

Thanks

findepi · 2020-10-09T21:31:56Z

@tedyu i am not sure what kind of repository this would be.
All the supported connectors are directly in this repository, as maven modules.
(https://github.com/prestosql/presto-cassandra-driver exists for technical reasons, this is not a Presto plugin)

tedyu · 2020-10-09T22:26:25Z

I was expecting https://github.com/prestosql/presto-yugabytedb-driver or something like that to be created.

Thanks

findepi · 2020-10-10T17:10:18Z

@tedyu presto-cassandra-driver exists because we need to re-package the driver, shading-in some dependencies.
Is it also the case for Yugabyte driver? what are its dependencies?

tedyu · 2020-10-10T22:32:36Z

yb-cassandra-dep.txt

Please see the attached file (up to line 76)

findepi · 2020-10-12T13:34:46Z

is Yugabyte driver a fork of cassandra driver?
i think the guava (v 19) dep can be a problem.
OK, i created a https://github.com/prestosql/presto-yugabyte-driver repo.

findepi · 2020-10-13T08:36:47Z

@tedyu can you please created a companion PR here adding the new module?
also, please make sure to sign the CLA

tedyu · 2020-10-13T15:41:43Z

For the companion PR, wouldn't it be dependent on the availability of presto-yugabyte-driver in maven ?

Signature has been sent to cla@prestosql.io. Is there more to be done in this regard ?

findepi · 2020-10-16T13:59:40Z

Signature has been sent to cla@prestosql.io. Is there more to be done in this regard ?

cc @martint

For the companion PR, wouldn't it be dependent on the availability of presto-yugabyte-driver in maven ?

the PR won't build on CI until presto-yugabyte-driver is published,
but will still allow a reviewer to build both and verify that what we have in trinodb/trino-yugabyte-driver#1 is sufficient

the next step would be to merge trinodb/trino-yugabyte-driver#1, release do maven, etc.

tedyu · 2020-10-16T15:43:41Z

The following would allow building against presto-yugabyte-driver :

diff --git a/pom.xml b/pom.xml
index a275578596..f3e9794cc7 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1727,6 +1727,18 @@
                 </exclusions>
             </dependency>

+            <dependency>
+                <groupId>com.facebook.presto.yugabyte</groupId>
+                <artifactId>yugabyte-driver</artifactId>
+                <version>3.6.0-2-SNAPSHOT</version>
+                <exclusions>
+                    <exclusion>
+                        <groupId>io.netty</groupId>
+                        <artifactId>*</artifactId>
+                    </exclusion>
+                </exclusions>
+            </dependency>
+
             <dependency>
                 <groupId>com.facebook.presto.pinot</groupId>
                 <artifactId>pinot-driver</artifactId>
diff --git a/presto-cassandra/pom.xml b/presto-cassandra/pom.xml
index ecd26c5512..ae4678dca4 100644
--- a/presto-cassandra/pom.xml
+++ b/presto-cassandra/pom.xml
@@ -17,8 +17,8 @@

     <dependencies>
         <dependency>
-            <groupId>com.facebook.presto.cassandra</groupId>
-            <artifactId>cassandra-driver</artifactId>
+            <groupId>com.facebook.presto.yugabyte</groupId>
+            <artifactId>yugabyte-driver</artifactId>
         </dependency>

         <dependency>

Do you have suggestion other than copying the whole presto-cassandra subproject ?

Thanks

findepi · 2020-10-20T14:57:23Z

Do you have suggestion other than copying the whole presto-cassandra subproject ?

Let's start simpler:

create new presto-yugabyte module
have presto-cassandra as a dependency
exclude cassandra driver and add yugabyte driver

tedyu · 2020-10-21T16:47:22Z

yuga-module.txt

See if the patch is on right track.

Currently trying to figure out which Enforcer rule fails:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M2:enforce (default) on project presto-yugabyte: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M2:enforce (default) on project presto-yugabyte: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed.

findepi · 2020-10-26T10:17:48Z

yuga-module.txt

See if the patch is on right track.

instead of a diff, why don't you make a PR?
it will be much easier to work with, comment, etc.

tedyu · 2020-10-26T12:40:00Z

Even with -X option, I didn't get more clue on which enforcer rule was broken.

Once the compilation passes, I would send out the PR.

Meanwhile, suggestion on how to fix compilation is welcome.

findepi · 2020-10-26T15:48:44Z

Meanwhile, suggestion on how to fix compilation is welcome.

let's see how it looks like in the PR.

tedyu · 2020-10-26T16:07:59Z

prestodb/presto#15357

ebyhr · 2020-10-27T00:11:36Z

@tedyu It seems you sent PR to different repository. Could you resend to prestosql/presto?

tedyu · 2020-10-27T14:09:00Z

#5708

ebyhr · 2021-08-15T13:50:57Z

Yugabyte (CQL) doesn't support setting a comment in both CREATE TABLE and ALTER TABLE.
https://docs.yugabyte.com/latest/api/ycql/ddl_create_table/#table-properties-1

This limitation breaks INSERT statement and affects to some operations because Cassandra connector uses the comment to manage hidden columns.

tedyu · 2021-09-01T16:00:31Z

I logged yugabyte/yugabyte-db#9902 for comment enhancement in YCQL.

ebyhr added the enhancement New feature or request label Sep 30, 2020

findepi mentioned this issue Sep 30, 2020

Presto server may not start when there is Error parsing schema #5346

Closed

tedyu mentioned this issue Oct 27, 2020

Add Yugabyte connector #5708

Closed

3 tasks

findepi mentioned this issue Oct 27, 2020

Fix JdbcRecordCursor.isNull for types mapped to slice #5711

Closed

ebyhr mentioned this issue Jun 13, 2023

Cassandra connector doesn't discover new keyspaces/tables with Yugabyte #4838

Closed

ebyhr mentioned this issue Jul 19, 2023

Add connector for YugabyteDB (YSQL) #18336

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for YugabyteDB connector #5352

Add support for YugabyteDB connector #5352

ebyhr commented Sep 30, 2020 •

edited

Loading

tedyu commented Sep 30, 2020

ebyhr commented Sep 30, 2020 •

edited

Loading

tedyu commented Sep 30, 2020 •

edited

Loading

tedyu commented Sep 30, 2020

findepi commented Oct 2, 2020

tedyu commented Oct 2, 2020

tedyu commented Oct 9, 2020

findepi commented Oct 9, 2020

tedyu commented Oct 9, 2020

findepi commented Oct 10, 2020

tedyu commented Oct 10, 2020

findepi commented Oct 12, 2020

findepi commented Oct 13, 2020

tedyu commented Oct 13, 2020

findepi commented Oct 16, 2020

tedyu commented Oct 16, 2020

findepi commented Oct 20, 2020

tedyu commented Oct 21, 2020

findepi commented Oct 26, 2020

tedyu commented Oct 26, 2020

findepi commented Oct 26, 2020

tedyu commented Oct 26, 2020

ebyhr commented Oct 27, 2020

tedyu commented Oct 27, 2020

ebyhr commented Aug 15, 2021 •

edited

Loading

tedyu commented Sep 1, 2021 •

edited

Loading

Add support for YugabyteDB connector #5352

Add support for YugabyteDB connector #5352

Comments

ebyhr commented Sep 30, 2020 • edited Loading

tedyu commented Sep 30, 2020

ebyhr commented Sep 30, 2020 • edited Loading

tedyu commented Sep 30, 2020 • edited Loading

tedyu commented Sep 30, 2020

findepi commented Oct 2, 2020

tedyu commented Oct 2, 2020

tedyu commented Oct 9, 2020

findepi commented Oct 9, 2020

tedyu commented Oct 9, 2020

findepi commented Oct 10, 2020

tedyu commented Oct 10, 2020

findepi commented Oct 12, 2020

findepi commented Oct 13, 2020

tedyu commented Oct 13, 2020

findepi commented Oct 16, 2020

tedyu commented Oct 16, 2020

findepi commented Oct 20, 2020

tedyu commented Oct 21, 2020

findepi commented Oct 26, 2020

tedyu commented Oct 26, 2020

findepi commented Oct 26, 2020

tedyu commented Oct 26, 2020

ebyhr commented Oct 27, 2020

tedyu commented Oct 27, 2020

ebyhr commented Aug 15, 2021 • edited Loading

tedyu commented Sep 1, 2021 • edited Loading

ebyhr commented Sep 30, 2020 •

edited

Loading

ebyhr commented Sep 30, 2020 •

edited

Loading

tedyu commented Sep 30, 2020 •

edited

Loading

ebyhr commented Aug 15, 2021 •

edited

Loading

tedyu commented Sep 1, 2021 •

edited

Loading