Add SQL Server Connector #481

sanjay990 · 2017-01-12T01:15:25Z

Initial commit with SQL Server connector. I am working on adding additional tests.
The tests run SQL Server in docker.

mssql-jdbc requires commons-codec 1.10

ArturGajowy

Didn't look at the product tests in depth. Also, the connector changes seem really minimal - surprising only this much is needed. Maybe we should have a generic (no code changes needed) jdbc connector already.

ArturGajowy · 2017-01-16T15:25:39Z

presto-sqlserver/pom.xml

+                    <groupId>com.sun.jersey</groupId>
+                    <artifactId>jersey-core</artifactId>
+                </exclusion>
+


Unneded whitespace

ArturGajowy · 2017-01-16T15:30:22Z

pom.xml

+                                <excludes>
+                                    <exclude>META-INF/*.SF</exclude>
+                                    <exclude>META-INF/*.DSA</exclude>
+                                    <exclude>META-INF/*.RSA</exclude>


This seems like an unrelated change. Also, there should be a comment why those excludes are needed.

OK, will add comment and probably put them in a separate commit.

ArturGajowy · 2017-01-16T15:31:39Z

presto-sqlserver/pom.xml

@@ -5,7 +5,7 @@
    <parent>
        <artifactId>presto-root</artifactId>
        <groupId>com.facebook.presto</groupId>
-        <version>0.160-SNAPSHOT</version>
+        <version>0.162-SNAPSHOT-t.1-SNAPSHOT</version>


Unrelated change, probably should be in previous commit

ArturGajowy · 2017-01-16T15:32:22Z

presto-main/etc/config.properties

@@ -38,7 +38,6 @@ plugin.bundles=\
  ../presto-local-file/pom.xml, \
  ../presto-mysql/pom.xml,\
  ../presto-postgresql/pom.xml,\
-  ../presto-postgresql/pom.xml, \


Unrelated change

I'll put that into separate commit.

ArturGajowy · 2017-01-16T15:35:06Z

presto-product-tests/bin/run_on_docker.sh

@@ -207,7 +207,7 @@ fi
 trap terminate INT TERM EXIT

 # start external services
-EXTERNAL_SERVICES="hadoop-master mysql postgres cassandra"
+EXTERNAL_SERVICES="hadoop-master mysql postgres cassandra sqlserver"


We're running quite a lot of additional services for the product tests, even if they aren't used. Does it pass on travis? We should think how to only run the necessary services. For this PR it's fine though as long as the build is stable

Yes, the tests are passing on Travis. I strong believe we should have a way to run only hadoop-master + presto-master + related container, rather than starting all containers while running any given product-tests.

ArturGajowy · 2017-01-16T15:36:59Z

pom.xml

@@ -50,7 +50,7 @@
        <dep.packaging.version>${dep.airlift.version}</dep.packaging.version>
        <dep.slice.version>0.27</dep.slice.version>
        <dep.aws-sdk.version>1.11.30</dep.aws-sdk.version>
-        <dep.tempto.version>1.22</dep.tempto.version>
+        <dep.tempto.version>1.23</dep.tempto.version>


I'd expect a separate 'update tempto' commit

I already updated tempto to 1.23 on prestodb/master. This will be removed while merging.

This should be separate commit anyway though. When on master it would it would be empty commit.

ArturGajowy · 2017-01-16T15:46:47Z

Someone having more experience with connectors should take a look.

ArturGajowy · 2017-01-16T15:52:18Z

Took another look at the product tests and they seem far simpler than what we have for cassandra ATM. We wanted those to be similar.

Also, cassandra uses code-defined product tests while sqlserver - convention based. I'd opt for having code-defined product tests reused for testing both connectors.

sanjay990 · 2017-01-16T15:57:38Z

@ArturGajowy - Yes, java based based are on their way. The convention based tests are only for testing basic SQL queries.

maciejgrzybek · 2017-01-16T21:00:41Z

presto-sqlserver/pom.xml

+    <parent>
+        <artifactId>presto-root</artifactId>
+        <groupId>com.facebook.presto</groupId>
+        <version>0.160-SNAPSHOT</version>


It's against sprint-47 so it should be 0.162-SNAPSHOT-t.1-SNAPSHOT.

maciejgrzybek · 2017-01-16T21:04:53Z

presto-sqlserver/pom.xml

+            <scope>provided</scope>
+        </dependency>
+
+        <!-- Presto SPI -->


This is unnecessary.

maciejgrzybek · 2017-01-16T21:12:01Z

pom.xml

@@ -50,7 +50,7 @@
        <dep.packaging.version>${dep.airlift.version}</dep.packaging.version>
        <dep.slice.version>0.27</dep.slice.version>
        <dep.aws-sdk.version>1.11.30</dep.aws-sdk.version>
-        <dep.tempto.version>1.22</dep.tempto.version>
+        <dep.tempto.version>1.23</dep.tempto.version>


This should be separate commit anyway though. When on master it would it would be empty commit.

maciejgrzybek · 2017-01-16T21:13:44Z

presto-product-tests/conf/docker/common/jdbc_db.yml

+    hostname: sqlserver
+    image: 'microsoft/mssql-server-linux'
+    ports:
+      - '1433:1433'


Please use not default port if possible, similarly to what we have for postgres and mysql.
This is to prevent clash with dev env while working with another instance of MsSQL.

Unlike MySQL and PostgreSQL, we can't pass port as the parameter. We will have to create a custom docker image on top of sql-server. As discussed, for now I'll just run on 1433

I have created this issue to track this for SQL Server in Docker microsoft/mssql-docker#21

maciejgrzybek · 2017-01-16T21:14:44Z

presto-product-tests/conf/docker/common/jdbc_db.yml

+    ports:
+      - '1433:1433'
+    environment:
+      ACCEPT_EULA: Y


How does it work with our apache-licensed software and our automation? Can we legally use it on Travis?

This should be AT LEAST mentioned in README.

maciejgrzybek · 2017-01-16T21:15:18Z

presto-product-tests/conf/presto/etc/catalog/sqlserver.properties

@@ -0,0 +1,4 @@
+connector.name=sqlserver
+connection-url=jdbc:sqlserver://sqlserver
+connection-user=sa


Why sa? Why don't we use the same credentials as for other JDBC connectors?

We don't have that provision. Microsoft is working on that

I'll ignore it for this PR

maciejgrzybek · 2017-01-16T21:15:26Z

presto-product-tests/src/main/resources/tempto-configuration.yaml

+    jdbc_driver_class: com.microsoft.sqlserver.jdbc.SQLServerDriver
+    jdbc_url: jdbc:sqlserver://sqlserver
+    jdbc_user: sa
+    jdbc_password: SQLServerPass1


maciejgrzybek · 2017-01-16T21:16:27Z

presto-product-tests/bin/run_on_docker.sh

@@ -102,7 +102,7 @@ function stop_docker_compose_containers() {
    stop_application_runner_containers ${ENVIRONMENT}

    # stop containers started with "up", removing their volumes
-    environment_compose down -v
+    environment_compose down -v || true


Why this is needed? Is there a legitimate reason to accept not gracefully shut down sql server container?

I'd say yes - we had cases where all the tests passed, yet docker-compose got a timeout when trying to communicate with the sqlserver container. Since the problem is hard to reproduce locally, we've no idea how to approach a fix, and it does not affect the tests themselves - I'd stick with it.

Then we should have a comment next to that line to make it clear it shouldn't be that way but we cannot fix it.

will add a comment.

fixed - 7929af9

maciejgrzybek · 2017-01-16T21:17:35Z

presto-sqlserver/src/main/java/com/facebook/presto/plugin/sqlserver/SqlServerClient.java

+    {
+        super(connectorId, config, "\"", new SQLServerDriver());
+    }
+}


SqlServer supports transactional DDLs. Why don't you use it instead of relying on default implementation of BaseJdbcClient providing non-ACID compliant DDLs?

we can't use transactions because the statements are executed from multiple connections.

rschlussel-zz · 2017-01-24T17:06:27Z

I fixed an issue with create table for SQL Server where the syntax used in the BaseJdbcClient wasn't supported in SQL Server. I'll talk to Sanjay about product tests.

maciejgrzybek · 2017-01-24T17:41:22Z

presto-sqlserver/src/main/java/com/facebook/presto/plugin/sqlserver/SqlServerClient.java

+    public void commitCreateTable(JdbcOutputTableHandle handle)
+    {
+        StringBuilder sql = new StringBuilder()
+                .append("sp_rename ")


Why?
Why not overriding beginCreateTable, then having commitCreateTable straightforward - simply calling commit?

We can't do that because it's done in two different connections, so the transaction's gone (you get an error "The commit transaction request has no corresponding begin transaction"). also in the interim you could have an insert into, which is done separately, but also needs to be a part of the same transaction.

maciejgrzybek · 2017-01-25T01:55:44Z

After offline discussion, when it was confirmed by @rschlussel that:

we cannot keep transaction between SPI metadata calls (due to non-persistent connection)
it's not possible to attach to transaction started in other connection

I approve these changes.

akshatnair · 2017-01-25T20:10:03Z

https://prestodb.io/docs/current/connector/mysql.html
We need a similar doc page, if not in this PR then please make sure this is done after this,

akshatnair

LGTM mod small comment

akshatnair · 2017-01-25T20:10:50Z

pom.xml

@@ -616,6 +623,12 @@
            </dependency>

            <dependency>
+                <groupId>com.microsoft.sqlserver</groupId>
+                <artifactId>mssql-jdbc</artifactId>
+                <version>6.1.0.jre8</version>


Make sure we know and document what versions of MS SQL Server this driver supports

sanjay990 · 2017-01-31T23:00:17Z

I'll close this PR and make the docs changes in a separate commit

Presto-product-tests has a dependency on SQL Server (mssql-jdbc) which is a signed jar. Its invalidated when we repackage it into an uber jar. Hence to avoid Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes We remove signed information from META-INF

ALTER TABLE cannot be used to rename a table in SQL Server, instead sp_rename must be used. Override the commitCreatetable method from BaseJdbcClient to use the correct SQL Server syntax.

Upgrade commons-codec to 1.10

d9e508f

mssql-jdbc requires commons-codec 1.10

sanjay990 mentioned this pull request Jan 12, 2017

Add SQL Server connector #389

Closed

sanjay990 force-pushed the sprint-sqlserver branch from 401c251 to d232ca3 Compare January 12, 2017 22:46

ArturGajowy reviewed Jan 16, 2017

View reviewed changes

maciejgrzybek suggested changes Jan 16, 2017

View reviewed changes

sanjay990 force-pushed the sprint-sqlserver branch from d232ca3 to 7929af9 Compare January 17, 2017 23:05

LuisBosquez mentioned this pull request Jan 20, 2017

Need to change default port on container execution microsoft/mssql-docker#21

Open

maciejgrzybek suggested changes Jan 24, 2017

View reviewed changes

maciejgrzybek approved these changes Jan 25, 2017

View reviewed changes

akshatnair approved these changes Jan 25, 2017

View reviewed changes

Sanjay Sharma and others added 8 commits January 31, 2017 18:17

Add sqlserver connector

3eabf2c

Add Sql-Server convention based product-tests

4002bb1

Update for tempto 1.23

7c47c52

Enable sql-server product-tests on docker

3c85247

Pass product-tests even if containers fail to stop

9e7f1ea

Add SqlServer connector to config.properties

8ed5b07

Support CREATE TABLE AS SELECT for SQL Server

627313b

ALTER TABLE cannot be used to rename a table in SQL Server, instead sp_rename must be used. Override the commitCreatetable method from BaseJdbcClient to use the correct SQL Server syntax.

sanjay990 force-pushed the sprint-sqlserver branch from 19b2b7b to 627313b Compare February 1, 2017 15:30

sanjay990 closed this Feb 14, 2017

sanjay990 mentioned this pull request Feb 17, 2017

Add support for SQL Server connector prestodb/presto#7401

Closed

sanjay990 deleted the sprint-sqlserver branch March 29, 2017 23:59

Add SQL Server Connector #481

Add SQL Server Connector #481

Conversation

sanjay990 commented Jan 12, 2017

ArturGajowy left a comment

Choose a reason for hiding this comment

ArturGajowy Jan 16, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ArturGajowy commented Jan 16, 2017

ArturGajowy commented Jan 16, 2017

sanjay990 commented Jan 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanjay990 Jan 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanjay990 Jan 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanjay990 Jan 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rschlussel-zz commented Jan 24, 2017

Choose a reason for hiding this comment

rschlussel-zz Jan 24, 2017 • edited Loading

Choose a reason for hiding this comment

maciejgrzybek commented Jan 25, 2017

akshatnair commented Jan 25, 2017

akshatnair left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanjay990 commented Jan 31, 2017

ArturGajowy Jan 16, 2017 •

edited

Loading

sanjay990 Jan 18, 2017 •

edited

Loading

sanjay990 Jan 18, 2017 •

edited

Loading

sanjay990 Jan 17, 2017 •

edited

Loading

rschlussel-zz Jan 24, 2017 •

edited

Loading