Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for new JDBC connector Snowflake #21842

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Support for new JDBC connector Snowflake #21842

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions pom.xml
View file
Open in desktop
Expand Up @@ -190,6 +190,7 @@
<module>redis-hbo-provider</module>
<module>presto-singlestore</module>
<module>presto-hana</module>
<module>presto-snowflake</module>
</modules>

<dependencyManagement>
Expand Down
1 change: 1 addition & 0 deletions presto-docs/src/main/sphinx/connector.rst
View file
Open in desktop
Expand Up @@ -39,6 +39,7 @@ from different data sources.
connector/redshift
connector/scylladb
connector/singlestore
connector/snowflake
connector/sqlserver
connector/system
connector/thrift
Expand Down
115 changes: 115 additions & 0 deletions presto-docs/src/main/sphinx/connector/snowflake.rst
View file
Open in desktop
@@ -0,0 +1,115 @@
===============
Snowflake Connector
===============

The Snowflake connector allows querying and creating tables in an external
Snowflake database. This can be used to join data between different
systems like Snowflake and Hive, or between two different Snowflake accounts.

Configuration
-------------

To configure the Snowflake connector, create a catalog properties file
in ``etc/catalog`` named, for example, ``snowflake.properties``.
Create the file with the following contents, replacing the
connection properties as appropriate for your setup:

.. code-block:: none

connector.name=snowflake
connection-url=jdbc:snowflake://<account_identifier>.snowflakecomputing.com/?db={dbname}
connection-user=root
connection-password=secret

The ``connection-url`` defines the connection information and parameters to pass
to the Snowflake JDBC driver.

.. code-block:: text

connection-url=jdbc:snowflake://<account_identifier>.snowflakecomputing.com/?db={dbname}

The ``connection-user`` and ``connection-password`` are typically required and
determine the user credentials for the connection, often a service user.

Multiple Snowflake Databases or Accounts
^^^^^^^^^^^^^^^^^^^^^^

The Snowflake connector can only access a single database within a Snowflake account.
If you have multiple Snowflake databases, or want to connect to multiple
Snowflake accounts, you must configure multiple instances of the Snowflake connector.
steveburnett marked this conversation as resolved.
Show resolved Hide resolved
To add another catalog, add another properties file to ``etc/catalog``
with a different name (making sure it ends in ``.properties``). For
example, if you name the property file ``sales.properties``, Presto
will create a catalog named ``sales`` using the configured connector.

General Configuration Properties
---------------------------------

================================================== ==================================================================== ===========
Property Name Description Default
================================================== ==================================================================== ===========
``user-credential-name`` Name of the ``extraCredentials`` property whose value is the JDBC
driver's user name. See ``extraCredentials`` in `Parameter Reference
<https://prestodb.io/docs/current/installation/jdbc.html
#parameter-reference>`_.

``password-credential-name`` Name of the ``extraCredentials`` property whose value is the JDBC
driver's user password. See ``extraCredentials`` in `Parameter
Reference <https://prestodb.io/docs/current/installation/jdbc.html
#parameter-reference>`_.

``case-insensitive-name-matching`` Match dataset and table names case-insensitively. ``false``

``case-insensitive-name-matching.cache-ttl`` Duration for which remote dataset and table names will be
cached. Set to ``0ms`` to disable the cache. ``1m``
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a standard set of abbreviations for measurements of time in this context? This reads to me as both milliseconds and minutes are acceptable values. I searched but didn't find anything quickly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@steveburnett - Both units milliseconds and minutes are supported.

================================================== ==================================================================== ===========

SSL Configuration Properties
^^^^^^^^^^^^^^^^^^^^^^

By default, connections to Snowflake use SSL.

Querying Snowflake
steveburnett marked this conversation as resolved.
Show resolved Hide resolved
--------------

The Snowflake connector provides a schema for every Snowflake *database*.
You can see the available Snowflake schemas by running ``SHOW SCHEMAS``::

SHOW SCHEMAS FROM snowflake;

If you have a Snowflake schema named ``web``, you can view the tables
in this schema by running ``SHOW TABLES``::

SHOW TABLES FROM snowflake.web;

You can see a list of the columns in the ``clicks`` table in the ``web`` schema
using either of the following::

DESCRIBE snowflake.web.clicks;
SHOW COLUMNS FROM snowflake.web.clicks;

You can access the ``clicks`` table in the ``web`` schema::

SELECT * FROM snowflake.web.clicks;

If you used a different name for your catalog properties file, use
that catalog name instead of ``snowflake`` in the above examples.

Snowflake Connector Limitations
---------------------------

The following SQL statements are not supported:

* :doc:`/sql/create-schema`
* :doc:`/sql/alter-schema`
* :doc:`/sql/grant`
* :doc:`/sql/revoke`
* :doc:`/sql/show-roles`
* :doc:`/sql/show-role-grants`
* :doc:`/sql/create-role`
* :doc:`/sql/create-view`
* :doc:`/sql/drop-schema`
* :doc:`/sql/drop-view`
* :doc:`/sql/truncate`
* :doc:`/sql/update`
* :doc:`/sql/delete`
3 changes: 2 additions & 1 deletion presto-main/etc/config.properties
View file
Open in desktop
Expand Up @@ -51,7 +51,8 @@ plugin.bundles=\
../presto-node-ttl-fetchers/pom.xml,\
../presto-hive-function-namespace/pom.xml,\
../presto-delta/pom.xml,\
../presto-hudi/pom.xml
../presto-hudi/pom.xml,\
../presto-snowflake/pom.xml

presto.version=testversion
node-scheduler.include-coordinator=true
8 changes: 8 additions & 0 deletions presto-server/pom.xml
View file
Open in desktop
Expand Up @@ -380,6 +380,14 @@
<scope>provided</scope>
</dependency>

<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-snowflake</artifactId>
<version>${project.version}</version>
<type>zip</type>
<scope>provided</scope>
</dependency>

</dependencies>

<build>
Expand Down
4 changes: 4 additions & 0 deletions presto-server/src/main/assembly/presto.xml
View file
Open in desktop
Expand Up @@ -216,5 +216,9 @@
<directory>${project.build.directory}/dependency/presto-clickhouse-${project.version}</directory>
<outputDirectory>plugin/clickhouse</outputDirectory>
</fileSet>
<fileSet>
<directory>${project.build.directory}/dependency/presto-snowflake-${project.version}</directory>
<outputDirectory>plugin/snowflake</outputDirectory>
</fileSet>
</fileSets>
</assembly>
124 changes: 124 additions & 0 deletions presto-snowflake/pom.xml
View file
Open in desktop
@@ -0,0 +1,124 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.287-SNAPSHOT</version>
</parent>
<artifactId>presto-snowflake</artifactId>
<description>Presto - snowflake Connector</description>
<packaging>presto-plugin</packaging>

<properties>
<!--suppress UnresolvedMavenProperty -->
<air.main.basedir>${project.parent.basedir}</air.main.basedir>
<air.check.skip-enforcer>true</air.check.skip-enforcer>
<air.check.skip-dependency>true</air.check.skip-dependency>
</properties>

<dependencies>
<!-- Presto SPI -->
<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-spi</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-base-jdbc</artifactId>
</dependency>

<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-common</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>io.airlift</groupId>
<artifactId>slice</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>io.airlift</groupId>
<artifactId>units</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>net.snowflake</groupId>
<artifactId>snowflake-jdbc</artifactId>
<version>3.13.29</version>
</dependency>

<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>

<dependency>
<groupId>com.google.inject</groupId>
<artifactId>guice</artifactId>
</dependency>

<dependency>
<groupId>com.facebook.airlift</groupId>
<artifactId>configuration</artifactId>
</dependency>

<!-- for testing -->
<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-testng-services</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-main</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.airlift.tpch</groupId>
<artifactId>tpch</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-tests</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.basepom.maven</groupId>
<artifactId>duplicate-finder-maven-plugin</artifactId>
<configuration>
<ignoredClassPatterns>
<ignoredClassPattern>META-INF.versions.9.module-info</ignoredClassPattern>
<ignoredClassPattern> module-info</ignoredClassPattern>
</ignoredClassPatterns>
</configuration>
</plugin>
</plugins>
</build>
</project>
47 changes: 47 additions & 0 deletions presto-snowflake/src/main/java/com/facebook/presto/plugin/snowflake/SnowflakeClient.java
View file
Open in desktop
@@ -0,0 +1,47 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.facebook.presto.plugin.snowflake;

import com.facebook.presto.plugin.jdbc.BaseJdbcClient;
import com.facebook.presto.plugin.jdbc.BaseJdbcConfig;
import com.facebook.presto.plugin.jdbc.DriverConnectionFactory;
import com.facebook.presto.plugin.jdbc.JdbcConnectorId;
import com.facebook.presto.spi.ConnectorSession;
import net.snowflake.client.jdbc.SnowflakeDriver;

import javax.inject.Inject;

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;

public class SnowflakeClient
extends BaseJdbcClient
{
@Inject
public SnowflakeClient(JdbcConnectorId connectorId, BaseJdbcConfig config)
{
super(connectorId, config, "\"", new DriverConnectionFactory(new SnowflakeDriver(), config));
}

@Override
public PreparedStatement getPreparedStatement(ConnectorSession session, Connection connection, String sql)
throws SQLException
{
connection.setAutoCommit(false);
PreparedStatement statement = connection.prepareStatement(sql);
statement.setFetchSize(1000);
return statement;
}
}
33 changes: 33 additions & 0 deletions ...o-snowflake/src/main/java/com/facebook/presto/plugin/snowflake/SnowflakeClientModule.java
View file
Open in desktop
@@ -0,0 +1,33 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.facebook.presto.plugin.snowflake;

import com.facebook.presto.plugin.jdbc.BaseJdbcConfig;
import com.facebook.presto.plugin.jdbc.JdbcClient;
import com.google.inject.Binder;
import com.google.inject.Module;
import com.google.inject.Scopes;

import static com.facebook.airlift.configuration.ConfigBinder.configBinder;

public class SnowflakeClientModule
implements Module
{
@Override
public void configure(Binder binder)
{
binder.bind(JdbcClient.class).to(SnowflakeClient.class).in(Scopes.SINGLETON);
configBinder(binder).bindConfig(BaseJdbcConfig.class);
}
}
25 changes: 25 additions & 0 deletions presto-snowflake/src/main/java/com/facebook/presto/plugin/snowflake/SnowflakePlugin.java
View file
Open in desktop
@@ -0,0 +1,25 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.facebook.presto.plugin.snowflake;

import com.facebook.presto.plugin.jdbc.JdbcPlugin;

public class SnowflakePlugin
extends JdbcPlugin
{
public SnowflakePlugin()
{
super("snowflake", new SnowflakeClientModule());
}
}