🎉 Destination databricks: move driver downloading to gradle (#9995)

* Download jdbc driver in dockerfile * Download jdbc driver in dockerfile * Remove driver downloading from ci * Update document * Update public doc * Download jdbc driver in gradle * Clean up gradle file * Clean up dockerfile * Fix typo
airbytehq · Feb 2, 2022 · 679ce18 · 679ce18
1 parent a6d89ce
commit 679ce18
Show file tree

Hide file tree

Showing 9 changed files with 41 additions and 48 deletions.
diff --git a/.github/workflows/publish-command.yml b/.github/workflows/publish-command.yml
@@ -67,10 +67,6 @@ jobs:
         uses: actions/checkout@v2
         with:
           repository: ${{github.event.pull_request.head.repo.full_name}} # always use the branch's repository
-      - name: Install Unzip for Databricks
-        if: github.event.inputs.connector == 'connectors/destination-databricks'
-        run: |
-          apt-get update && apt-get install -y unzip
       - name: Install Java
         uses: actions/setup-java@v1
         with:

diff --git a/.github/workflows/test-command.yml b/.github/workflows/test-command.yml
@@ -62,10 +62,6 @@ jobs:
         uses: actions/checkout@v2
         with:
           repository: ${{ github.event.inputs.repo }}
-      - name: Install Unzip for Databricks
-        if: github.event.inputs.connector == 'connectors/destination-databricks'
-        run: |
-          apt-get update && apt-get install -y unzip
       - name: Install Java
         uses: actions/setup-java@v1
         with:

diff --git a/airbyte-integrations/connectors/destination-databricks/.gitignore b/airbyte-integrations/connectors/destination-databricks/.gitignore
@@ -3,4 +3,4 @@
 # https://databricks.com/spark/jdbc-drivers-download
 # By downloading this driver, you agree to the terms & conditions:
 # https://databricks.com/jdbc-odbc-driver-license
-lib/SparkJDBC42.jar
+lib/*
diff --git a/airbyte-integrations/connectors/destination-databricks/README.md b/airbyte-integrations/connectors/destination-databricks/README.md
@@ -4,11 +4,16 @@ This is the repository for the Databricks destination connector in Java.
 For information about how to use this connector within Airbyte, see [the User Documentation](https://docs.airbyte.io/integrations/destinations/databricks).
 
 ## Databricks JDBC Driver
-This connector requires a JDBC driver to connect to Databricks cluster. The driver is developed by Simba. Before downloading and using this driver, you must agree to the [JDBC ODBC driver license](https://databricks.com/jdbc-odbc-driver-license). This means that you can only use this driver to connector third party applications to Apache Spark SQL within a Databricks offering using the ODBC and/or JDBC protocols. The driver can be downloaded from [here](https://databricks.com/spark/jdbc-drivers-download).
+This connector requires a JDBC driver to connect to Databricks cluster. The driver is developed by Simba.
 
-The CI downloads the JDBC driver in [this script](https://github.com/airbytehq/airbyte/blob/master/tools/lib/databricks.sh).
+{% hint style="warning" %}
+Before building, or using this connector, you must agree to the [JDBC ODBC driver license](https://databricks.com/jdbc-odbc-driver-license). This means that you can only use this driver to connector third party applications to Apache Spark SQL within a Databricks offering using the ODBC and/or JDBC protocols.
+{% endhint %}
 
-This is currently a private connector that is only available in Airbyte Cloud. To build and publish this connector, first download the driver and put it under the `lib` directory. Please do not publish this connector publicly.  We are working on a solution to publicize it.
+This is currently a private connector that is only available on Airbyte Cloud. We are working on a solution to publicize it (issue [\#6043](https://github.com/airbytehq/airbyte/issues/6043)).
+
+- If you want to use this connector now, you can build the connector locally, and publish it to your own docker registry. See the "[Build](#build)" or "[Building via Gradle](#building-via-gradle)" sections below for details. Please do not publish this connector publicly.
+- If you want to work on the connector code, first build the connector. The building process will automatically download the driver to the [`lib`](./lib) directory. You can also manually download the driver from [here](https://databricks.com/spark/jdbc-drivers-download).
 
 ## Local development
 

diff --git a/airbyte-integrations/connectors/destination-databricks/build.gradle b/airbyte-integrations/connectors/destination-databricks/build.gradle
@@ -1,7 +1,22 @@
+// #############################################################################################
+// #                                                                                           #
+// #    _______________ _____  ______      _                  _     _                          #
+// #   |_  |  _  | ___ /  __ \ |  _  \    (_)                | |   (_)                         #
+// #     | | | | | |_/ | /  \/ | | | |_ __ ___   _____ _ __  | |    _  ___ ___ _ __  ___  ___  #
+// #     | | | | | ___ | |     | | | | '__| \ \ / / _ | '__| | |   | |/ __/ _ | '_ \/ __|/ _ \ #
+// # /\__/ | |/ /| |_/ | \__/\ | |/ /| |  | |\ V |  __| |    | |___| | (_|  __| | | \__ |  __/ #
+// # \____/|___/ \____/ \____/ |___/ |_|  |_| \_/ \___|_|    \_____|_|\___\___|_| |_|___/\___| #
+// #                                                                                           #
+// #           By building this connector, you agree to the JDBC ODBC driver license:          #
+// #                  https://databricks.com/jdbc-odbc-driver-license                          #
+// #                                                                                           #
+// #############################################################################################
+
 plugins {
     id 'application'
     id 'airbyte-docker'
     id 'airbyte-integration-test-java'
+    id "de.undercouch.download" version "5.0.1"
 }
 
 application {
@@ -33,3 +48,19 @@ dependencies {
     integrationTestJavaImplementation project(':airbyte-integrations:bases:standard-destination-test')
     integrationTestJavaImplementation project(':airbyte-integrations:connectors:destination-databricks')
 }
+
+task downloadJdbcDriverZip(type: Download) {
+    src 'https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/jdbc/2.6.21/SimbaSparkJDBC42-2.6.21.1039.zip'
+    dest new File(buildDir, 'SimbaSparkJDBC42-2.6.21.1039.zip')
+    overwrite false
+}
+
+task extractJdbcDriverFile(dependsOn: downloadJdbcDriverZip, type: Copy) {
+    from {
+        zipTree(downloadJdbcDriverZip.dest)
+    }
+    into 'lib/'
+    include 'SparkJDBC42.jar'
+}
+
+compileJava.dependsOn tasks.extractJdbcDriverFile
diff --git a/docs/integrations/destinations/databricks.md b/docs/integrations/destinations/databricks.md
@@ -6,7 +6,7 @@ This destination syncs data to Databricks cluster. Each stream is written to its
 
 This connector requires a JDBC driver to connect to Databricks cluster. The driver is developed by Simba. Before using the driver and the connector, you must agree to the [JDBC ODBC driver license](https://databricks.com/jdbc-odbc-driver-license). This means that you can only use this connector to connector third party applications to Apache Spark SQL within a Databricks offering using the ODBC and/or JDBC protocols.
 
-Due to legal reasons, this is currently a private connector that is only available in Airbyte Cloud. We are working on publicizing it. Please follow [this issue](https://github.com/airbytehq/airbyte/issues/6043) for progress.
+Due to legal reasons, this is currently a private connector that is only available on Airbyte Cloud. We are working on publicizing it. Please follow [this issue](https://github.com/airbytehq/airbyte/issues/6043) for progress. In the interim, you can build the connector locally, publish it to your own image registry, and use it privately. See the [developer doc](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/destination-databricks/README.md) for details. Please do not publish this connector publicly.
 
 ## Sync Mode
 

diff --git a/tools/bin/ci_integration_test.sh b/tools/bin/ci_integration_test.sh
@@ -3,15 +3,13 @@
 set -e
 
 . tools/lib/lib.sh
-. tools/lib/databricks.sh
 
 # runs integration tests for an integration name
 
 connector="$1"
 all_integration_tests=$(./gradlew integrationTest --dry-run | grep 'integrationTest SKIPPED' | cut -d: -f 4)
 run() {
 if [[ "$connector" == "all" ]] ; then
-  _get_databricks_jdbc_driver
   echo "Running: ./gradlew --no-daemon --scan integrationTest"
   ./gradlew --no-daemon --scan integrationTest
 else
@@ -36,10 +34,6 @@ else
     integrationTestCommand=":airbyte-integrations:connectors:$connector:integrationTest"
   fi
   if [ -n "$selected_integration_test" ] ; then
-    if [[ "$selected_integration_test" == *"databricks"* ]] ; then
-      _get_databricks_jdbc_driver
-    fi
-
     echo "Running: ./gradlew --no-daemon --scan $integrationTestCommand"
     ./gradlew --no-daemon --scan "$integrationTestCommand"
   else

diff --git a/tools/integrations/manage.sh b/tools/integrations/manage.sh
@@ -4,7 +4,6 @@ set -e
 set -x
 
 . tools/lib/lib.sh
-. tools/lib/databricks.sh
 
 USAGE="
 Usage: $(basename "$0") <cmd>
@@ -40,10 +39,6 @@ cmd_build() {
 
   local run_tests=$1; shift || run_tests=true
 
-  if [[ "airbyte-integrations/connectors/destination-databricks" == "${path}" ]]; then
-    _get_databricks_jdbc_driver
-  fi
-
   echo "Building $path"
   ./gradlew --no-daemon "$(_to_gradle_path "$path" clean)"
   ./gradlew --no-daemon "$(_to_gradle_path "$path" build)"

diff --git a/tools/lib/databricks.sh b/tools/lib/databricks.sh