upgrade to Gradle 7.3.1 / Java 17 (#7964)

* upgrade gradle * upgrade to Java 17 (and fix a few of the node versioning misses) * oops * try to run a different format version * fix spotless by upgrading / reformatting some files * fix ci settings * upgrade mockito to avoid other errors * undo bad format * fix "incorrect" sql comments * fmt * add debug flag * remove * bump * bump jooq to a version that has a java 17 dist * fix * remove logs * oops * revert jooq upgrade * fix * set up java for connector test * fix yaml * generate std source tests * fail zombie job attempts and add failure reason (#8709) * fail zombie job attempts and add failure reason * remove failure reason * bump gcp dependencies to pick up grpc update (#8713) * Bump Airbyte version from 0.33.9-alpha to 0.33.10-alpha (#8714) Co-authored-by: jrhizor <jrhizor@users.noreply.github.com> * Change CDK "Caching" header to "nested streams & caching" * Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable (#8524) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * move S3Config into destination-s3; update dependencies accordingly (#8562) Co-authored-by: Lake Mossman <lake@airbyte.io> Co-authored-by: jrhizor <jrhizor@users.noreply.github.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: Iryna Grankova <87977540+igrankova@users.noreply.github.com> Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com> Co-authored-by: Edward Gao <edward.gao@airbyte.io>
airbytehq · Dec 11, 2021 · 25674fc · 25674fc
1 parent e9f414d
commit 25674fc
Show file tree

Hide file tree

Showing 28 changed files with 227 additions and 185 deletions.
diff --git a/.github/workflows/gke-kube-test-command.yml b/.github/workflows/gke-kube-test-command.yml
@@ -46,11 +46,11 @@ jobs:
 
       - uses: actions/setup-java@v1
         with:
-          java-version: '14'
+          java-version: '17'
 
       - uses: actions/setup-node@v1
         with:
-          node-version: '14.7'
+          node-version: '16.13.0'
 
       - name: Fix EC-2 Runner
         run: |

diff --git a/.github/workflows/gradle.yml b/.github/workflows/gradle.yml
@@ -79,7 +79,7 @@ jobs:
 
       - uses: actions/setup-java@v1
         with:
-          java-version: "14"
+          java-version: "17"
 
       - uses: actions/setup-node@v1
         with:
@@ -96,7 +96,11 @@ jobs:
         run: |
           mkdir -p ~/.gradle/
           cat > ~/.gradle/gradle.properties <<EOF
-          org.gradle.jvmargs=-Xmx8g -Xss4m
+          org.gradle.jvmargs=-Xmx8g -Xss4m --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
           org.gradle.workers.max=8
           org.gradle.vfs.watch=false
           EOF
@@ -206,7 +210,7 @@ jobs:
 
       - uses: actions/setup-java@v1
         with:
-          java-version: "14"
+          java-version: "17"
 
       - uses: actions/setup-node@v1
         with:
@@ -216,7 +220,11 @@ jobs:
         run: |
           mkdir -p ~/.gradle/
           cat > ~/.gradle/gradle.properties <<EOF
-          org.gradle.jvmargs=-Xmx8g -Xss4m
+          org.gradle.jvmargs=-Xmx8g -Xss4m --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
           org.gradle.workers.max=8
           org.gradle.vfs.watch=false
           EOF
@@ -330,7 +338,7 @@ jobs:
 
       - uses: actions/setup-java@v1
         with:
-          java-version: "14"
+          java-version: "17"
 
       - uses: actions/setup-node@v1
         with:
@@ -343,7 +351,11 @@ jobs:
         run: |
           mkdir -p ~/.gradle/
           cat > ~/.gradle/gradle.properties <<EOF
-          org.gradle.jvmargs=-Xmx8g -Xss4m
+          org.gradle.jvmargs=-Xmx8g -Xss4m --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
           org.gradle.workers.max=8
           org.gradle.vfs.watch=false
           EOF
@@ -411,7 +423,7 @@ jobs:
 
       - uses: actions/setup-java@v1
         with:
-          java-version: "14"
+          java-version: "17"
 
       - uses: actions/setup-node@v1
         with:
@@ -425,7 +437,11 @@ jobs:
         run: |
           mkdir -p ~/.gradle/
           cat > ~/.gradle/gradle.properties <<EOF
-          org.gradle.jvmargs=-Xmx8g -Xss4m
+          org.gradle.jvmargs=-Xmx8g -Xss4m --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
+            --add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
           org.gradle.workers.max=8
           org.gradle.vfs.watch=false
           EOF

diff --git a/.github/workflows/publish-cdk-command.yml b/.github/workflows/publish-cdk-command.yml
@@ -23,7 +23,7 @@ jobs:
           python-version: ${{ matrix.python-version }}
       - uses: actions/setup-java@v1
         with:
-          java-version: '14'
+          java-version: '17'
       - name: Checkout Airbyte
         uses: actions/checkout@v2
       - name: Build CDK Package

diff --git a/.github/workflows/release-airbyte-os.yml b/.github/workflows/release-airbyte-os.yml
@@ -17,11 +17,11 @@ jobs:
           fetch-depth: 0
       - uses: actions/setup-java@v1
         with:
-          java-version: '14'
+          java-version: '17'
 
       - uses: actions/setup-node@v1
         with:
-          node-version: '14.7'
+          node-version: '16.13.0'
       - name: Save Old Version
         id: old_version
         run: |

diff --git a/.github/workflows/test-command.yml b/.github/workflows/test-command.yml
@@ -64,6 +64,9 @@ jobs:
       # Beside PyEnv, this does not set any runtimes up because it uses an AMI image that has everything pre-installed. See https://github.com/airbytehq/airbyte/issues/4559/
       - name: Install Pyenv
         run: python3 -m pip install virtualenv==16.7.9 --user
+      - uses: actions/setup-java@v1
+        with:
+          java-version: '17'
       - name: Write Integration Test Credentials
         run: ./tools/bin/ci_credentials.sh ${{ github.event.inputs.connector }}
         env:

diff --git a/airbyte-db/lib/src/main/resources/configs_database/schema.sql b/airbyte-db/lib/src/main/resources/configs_database/schema.sql
@@ -1,5 +1,5 @@
 -- tables
- CREATE
+CREATE
     TABLE
         IF NOT EXISTS AIRBYTE_CONFIGS(
             id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
@@ -11,7 +11,7 @@
         );
 
 -- indices
- CREATE
+CREATE
     UNIQUE INDEX IF NOT EXISTS airbyte_configs_type_id_idx ON
     AIRBYTE_CONFIGS(
         config_type,

diff --git a/airbyte-db/lib/src/main/resources/jobs_database/schema.sql b/airbyte-db/lib/src/main/resources/jobs_database/schema.sql
@@ -1,5 +1,5 @@
 -- types
- CREATE
+CREATE
     TYPE JOB_STATUS AS ENUM(
         'pending',
         'running',
@@ -27,7 +27,7 @@ CREATE
     );
 
 -- tables
- CREATE
+CREATE
     TABLE
         IF NOT EXISTS AIRBYTE_METADATA(
             KEY VARCHAR(255) PRIMARY KEY,

diff --git a/...a/io/airbyte/db/instance/configs/migrations/V0_30_22_001__Store_last_sync_state_test.java b/...a/io/airbyte/db/instance/configs/migrations/V0_30_22_001__Store_last_sync_state_test.java
@@ -158,7 +158,13 @@ public void testCopyData() throws SQLException {
             .withConnectionId(CONNECTION_2_ID)
             .withState(new State().withState(Jsons.deserialize("{ \"cursor\": 3 }"))));
 
-    final OffsetDateTime timestamp = OffsetDateTime.now();
+    final OffsetDateTime timestampWithFullPrecision = OffsetDateTime.now();
+    /*
+     * The AWS CI machines get a higher precision value here (2021-12-07T19:56:28.967213187Z) vs what is
+     * retrievable on Postgres or on my local machine (2021-12-07T19:56:28.967213Z). Truncating the
+     * value to match.
+     */
+    final OffsetDateTime timestamp = timestampWithFullPrecision.withNano(1000 * (timestampWithFullPrecision.getNano() / 1000));
 
     database.query(ctx -> {
       V0_30_22_001__Store_last_sync_state.copyData(ctx, STD_CONNECTION_STATES, timestamp);

diff --git a/airbyte-integrations/bases/base-java/Dockerfile b/airbyte-integrations/bases/base-java/Dockerfile
@@ -1,4 +1,4 @@
-ARG JDK_VERSION=14.0.2
+ARG JDK_VERSION=17.0.1
 FROM openjdk:${JDK_VERSION}-slim
 COPY --from=airbyte/integration-base:dev /airbyte /airbyte
 

diff --git a/airbyte-integrations/bases/base-standard-source-test-file/Dockerfile b/airbyte-integrations/bases/base-standard-source-test-file/Dockerfile
@@ -1,4 +1,4 @@
-ARG JDK_VERSION=14.0.2
+ARG JDK_VERSION=17.0.1
 FROM openjdk:${JDK_VERSION}-slim
 
 # Install Docker to launch worker images. Eventually should be replaced with Docker-java.

diff --git a/airbyte-integrations/bases/standard-source-test/Dockerfile b/airbyte-integrations/bases/standard-source-test/Dockerfile
@@ -1,4 +1,4 @@
-ARG JDK_VERSION=14.0.2
+ARG JDK_VERSION=17.0.1
 FROM openjdk:${JDK_VERSION}-slim
 
 # Install Docker to launch worker images. Eventually should be replaced with Docker-java.

diff --git a/airbyte-integrations/connectors/source-jdbc/build.gradle b/airbyte-integrations/connectors/source-jdbc/build.gradle
@@ -47,7 +47,7 @@ dependencies {
     testFixturesRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.4.2'
     testFixturesImplementation 'org.junit.jupiter:junit-jupiter-api:5.4.2'
     testFixturesImplementation 'org.junit.jupiter:junit-jupiter-params:5.4.2'
-    testFixturesImplementation group: 'org.mockito', name: 'mockito-junit-jupiter', version: '3.4.6'
+    testFixturesImplementation group: 'org.mockito', name: 'mockito-junit-jupiter', version: '4.0.0'
 
     implementation files(project(':airbyte-integrations:bases:base-java').airbyteDocker.outputs)
 }
diff --git a/airbyte-integrations/infrastructure/ssh_tunnel/module/sql/postgresql-01-dbcreate.sql b/airbyte-integrations/infrastructure/ssh_tunnel/module/sql/postgresql-01-dbcreate.sql
@@ -1,5 +1,5 @@
 -- generic setup for a brand new empty postgresql RDS
- CREATE
+CREATE
     ROLE integrationtest_rw;
 
 GRANT integrationtest_rw TO airbyte;

diff --git a/airbyte-integrations/infrastructure/ssh_tunnel/module/sql/postgresql-02-user-create.sql b/airbyte-integrations/infrastructure/ssh_tunnel/module/sql/postgresql-02-user-create.sql
@@ -1,5 +1,5 @@
 -- create the specific user we want to use from airbyte
- CREATE
+CREATE
     USER testcaseuser WITH password 'ThisIsNotTheRealPassword.PleaseSetThisByHand';
 
 GRANT integrationtest_rw TO testcaseuser;
diff --git a/airbyte-scheduler/app/Dockerfile b/airbyte-scheduler/app/Dockerfile
@@ -1,4 +1,4 @@
-ARG JDK_VERSION=14.0.2
+ARG JDK_VERSION=17.0.1
 FROM openjdk:${JDK_VERSION}-slim AS scheduler
 
 ENV APPLICATION airbyte-scheduler

diff --git a/airbyte-scheduler/persistence/src/main/resources/job_history_purge.sql b/airbyte-scheduler/persistence/src/main/resources/job_history_purge.sql
@@ -18,8 +18,8 @@ WHERE
             ) counts ON
             jobs.scope = counts.scope
         WHERE
-            -- job must be at least MINIMUM_AGE_IN_DAYS old or connection has more than EXCESSIVE_NUMBER_OF_JOBS 
-(
+            /* job must be at least MINIMUM_AGE_IN_DAYS old or connection has more than EXCESSIVE_NUMBER_OF_JOBS */
+            (
                 jobs.created_at <(
                     TO_TIMESTAMP(
                         ?,
@@ -29,8 +29,8 @@ WHERE
                 OR counts.jobCount >?
             )
             AND jobs.id NOT IN(
-                -- cannot be the most recent job with saved state 
- SELECT
+                /* cannot be the most recent job with saved state */
+                SELECT
                     job_id AS latest_job_id_with_state
                 FROM
                     (
@@ -69,8 +69,8 @@ WHERE
                     stateRecency = 1
             )
             AND jobs.id NOT IN(
-                -- cannot be one of the last MINIMUM_RECENCY jobs for that connection/scope
- SELECT
+                /* cannot be one of the last MINIMUM_RECENCY jobs for that connection/scope */
+                SELECT
                     id
                 FROM
                     (

diff --git a/airbyte-server/Dockerfile b/airbyte-server/Dockerfile
@@ -1,4 +1,4 @@
-ARG JDK_VERSION=14.0.2
+ARG JDK_VERSION=17.0.1
 FROM openjdk:${JDK_VERSION}-slim AS server
 
 EXPOSE 8000

diff --git a/airbyte-server/src/test/resources/migration/schema.sql b/airbyte-server/src/test/resources/migration/schema.sql
@@ -1,9 +1,9 @@
 -- extensions
- CREATE
+CREATE
     EXTENSION IF NOT EXISTS "uuid-ossp";
 
 -- types
- CREATE
+CREATE
     TYPE JOB_STATUS AS ENUM(
         'pending',
         'running',
@@ -31,7 +31,7 @@ CREATE
     );
 
 -- tables
- CREATE
+CREATE
     TABLE
         AIRBYTE_METADATA(
             KEY VARCHAR(255) PRIMARY KEY,
@@ -73,7 +73,7 @@ CREATE
     );
 
 -- entries
- INSERT
+INSERT
     INTO
         AIRBYTE_METADATA
     VALUES(

diff --git a/airbyte-workers/Dockerfile b/airbyte-workers/Dockerfile
@@ -1,4 +1,4 @@
-ARG JDK_VERSION=14.0.2
+ARG JDK_VERSION=17.0.1
 FROM openjdk:${JDK_VERSION}-slim AS worker
 
 ARG DOCKER_BUILD_ARCH=amd64

diff --git a/airbyte-workers/build.gradle b/airbyte-workers/build.gradle
@@ -31,7 +31,7 @@ dependencies {
     implementation project(':airbyte-protocol:models')
     implementation project(':airbyte-scheduler:persistence')
 
-    testImplementation 'org.mockito:mockito-inline:2.13.0'
+    testImplementation 'org.mockito:mockito-inline:4.0.0'
     testImplementation 'org.postgresql:postgresql:42.2.18'
     testImplementation "org.flywaydb:flyway-core:7.14.0"
     testImplementation 'io.temporal:temporal-testing:1.6.0'

diff --git a/build.gradle b/build.gradle
@@ -14,7 +14,7 @@ buildscript {
 plugins {
     id 'base'
     id 'pmd'
-    id 'com.diffplug.spotless' version '5.7.0'
+    id 'com.diffplug.spotless' version '6.0.0'
     id 'com.github.hierynomus.license' version '0.16.1'
 }
 
@@ -103,7 +103,7 @@ spotless {
 
         importOrder()
 
-        eclipse('4.16.0').configFile(rootProject.file('tools/gradle/codestyle/java-google-style.xml'))
+        eclipse('4.21.0').configFile(rootProject.file('tools/gradle/codestyle/java-google-style.xml'))
 
         licenseHeaderFile createJavaLicenseWith(rootProject.file('LICENSE_SHORT'))
         removeUnusedImports()
@@ -132,7 +132,7 @@ def Task getDockerBuildTask(String artifactName, String projectDir) {
         def buildPlatform = System.getenv('DOCKER_BUILD_PLATFORM') ?: 'linux/amd64'
         def alpineImage = System.getenv('ALPINE_IMAGE') ?: 'alpine:3.4'
         def postgresImage = System.getenv('POSTGRES_IMAGE') ?: 'postgres:13-alpine'
-        def jdkVersion = System.getenv('JDK_VERSION') ?: '14.0.2'
+        def jdkVersion = System.getenv('JDK_VERSION') ?: '17.0.1'
         def buildArch = System.getenv('DOCKER_BUILD_ARCH') ?: 'amd64'
 
         inputDir = file("$projectDir/build/docker")
@@ -178,8 +178,8 @@ subprojects {
     apply plugin: 'java'
     apply plugin: 'jacoco'
 
-    sourceCompatibility = JavaVersion.VERSION_14
-    targetCompatibility = JavaVersion.VERSION_14
+    sourceCompatibility = JavaVersion.VERSION_17
+    targetCompatibility = JavaVersion.VERSION_17
 
     repositories {
         mavenCentral()
@@ -300,7 +300,7 @@ subprojects {
         testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.7.2'
         testImplementation 'org.junit.jupiter:junit-jupiter-api:5.7.2'
         testImplementation 'org.junit.jupiter:junit-jupiter-params:5.7.2'
-        testImplementation 'org.mockito:mockito-junit-jupiter:3.12.4'
+        testImplementation 'org.mockito:mockito-junit-jupiter:4.0.0'
         testImplementation 'org.assertj:assertj-core:3.21.0'
 
     }

diff --git a/docs/connector-development/testing-connectors/standard-source-tests.md b/docs/connector-development/testing-connectors/standard-source-tests.md
@@ -2,35 +2,3 @@
 
 Test methods start with `test`. Other methods are internal helpers in the java class implementing the test suite.
 
-## testGetSpec
-
-Verify that a spec operation issued to the connector returns a valid spec.
-
-## testCheckConnection
-
-Verify that a check operation issued to the connector with the input config file returns a success response.
-
-## testDiscover
-
-Verifies when a discover operation is run on the connector using the given config file, a valid catalog is output by the connector.
-
-## testFullRefreshRead
-
-Configuring all streams in the input catalog to full refresh mode, verifies that a read operation produces some RECORD messages.
-
-## testIdenticalFullRefreshes
-
-Configuring all streams in the input catalog to full refresh mode, performs two read operations on all streams which support full refresh syncs. It then verifies that the RECORD messages output from both were identical.
-
-## testIncrementalSyncWithState
-
-This test verifies that all streams in the input catalog which support incremental sync can do so correctly. It does this by running two read operations on the connector's Docker image: the first takes the configured catalog and config provided to this test as input. It then verifies that the sync produced a non-zero number of RECORD and STATE messages. The second read takes the same catalog and config used in the first test, plus the last STATE message output by the first read operation as the input state file. It verifies that no records are produced \(since we read all records in the first sync\). This test is performed only for streams which support incremental. Streams which do not support incremental sync are ignored. If no streams in the input catalog support incremental sync, this test is skipped.
-
-## testEmptyStateIncrementalIdenticalToFullRefresh
-
-If the source does not support incremental sync, this test is skipped. Otherwise, this test runs two syncs: one where all streams provided in the input catalog sync in full refresh mode, and another where all the streams which in the input catalog which support incremental, sync in incremental mode \(streams which don't support incremental sync in full refresh mode\). Then, the test asserts that the two syncs produced the same RECORD messages. Any other type of message is disregarded.
-
-## testEntrypointEnvVar
-
-In order to launch a source on Kubernetes in a pod, we need to be able to wrap the entrypoint. The source connector must specify its entrypoint in the AIRBYTE_ENTRYPOINT variable. This test ensures that the entrypoint environment variable is set.
-