2.1.6: Spark 3.5 support

Summary

Added support of spark 3.5.

Features

• #221: Add support for spark 3.5
• #219: Updated dependency to fix CVE-2023-52428

2.1.5: Fix CVE-2024-21634 and CVE-2023-33546

Summary

This release fixes CVE-2024-21634 in transitive test dependency software.amazon.ion:ion-java.

Security

• #214: Fixed CVE-2024-21634 in software.amazon.ion:ion-java

Dependency Updates

Spark Exasol Connector Parent POM

Plugin Dependency Updates

• Updated org.apache.maven.plugins:maven-surefire-plugin:3.1.2 to 3.2.3
• Added org.apache.maven.plugins:maven-toolchains-plugin:3.1.0
• Updated org.codehaus.mojo:versions-maven-plugin:2.16.1 to 2.16.2

Spark Exasol Connector With JDBC

Plugin Dependency Updates

• Added org.apache.maven.plugins:maven-toolchains-plugin:3.1.0

Spark Exasol Connector With S3

Test Dependency Updates

• Updated com.amazonaws:aws-java-sdk-s3:1.12.518 to 1.12.639

Plugin Dependency Updates

• Added org.apache.maven.plugins:maven-toolchains-plugin:3.1.0

2.1.4: PK back in config, CVE fixed in transitive deps.

Summary

Project keeper added to Maven configuration.
Vulnerability fixed in transitive dependency.

Features

• #211: Brought PK plugin back to the Maven config
• #212: Fix vulnerability in org.eclipse.parsson:parsson

Dependency Updates

Spark Exasol Connector Parent POM

Plugin Dependency Updates

• Updated com.exasol:error-code-crawler-maven-plugin:1.3.0 to 1.3.1
• Updated org.apache.maven.plugins:maven-enforcer-plugin:3.4.0 to 3.4.1
• Updated org.codehaus.mojo:versions-maven-plugin:2.16.0 to 2.16.1
• Updated org.jacoco:jacoco-maven-plugin:0.8.10 to 0.8.11
• Updated org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184 to 3.10.0.2594

Spark Exasol Connector With JDBC

Test Dependency Updates

• Updated com.exasol:exasol-testcontainers:6.6.2 to 6.6.3

Spark Exasol Connector With S3

Test Dependency Updates

• Updated com.exasol:exasol-testcontainers:6.6.2 to 6.6.3

2.1.3: More flexibility for AWS Credentials specification in spark-connector-s3

Summary

In addition to explicit AWS Credentials specification we now support environment variables and EC2 instance profiles.
Fixes CVE-2023-39410 in apache avro (transitive dependency).

Features

• 192: Add support for AWS IAM Profile Credentials for s3 connector.

Dependency Updates

Spark Exasol Connector With JDBC

Compile Dependency Updates

• Added org.apache.avro:avro:1.11.3

Spark Exasol Connector With S3

Compile Dependency Updates

• Added org.apache.avro:avro:1.11.3

Test Dependency Updates

• Added org.junit-pioneer:junit-pioneer:2.1.0

2.1.2: Transaction commit fixes, security fixes in dependencies

Summary

Fixes CVE-2023-42503 in org.apache.commons:commons-compress.
Fixes logic in transaction commit during the DB import.

Features

• 194: Wrong name of Exasol JDBC format in documentation
• 197: Committing transaction in the finally handler
• 198: Remove unused Statement variable
• 206: Fix CVE-2023-42503 and update dependencies

Dependency Updates

Spark Exasol Connector Parent POM

Plugin Dependency Updates

• Updated org.apache.maven.plugins:maven-enforcer-plugin:3.3.0 to 3.4.0

Spark Exasol Connector With JDBC

Compile Dependency Updates

• Updated com.exasol:spark-connector-common-java:2.0.0 to 2.0.1

Test Dependency Updates

• Updated com.exasol:exasol-testcontainers:6.6.1 to 6.6.2
• Updated com.exasol:hamcrest-resultset-matcher:1.6.0 to 1.6.1
• Updated com.exasol:test-db-builder-java:3.4.2 to 3.5.1
• Updated org.mockito:mockito-core:5.4.0 to 5.5.0
• Updated org.mockito:mockito-junit-jupiter:5.4.0 to 5.5.0
• Updated org.scalatestplus:scalatestplus-mockito_2.13:1.0.0-M2 to 1.0.0-SNAP5
• Updated org.scalatest:scalatest_2.13:3.2.9 to 3.3.0-SNAP4

Spark Exasol Connector With S3

Compile Dependency Updates

• Updated com.exasol:spark-connector-common-java:2.0.0 to 2.0.1
• Updated org.apache.hadoop:hadoop-aws:3.3.4 to 3.3.6

Test Dependency Updates

• Updated com.exasol:exasol-testcontainers:6.6.1 to 6.6.2
• Updated com.exasol:hamcrest-resultset-matcher:1.6.0 to 1.6.1
• Updated com.exasol:test-db-builder-java:3.4.2 to 3.5.1
• Updated org.mockito:mockito-junit-jupiter:5.4.0 to 5.5.0
• Updated org.testcontainers:junit-jupiter:1.18.3 to 1.19.0
• Updated org.testcontainers:localstack:1.18.3 to 1.19.0

2.1.1: JDBC connection cache drop, `magic` s3 hadoop committer

Summary

S3 connector uses faster hadoop committer, fix possible race condition in RDD operations

Features

• #163: Switched to magic hadoop committer for intermediate s3 files
• #200: Dropped JDBC connection cache in ExasolConnectionManager

2.1.0: Added filter pushdown and column selection for `S3` variant

Summary

This release adds predicate pushdown and column selection for the S3 variant. Similarly, uses the latest API changes from the spark-connector-common-java library.

Features

• #190: Added predicate pushdown and column selection for S3 variant

Dependency Updates

Spark Exasol Connector With JDBC

Compile Dependency Updates

• Updated com.exasol:spark-connector-common-java:1.1.1 to 2.0.0

Plugin Dependency Updates

• Updated com.diffplug.spotless:spotless-maven-plugin:2.37.0 to 2.38.0

Spark Exasol Connector With S3

Compile Dependency Updates

• Updated com.exasol:spark-connector-common-java:1.1.1 to 2.0.0
• Updated software.amazon.awssdk:s3:2.20.103 to 2.20.115

Test Dependency Updates

• Updated com.amazonaws:aws-java-sdk-s3:1.12.506 to 1.12.518
• Added com.exasol:java-util-logging-testing:2.0.3
• Updated org.junit.jupiter:junit-jupiter-api:5.9.3 to 5.10.0
• Updated org.junit.jupiter:junit-jupiter:5.9.3 to 5.10.0

2.0.0: Support S3 intermediate storage

Summary

In this release we added support to use AWS S3 bucket as an intermediate storage layer when accessing Exasol database from Spark cluster.

With this release, we separated the connector into two variants, S3 and JDBC. We recommend to use this new S3 variant instead of JDBC variant. It improves the stability of the connector.

Please refer to the user guide for updated API changes and usage instructions.

Features

• #149: Added s3 import query generator and runner
• #150: Added S3 intermediate storage layer
• #159: Added cleanup process to remove intermediate data after job finish
• #160: Add support for writing to Exasol database using S3 as intermediate storage
• #168: Refactored to add module setup

Security

• #151: Fixed vulnerability CVE-2023-26048 coming with jetty-util transitive dependency

Bugs

• #176: Fixed artifact upload of sha256sum files
• #188: Fixed Maven Central deployment bug

Refactoring

• #155: Unified user options
• #158: Refactored common options class
• #164: Validated that write directory is empty
• #171: Refactored artifact packaging and releasing for module setup
• #174: Refactored Github .github/workflow/ action files
• #183: Updated user guide and prepared for release

Dependency Updates

Spark Exasol Connector Parent POM

Plugin Dependency Updates

• Added com.exasol:error-code-crawler-maven-plugin:1.3.0
• Added io.github.zlika:reproducible-build-maven-plugin:0.16
• Added org.apache.maven.plugins:maven-clean-plugin:2.5
• Added org.apache.maven.plugins:maven-compiler-plugin:3.11.0
• Added org.apache.maven.plugins:maven-deploy-plugin:3.1.1
• Added org.apache.maven.plugins:maven-enforcer-plugin:3.3.0
• Added org.apache.maven.plugins:maven-install-plugin:2.4
• Added org.apache.maven.plugins:maven-site-plugin:3.3
• Added org.apache.maven.plugins:maven-surefire-plugin:3.1.2
• Added org.basepom.maven:duplicate-finder-maven-plugin:2.0.1
• Added org.codehaus.mojo:flatten-maven-plugin:1.5.0
• Added org.codehaus.mojo:versions-maven-plugin:2.16.0
• Added org.itsallcode:openfasttrace-maven-plugin:1.6.2
• Added org.jacoco:jacoco-maven-plugin:0.8.10
• Added org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184
• Added org.sonatype.ossindex.maven:ossindex-maven-plugin:3.2.0

Spark Exasol Connector With JDBC

Compile Dependency Updates

• Added com.exasol:error-reporting-java8:1.0.1
• Added com.exasol:exasol-jdbc:7.1.20
• Added com.exasol:spark-connector-common-java:1.1.1
• Added com.exasol:sql-statement-builder-java8:4.5.4

Test Dependency Updates

• Added com.exasol:exasol-testcontainers:6.6.1
• Added com.exasol:hamcrest-resultset-matcher:1.6.0
• Added com.exasol:test-db-builder-java:3.4.2
• Added org.apache.logging.log4j:log4j-1.2-api:2.20.0
• Added org.apache.logging.log4j:log4j-api:2.20.0
• Added org.apache.logging.log4j:log4j-core:2.20.0
• Added org.mockito:mockito-core:5.4.0
• Added org.mockito:mockito-junit-jupiter:5.4.0
• Added org.scalatestplus:scalatestplus-mockito_2.13:1.0.0-M2
• Added org.scalatest:scalatest_2.13:3.2.9

Plugin Dependency Updates

• Added com.diffplug.spotless:spotless-maven-plugin:2.37.0
• Added com.exasol:error-code-crawler-maven-plugin:1.3.0
• Added io.github.evis:scalafix-maven-plugin_2.13:0.1.4_0.9.31
• Added io.github.zlika:reproducible-build-maven-plugin:0.16
• Added net.alchim31.maven:scala-maven-plugin:4.8.1
• Added org.apache.maven.plugins:maven-clean-plugin:2.5
• Added org.apache.maven.plugins:maven-compiler-plugin:3.11.0
• Added org.apache.maven.plugins:maven-deploy-plugin:3.1.1
• Added org.apache.maven.plugins:maven-enforcer-plugin:3.3.0
• Added org.apache.maven.plugins:maven-failsafe-plugin:3.1.2
• Added org.apache.maven.plugins:maven-gpg-plugin:3.1.0
• Added org.apache.maven.plugins:maven-install-plugin:2.4
• Added org.apache.maven.plugins:maven-jar-plugin:3.3.0
• Added org.apache.maven.plugins:maven-javadoc-plugin:3.5.0
• Added org.apache.maven.plugins:maven-resources-plugin:2.6
• Added org.apache.maven.plugins:maven-shade-plugin:3.4.1
• Added org.apache.maven.plugins:maven-site-plugin:3.3
• Added org.apache.maven.plugins:maven-source-plugin:3.2.1
• Added org.apache.maven.plugins:maven-surefire-plugin:3.1.2
• Added org.basepom.maven:duplicate-finder-maven-plugin:2.0.1
• Added org.codehaus.mojo:flatten-maven-plugin:1.5.0
• Added org.codehaus.mojo:versions-maven-plugin:2.16.0
• Added org.itsallcode:openfasttrace-maven-plugin:1.6.2
• Added org.jacoco:jacoco-maven-plugin:0.8.10
• Added org.scalatest:scalatest-maven-plugin:2.2.0
• Added org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184
• Added org.sonatype.ossindex.maven:ossindex-maven-plugin:3.2.0
• Added org.sonatype.plugins:nexus-staging-maven-plugin:1.6.13

Spark Exasol Connector With S3

Compile Dependency Updates

• Added com.exasol:spark-connector-common-java:1.1.1
• Added org.apache.hadoop:hadoop-aws:3.3.4
• Added org.scala-lang:scala-library:2.13.11
• Added org.wildfly.openssl:wildfly-openssl:2.2.5.Final
• Added software.amazon.awssdk:s3:2.20.103

Test Dependency Updates

• Added com.amazonaws:aws-java-sdk-s3:1.12.506
• Added com.exasol:exasol-testcontainers:6.6.1
• Added com.exasol:hamcrest-resultset-matcher:1.6.0
• Added com.exasol:test-db-builder-java:3.4.2
• Added org.junit.jupiter:junit-jupiter-api:5.9.3
• Added org.junit.jupiter:junit-jupiter:5.9.3
• Added org.mockito:mockito-junit-jupiter:5.4.0
• Added org.testcontainers:junit-jupiter:1.18.3
• Added org.testcontainers:localstack:1.18.3

Plugin Dependency Updates

• Added com.exasol:error-code-crawler-maven-plugin:1.3.0
• Added io.github.zlika:reproducible-build-maven-plugin:0.16
• Added org.apache.maven.plugins:maven-clean-plugin:2.5
• Added org.apache.maven.plugins:maven-compiler-plugin:3.11.0
• Added org.apache.maven.plugins:maven-deploy-plugin:3.1.1
• Added org.apache.maven.plugins:maven-enforcer-plugin:3.3.0
• Added org.apache.maven.plugins:maven-failsafe-plugin:3.1.2
• Added org.apache.maven.plugins:maven-gpg-plugin:3.1.0
• Added org.apache.maven.plugins:maven-install-plugin:2.4
• Added org.apache.maven.plugins:maven-jar-plugin:2.4
• Added org.apache.maven.plugins:maven-javadoc-plugin:3.5.0
• Added org.apache.maven.plugins:maven-resources-plugin:2.6
• Added org.apache.maven.plugins:maven-shade-plugin:3.4.1
• Added org.apache.maven.plugins:maven-site-plugin:3.3
• Added org.apache.maven.plugins:maven-source-plugin:3.2.1
• Added org.apache.maven.plugins:maven-surefire-plugin:3.1.2
• Added org.basepom.maven:duplicate-finder-maven-plugin:2.0.1
• Added org.codehaus.mojo:flatten-maven-plugin:1.5.0
• Added org.codehaus.mojo:versions-maven-plugin:2.16.0
• Added org.itsallcode:openfasttrace-maven-plugin:1.6.2
• Added org.jacoco:jacoco-maven-plugin:0.8.10
• Added org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184
• Added org.sonatype.ossindex.maven:ossindex-maven-plugin:3.2.0
• Added org.sonatype.plugins:nexus-staging-maven-plugin:1.6.13

1.4.0: Support Apache Spark 3.4.0

Summary

In this release, we added support for the Apache Spark 3.4.0 version. With this release, users could use the Exasol connector in the latest Apache Spark clusters such as Azure Databricks.

We also fixed bugs due to the missing dependency classes when using the connector as a Maven dependency in the Spark applications.

Features

• #147: Added support for Apache Spark 3.4.0 version

Bug Fixes

• #143: Fixed broken link in Readme file
• #145: Fixed missing class issue when using the connector as Maven dependency

Dependency Updates

Compile Dependency Updates

• Updated com.exasol:error-reporting-java8:0.4.1 to 1.0.1
• Updated com.exasol:exasol-jdbc:7.1.17 to 7.1.19
• Updated com.exasol:sql-statement-builder-java8:4.5.0 to 4.5.4
• Added com.fasterxml.woodstox:woodstox-core:6.5.1
• Updated com.google.protobuf:protobuf-java:3.22.2 to 3.22.3

Test Dependency Updates

• Updated com.exasol:exasol-testcontainers:6.5.1 to 6.5.2
• Updated com.exasol:hamcrest-resultset-matcher:1.5.2 to 1.6.0
• Updated org.mockito:mockito-core:5.2.0 to 5.3.0

Plugin Dependency Updates

• Updated com.diffplug.spotless:spotless-maven-plugin:2.35.0 to 2.36.0
• Updated com.exasol:error-code-crawler-maven-plugin:1.2.2 to 1.2.3
• Updated com.exasol:project-keeper-maven-plugin:2.9.6 to 2.9.7
• Updated org.apache.maven.plugins:maven-deploy-plugin:3.1.0 to 3.1.1
• Updated org.apache.maven.plugins:maven-enforcer-plugin:3.2.1 to 3.3.0
• Updated org.apache.maven.plugins:maven-failsafe-plugin:3.0.0-M8 to 3.0.0
• Updated org.apache.maven.plugins:maven-javadoc-plugin:3.4.1 to 3.5.0
• Updated org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M8 to 3.0.0
• Added org.basepom.maven:duplicate-finder-maven-plugin:1.5.1
• Updated org.codehaus.mojo:flatten-maven-plugin:1.3.0 to 1.4.1
• Updated org.codehaus.mojo:versions-maven-plugin:2.14.2 to 2.15.0
• Updated org.itsallcode:openfasttrace-maven-plugin:1.6.1 to 1.6.2
• Updated org.jacoco:jacoco-maven-plugin:0.8.8 to 0.8.9

1.3.0: Release Artifact with Scala 2.12

Summary

In this release we added an artifact release for Scala 2.12 version.

Features

• #140: Added release artifact with Scala 2.12 variant

Bugs

• #131: Removed project-keeper check suppression on dependencies.md file

Dependency Updates

Compile Dependency Updates

• Updated com.google.protobuf:protobuf-java:3.22.1 to 3.22.2

Plugin Dependency Updates

• Updated com.diffplug.spotless:spotless-maven-plugin:2.34.0 to 2.35.0
• Updated com.exasol:project-keeper-maven-plugin:2.9.3 to 2.9.6
• Updated org.apache.maven.plugins:maven-deploy-plugin:3.0.0 to 3.1.0
• Updated org.apache.maven.plugins:maven-enforcer-plugin:3.1.0 to 3.2.1