Exasol Cloud Storage Extension provides Exasol user-defined functions (UDFs) for accessing formatted data stored in public cloud storage systems.
- Imports formatted data from public cloud storage systems.
- Supports the following data formats for importing: [Apache Avro][avro], Apache Orc and Apache Parquet.
- Allows data import from Delta Lake.
- Supports table export as Apache Parquet format to public cloud storage systems.
- Supports the following cloud storage systems: Amazon S3, Google Cloud Storage, Azure Blob Storage, Azure Data Lake (Gen1) Storage and Azure Data Lake (Gen2) Storage.
- Allows configuration of parallel importer or exporter processes.
For more information please check out the following guides.
The following sections list all the dependencies that are required for compiling, testing and running the project.
We compile and build the exasol-cloud-storage-extension releases using Java 8;
however, it should be safe to run it on the newer JVM versions. This is also
recommended way to build the Scala code.
| Dependency | Purpose | License |
|---|---|---|
| Exasol Import Export UDF Common | Common Import Export Library for Scala UDFs | MIT License |
| Apache Orc | Integration support for Orc format | Apache License 2.0 |
| Apache Parquet | Integration support for Parquet format | Apache License 2.0 |
| Hadoop AWS | Access for Amazon S3 object store and compatible implementations | Apache License 2.0 |
| Hadoop Azure | Access support for Azure Blob Storage | Apache License 2.0 |
| Hadoop Azure Datalake | Access support for Azure Data Lake Store | Apache License 2.0 |
| Hadoop Client | Apache Hadoop common dependencies as configuration or filesystem | Apache License 2.0 |
| Google Cloud Storage | Access support for Google Cloud Storage | Apache License 2.0 |
| Delta Lake Core | Integration support for Delta Lake format | Apache License 2.0 |
| Apache Spark SQL | Access support for Delta Lake formatted files | Apache License 2.0 |
| Dependency | Purpose | License |
|---|---|---|
| Scalatest | A testing tool for Scala and Java developers | Apache License 2.0 |
| Scalatest Plus | An integration support between Scalatest and Mockito | Apache License 2.0 |
| Mockito Core | A mocking framework for unit tests | MIT License |
| Java Hamcrest | Matchers to create the flexible expressions of intent | BSD License |
| Hamcrest Resultset Matcher | Matchers for JDBC ResultSet objects | MIT License |
| Exasol Test Database Builder | Library to create and clean up database for integration tests | MIT License |
| Exasol Testcontainers | Test container for Exasol on Docker | MIT License |
| Localstack Testcontainers | A local docker AWS cloud containers | MIT License |
These plugins help with project development.
| Plugin Name | Purpose | License |
|---|---|---|
| SBT Coursier | Pure Scala artifact fetching | Apache License 2.0 |
| SBT Wartremover | Flexible Scala code linting tool | Apache License 2.0 |
| SBT Wartremover Contrib | Community managed additional warts for wartremover | Apache License 2.0 |
| SBT Assembly | Create fat jars with all project dependencies | MIT License |
| SBT API Mappings | A plugin that fetches API mappings for common Scala libraries | Apache License 2.0 |
| SBT Scoverage | Integrates the scoverage code coverage library | Apache License 2.0 |
| SBT Coveralls | Uploads scala code coverage results to https://coveralls.io | Apache License 2.0 |
| SBT Updates | Checks Maven and Ivy repositories for dependency updates | BSD 3-Clause License |
| SBT Scalafmt | A plugin for https://scalameta.org/scalafmt/ formatting | Apache License 2.0 |
| SBT Scalastyle | A plugin for http://www.scalastyle.org/ Scala style checker | Apache License 2.0 |
| SBT Dependency Graph | A plugin for visualizing dependency graph of your project | Apache License 2.0 |
| SBT Explicit Dependencies | Checks which direct libraries required to compile your code | Apache License 2.0 |
| SBT Git | A plugin for Git integration, used to version the release jars | BSD 2-Clause License |