Skip to content
This repository has been archived by the owner on Jul 13, 2023. It is now read-only.

delta-io/connectors

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
October 29, 2020 09:33
October 28, 2021 13:58
October 28, 2021 13:58
July 13, 2023 13:10

🛑 DEPRECATED 🛑

The code under this repository has been moved under connectors module in https://github.com/delta-io/delta repository. No changes are allowed to this repository. Any issues or changes, please propose them on the https://github.com/delta-io/delta repository. More details on deprecation are found at #556.


Delta Lake Logo Connectors

Test License

We are building connectors to bring Delta Lake to popular big-data engines outside Apache Spark (e.g., Apache Hive, Presto, Apache Flink) and also to common reporting tools like Microsoft Power BI.

Introduction

This is the repository for Delta Lake Connectors. It includes

Please refer to the main Delta Lake repository if you want to learn more about the Delta Lake project.

API documentation

Delta Standalone

Delta Standalone, formerly known as the Delta Standalone Reader (DSR), is a JVM library to read and write Delta tables. Unlike https://github.com/delta-io/delta, this project doesn't use Spark to read or write tables and it has only a few transitive dependencies. It can be used by any application that cannot use a Spark cluster.

  • To compile the project, run build/sbt standalone/compile
  • To test the project, run build/sbt standalone/test
  • To publish the JAR, run build/sbt standaloneCosmetic/publishM2

See Delta Standalone for detailed documentation.

Connectors

Hive Connector

Read Delta tables directly from Apache Hive using the Hive Connector. See the dedicated README.md for more details.

Flink/Delta Connector

Use the Flink/Delta Connector to read and write Delta tables from Apache Flink applications. The connector includes a sink for writing to Delta tables from Apache Flink, and a source for reading Delta tables using Apache Flink (still in progress.) See the dedicated README.md for more details.

sql-delta-import

sql-delta-import allows for importing data from a JDBC source into a Delta table.

Power BI connector

The connector for Microsoft Power BI is basically just a custom Power Query function that allows you to read a Delta table from any file-based data source supported by Microsoft Power BI. Details can be found in the dedicated README.md.

Reporting issues

We use GitHub Issues to track community reported issues. You can also contact the community for getting answers.

Contributing

We welcome contributions to Delta Lake Connectors repository. We use GitHub Pull Requests for accepting changes.

Community

There are two mediums of communication within the Delta Lake community.

Local Development & Testing

  • Before local debugging of standalone tests in IntelliJ, run all standalone tests using SBT. This helps IntelliJ recognize the golden tables as class resources.