This library contains the source code for Azure Data Explorer DataStream Sink Connector for Apache Flink.
This sink can write data to : Azure Data Explorer, Azure Synapse Data Explorer and Real time analytics in Fabric
Azure Data Explorer (A.K.A. Kusto) is a lightning-fast indexing and querying service.
Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
Making Azure Data Explorer and flink work together enables building fast and scalable applications, targeting a variety of Machine Learning, Extract-Transform-Load, Log Analytics and other data-driven scenarios.
For major changes from previous releases, please refer to Releases. For known or new issues, please refer to the issues section.
For Java applications using Maven project definitions, link your application with the artifact below to use the Azure Data Explorer Connector for Flink.
In Maven:
groupId = com.microsoft.azure.kusto
artifactId = flink-connector-kusto
version = 1.0.0
From maven central look for the following coordinates:
com.microsoft.azure.kusto:flink-connector-kusto:1.0.0
(or)
<dependency>
<groupId>com.microsoft.azure.kusto</groupId>
<artifactId>flink-connector-kusto</artifactId>
<version>1.0.0</version>
</dependency>Or clone this repository and build it locally to add it to your local maven repository using mvn clean install -DskipTests
The jar can also be found under the released package
Samples are packaged as a separate module with the following artifact
<artifactId>samples-java</artifactId>To use the connector, you need:
// Clone and navigate to the repo
git clone git@github.com:Azure/flink-connector-kusto.git
cd flink-connector-kusto
// Builds jar and run all tests
mvn clean test integration-test package -DappId=<app-id> -Dcluster=<cluster> -DappKey=<app-key> -Dauthority=<tenant-d> -Ddatabase=<database>
// Builds jar, runs all tests, and installs jar to your local maven repository
mvn clean install -DappId=<app-id> -Dcluster=<cluster> -DappKey=<app-key> -Dauthority=<tenant-d> -Ddatabase=<database>To facilitate ramp-up from local jar on platforms such as Azure Databricks, pre-compiled libraries are published under GitHub Releases. These libraries include:
- Azure Data Explorer connector library
- User may also need to include Kusto Java SDK libraries (kusto-data and kusto-ingest), which are published under GitHub Releases
Flink Kusto connector depends on Azure Data Explorer Data Client Library and Azure Data Explorer Ingest Client Library, available in maven repository.
Detailed documentation can be found here.
Usage examples can be found here
- Have a feature request for SDKs? Please post it on User Voice to help us prioritize
- Have a technical question? Ask on Stack Overflow with tag "azure-data-explorer"
- Need Support? Every customer with an active Azure subscription has access to support with guaranteed response time. Consider submitting a ticket for assistance from the Microsoft support team.
- Found a bug? Please help us fix it by thoroughly documenting it and filing an issue.
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.
The Flink committers use IntelliJ IDEA to develop the Flink codebase. We recommend IntelliJ IDEA for developing projects that involve Scala code.
Minimal requirements for an IDE are:
- Support for Java and Scala (also mixed projects)
- Support for Maven with Java and Scala
The IntelliJ IDE supports Maven out of the box and offers a plugin for Scala development.
- IntelliJ download: https://www.jetbrains.com/idea/
- IntelliJ Scala Plugin: https://plugins.jetbrains.com/plugin/?id=1347
Check out our Setting up IntelliJ guide for details.
This is an active open-source project. We are always open to people who want to use the system or contribute to it. Contact us if you are looking for implementation tasks that fit your skills. This article describes how to contribute to Apache Flink.
Apache Flink is an open source project of The Apache Software Foundation (ASF). The Apache Flink project originated from the Stratosphere research project.