Spring Cloud Data Flow is a toolkit for building data integration and real-time data processing pipelines.
This makes Spring Cloud Data Flow suitable for a range of data processing use cases, from import/export to event streaming and predictive analytics.
The Core domain module includes the concept of a stream that is a composition of spring-cloud-stream modules in a linear pipeline from a source to a sink, optionally including processor application(s) in between. The domain also includes the concept of a task, which may be any process that does not run indefinitely, including Spring Batch jobs.
The App Registry
maintains the set of available apps, and their mappings to URIs.
For example, if relying on Maven coordinates, an app’s URI would be of the format:
The Data Flow Server is a Spring Boot application that provides a common REST API and UI. For each runtime environment there is a different version of the Data Flow Server that depends upon a deployer SPI implementation for that environment. The github locations for these Data Flow Servers are:
The deployer SPI mentioned above is defined within the Spring Cloud Deployer project. That provides an abstraction layer for deploying the apps of a given stream or task and managing their lifecycle. The github locations for the corresponding Spring Cloud Deployer SPI implementations are:
The Shell connects to the Data Flow Server’s REST API and supports a DSL that simplifies the process of defining a stream or task and managing its lifecycle.
Instructions for running the Data Flow Server for each runtime environment can be found in their respective github repositories.
Clone the repo and type
$ ./mvnw clean install
For more information on building, see this link.
Building on Windows
When using Git on Windows to check out the project, it is important to handle line-endings correctly during checkouts. By default Git will change the line-endings during checkout to
CRLF. This is, however, not desired for Spring Cloud Data Flow as this may lead to test failures under Windows.
Therefore, please ensure that you set Git property
false, e.g. using:
$ git config core.autocrlf false. Fore more information please refer to the Git documentation, Formatting and Whitespace.
We welcome contributions! Follow this link for more information on how to contribute.
When reporting problems, it’d be helpful if the bug report includes the details listed on this wiki-article.
Code formatting guidelines
The directory ./src/eclipse has two files for use with code formatting,
eclipse-code-formatter.xmlfor the majority of the code formatting rules and
eclipse.importorderto order the import statements.
In eclipse you import these files by navigating
Windows → Preferencesand then the menu items
Preferences > Java > Code Style > Formatterand
Preferences > Java > Code Style > Organize Importsrespectfully.
IntelliJ, install the plugin
Eclipse Code Formatter. You can find it by searching the "Browse Repositories" under the plugin option within
IntelliJ(Once installed you will need to reboot Intellij for it to take effect). Then navigate to
Intellij IDEA > Preferencesand select the Eclipse Code Formatter. Select the
eclipse-code-formatter.xmlfile for the field
Eclipse Java Formatter config fileand the file
eclipse.importorderfor the field
Import order. Enable the
Eclipse code formatterby clicking
Use the Eclipse code formatterthen click the OK button.
NOTE: If you configure the
Eclipse Code Formatterfrom
File > Other Settings > Default Settingsit will set this policy across all of your Intellij projects.