| Branch | Build | Coverage |
|---|---|---|
| Develop |  |
 |
| Master |  |
 |
The Metalus library was created as a way to create Spark applications at runtime without the need to write or compile code. The library is written in Scala and provides binaries for different version of Spark and Scala. Developers build applications by providing a JSON configuration file which gets loaded and executed by the metalus core library.
Documentation for this project may be found here.
Instructions for contributing to this project and instructions on building may be found here.
There are several sub-projects:
This project contains the core library and is the minimum requirement for any application.
This step library contains steps that are considered generic enough to be used in any project.
This step library contains AWS specific components. The Kinesis driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.
This step library contains GCP specific components. The Pub/Sub driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.
This step library contains GCP specific components. The Kafka driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.
This step library adds support for working with Mongo.
This project provides several examples to help demonstrate how to use the library.
This project provides utilities that help work with the project.
This project provides a single jar that can be used to run the application. Additional components provide jars that can be added to the classpath.
Examples of building pipelines can be found in the metalus-examples project.