forked from Acxiom/metalus
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Acxiom#345 Updated readme files and fixed aws-spark pom.
- Loading branch information
Showing
3 changed files
with
97 additions
and
40 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| [Documentation Home](../docs/readme.md) | ||
|
|
||
| # Metalus Spark | ||
| The core library is the minimum required to start building applications that can run on any machine. This library provides | ||
| the core mapping, data reference, application, pipeline and extension mechanisms. There are versions available for Scala | ||
| 2.12 and 2.13. This library may be integrated into any Maven project with the following dependency: | ||
|
|
||
| ```xml | ||
| <dependency> | ||
| <groupId>com.acxiom</groupId> | ||
| <artifactId>metalus-core_${scala.compat.version}</artifactId> | ||
| <version>${version}</version> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,52 +1,88 @@ | ||
| |Branch|Build|Coverage| | ||
| -------|-----|--------| | ||
| |Develop|[](https://github.com/Acxiom/metalus/actions/workflows/develop.yml)|[](https://coveralls.io/github/Acxiom/metalus?branch=develop)| | ||
| |Master|[](https://github.com/Acxiom/metalus/actions/workflows/release.yml)|[](https://coveralls.io/github/Acxiom/metalus?branch=master)| | ||
|
|
||
| # Metalus Pipeline Library | ||
| The Metalus library was created as a way to create Spark applications at runtime without the need to write or compile | ||
| code. The library is written in Scala and provides binaries for different version of Spark and Scala. Developers build | ||
| applications by providing a JSON configuration file which gets loaded and executed by the metalus core library. | ||
| The Metalus pipeline library provides a convenient way to build applications using JSON. Additional libraries are provided | ||
| for working in different cloud environments and with different versions of Spark. | ||
|
|
||
| ## [Documentation](docs/readme.md) | ||
| Documentation for this project may be found [here](docs/readme.md). | ||
|
|
||
| ## [Contributing](docs/contributions.md) | ||
| Instructions for contributing to this project and instructions on building may be found [here](docs/contributions.md). | ||
|
|
||
| ## Projects | ||
| There are several sub-projects: | ||
| ## Libraries | ||
| There are several libraries provided: | ||
|
|
||
| ### [Metalus Pipeline Core](metalus-core/readme.md) | ||
| This project contains the core library and is the minimum requirement for any application. | ||
| ### [Metalus Core Library](metalus-core/readme.md) | ||
| The core library is the minimum required to start building applications that can run on any machine. This library provides | ||
| the core mapping, data reference, application, pipeline and extension mechanisms. There are versions available for Scala | ||
| 2.12 and 2.13. This library may be integrated into any Maven project with the following dependency: | ||
|
|
||
| ### [Metalus Common Step Library](metalus-common/readme.md) | ||
| This step library contains steps that are considered generic enough to be used in any project. | ||
| ```xml | ||
| <dependency> | ||
| <groupId>com.acxiom</groupId> | ||
| <artifactId>metalus-core_${scala.compat.version}</artifactId> | ||
| <version>${version}</version> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| ``` | ||
|
|
||
| ### [Metalus AWS Step Library](metalus-aws/readme.md) | ||
| This step library contains AWS specific components. The [Kinesis](https://aws.amazon.com/kinesis/) driver provides a | ||
| basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data. | ||
| ### [Metalus AWS Library](metalus-aws/readme.md) | ||
| The AWS library is built on top of the core library and provides components and steps for working with AWS services such | ||
| as Kinesis, S3, Lambda and Secrets Manager. There are builds for Scala 2.12 and 2.13. This library may be integrated into | ||
| any Maven project using the core dependency and this additional dependency: | ||
|
|
||
| ### [Metalus GCP Step Library](metalus-gcp/readme.md) | ||
| This step library contains GCP specific components. The [Pub/Sub](https://cloud.google.com/pubsub/docs/overview) driver provides a | ||
| basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data. | ||
| ```xml | ||
| <dependency> | ||
| <groupId>com.acxiom</groupId> | ||
| <artifactId>metalus-aws_${scala.compat.version}</artifactId> | ||
| <version>${version}</version> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| ``` | ||
|
|
||
| ### [Metalus Kafka Step Library](metalus-kafka/readme.md) | ||
| This step library contains GCP specific components. The [Kafka](https://kafka.apache.org/) driver provides a | ||
| basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data. | ||
| ### [Metalus GCP Library](metalus-gcp/readme.md) | ||
| The GCP library is built on top of the core library and provides components and steps for working with GCP services such | ||
| as Pub/Sub, GCS, Cloud Functions and Secrets Manager. There are builds for Scala 2.12 and 2.13. This library may be | ||
| integrated into any Maven project using the core dependency and this additional dependency: | ||
|
|
||
| ### [Metalus Mongo Step Library](metalus-mongo/readme.md) | ||
| This step library adds support for working with Mongo. | ||
| ```xml | ||
| <dependency> | ||
| <groupId>com.acxiom</groupId> | ||
| <artifactId>metalus-gcp_${scala.compat.version}</artifactId> | ||
| <version>${version}</version> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| ``` | ||
|
|
||
| ### [Metalus Pipeline Examples](metalus-examples/readme.md) | ||
| This project provides several examples to help demonstrate how to use the library. | ||
| ### [Metalus Spark Library](metalus-spark/readme.md) | ||
| The Spark library is built on top of the core library and provides integration with Apache Spark. A special `Context` | ||
| is provided that allows setting up a `SparkSession` as wells as steps for preforming runtime configurations and steps | ||
| /components for working with Structured Streaming. There are builds for Scala 2.12 and 2.13 and Spark versions 3.1, 3.2 | ||
| and 3.3. This library may be integrated into any Maven project using the core dependency and this additional dependency: | ||
|
|
||
| ### [Metalus Utilities](metalus-utils/readme.md) | ||
| This project provides utilities that help work with the project. | ||
| ```xml | ||
| <dependency> | ||
| <groupId>com.acxiom</groupId> | ||
| <artifactId>metalus-spark_${scala.compat.version}-spark_${spark.compat.version}</artifactId> | ||
| <version>${version}</version> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| ``` | ||
|
|
||
| ### [Metalus Application](metalus-application/readme.md) | ||
| This project provides a single jar that can be used to run the application. Additional components provide jars that can be | ||
| added to the classpath. | ||
| ### [Metalus AWS Spark Library](metalus-aws-spark/readme.md) | ||
| The AWS Spark library is built on top of the core, AWS and Spark libraries and provides components and steps for working | ||
| with AWS services using Spark as the execution engine. There are builds for Scala 2.12 and 2.13 and Spark versions 3.1, | ||
| 3.2 and 3.3. This library may be integrated into any Maven project using the core, aws and spark dependencies and this | ||
| additional dependency: | ||
|
|
||
| ## Examples | ||
| Examples of building pipelines can be found in the [metalus-examples](metalus-examples/readme.md) project. | ||
| ```xml | ||
| <dependency> | ||
| <groupId>com.acxiom</groupId> | ||
| <artifactId>metalus-aws-spark_${scala.compat.version}-spark_${spark.compat.version}</artifactId> | ||
| <version>${version}</version> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| ``` | ||
|
|
||
| ### [Metalus Utilities](metalus-utils/readme.md) | ||
| The Metalus Utilities project provides tools for extracting step, pipeline and application metadata from the standard and | ||
| extended libraries. The dependency resolution tool provides the ability to build a classpath from one or more of the libraries. |