Spark SBT Multi Module Example

A sample project to bootstrap all scala/spark project with multiple sbt modules.

Please change the module names, dependencies, project settings accordingly.

Files to change: build.sbt, ProjectSettings, Dependencies and Versions (if needed).

Features

• Publish the artifact to GitHub Packages
• Uses LazyLogging from Lightbend
• Uses HOCON format for configuration file
• Uses scopt to parse command line arguments

Plugins

• sbt-scalafmt
• sbt-assembly
• sbt-dynver
• sbt-docker

How to publish packages

create file : nano ~/.sbt/1.0/.credentials

realm=GitHub Package Registry
host=maven.pkg.github.com
user=helkaroui
password=xxxxxxxxxxxxxxxx

then run publish in sbt

Test

Run the tests with enabled coverage:

sbt clean test

To generate the coverage reports run

sbt coverageReport

By default, scoverage will generate reports for each project separately. You can merge them into an aggregated report by using the following:

sbt coverageAggregate

NOTE: You do not need to run coverageReport before coverageAggregate

Todo :

• Add a spark streaming example
• Add test mocking kafka
• Add a spark cassandra example
• Add test mocking cassandra
• Add Coveralls support
• Add sbt-docker
• Add CICD example