A Jenkins plugin that allows to deploy / stop Apache Spark applications in Spark standalone clusters.
Switch branches/tags
Nothing to show
Clone or download
Permalink
Failed to load latest commit information.
src/main Initial Commit Oct 25, 2015
.gitignore Initial Commit Oct 25, 2015
LICENSE Initial Commit Oct 25, 2015
README.md Updated README Oct 25, 2015
pom.xml [maven-release-plugin] prepare for next development iteration Oct 25, 2015

README.md

Jenkins Spark Deployer

This Jenkins plugin enables you to deploy Apache Spark Applications to Spark Standalone clusters as a post build action. Keeps track of previous deployments and you can use the 'kill' feature to kill the previous submission before deploying a new version of your application.

Features

  • Support for custom Spark Configuration options.
  • Kill previous submissions - Allows you to kill your last submission automatically before deploying the new version, if desired.
  • Fail build on unsuccessful deployments (configurable).
  • A verbose mode, which prints out requests / responses for easier debugging when things doesn't work as expected.

Installation

TBD

Screenshots

Options

Configuration Options

Spark Master URL (*) This is passed as a configuration to executors. Example: spark://master.spark.cluster.com:6066
Spark Master REST HTTP URL Optional. If your Spark Masters are behind a load balancer / proxy, you can use this to tell Jenkins which URL to use for communicating over REST. Also, if you have HTTPS enabled, you will have to use this option. If this is not provided, the Spark Master URL will be used to determine the default (that is spark:// will become http://). Example: http://master.spark.cluster.com:6066
Spark Scala Version Optional. Spark Scala Version. Default: 2.10.
Application Resource URL (*) Application JAR URL. If the JAR file is in HDFS, this should be the HDFS URL. If it is available over HTTP or HTTPS, this should be the relevant URL. File URLs are also supported, but those URLs should be valid within executors for that to work. Example: http://some.file.server/spark-app.jar
Main Class (*) The fully qualified main class name. For example, org.spark.demo.app.Main
Application Arguments These are the command line arguments that will passed to the Spark application. Multiple arguments should be separated by a space. For example: arg1 arg2 arg3
Spark Configuration Properties Additional properties for Spark. These are typically passed in as --conf parameters for Spark Submit. The settings should be specified as key value pairs separated by commas. Example: spark.driver.supervise=false,spark.executor.memory=2G
Kill previous submissions before deploying If enabled, Jenkins will try to kill the last submission done for this job before deploying latest version.
Fail the build if deployment fails It does exactly what it says :)
Verbose Prints the REST request / response in build output for debugging purposes.
(*) - Required Field

Notes

  • Spark on YARN and Mesos are not supported yet.