Livy Spark Server parcel and CSD
Livy is a wrapper
spark-submit to expose Spark as a REST service for integration with external applications.
As it is still in beta, it is not included in the CDH bundle (as of CDH 5.5.1). This scripts in this repo creates both a Livy parcel and CSD for deploying it with Cloudera Manager.
Cloudera publishes a tool for making CM parcels. Install it on your workstation like so:
mkdir -p ~/github/cloudera; cd ~/github/cloudera
git clone firstname.lastname@example.org:cloudera/cm_ext.git; cd cm_ext
- Create the Livy parcel:
./build_parcel.sh <Version> <Distro>. Example:
./build_parcel.sh 1.0 wheezy
- Serve the parcel:
- In CM, add your machine to the list of
Remote Parcel Repository URLsand click
Check for New Parcels.
- Download, Distribute, Activate. No need to restart the cluster as this parcel is not a dependency for any service.
- Create the CSD:
./build_csd.sh <Version>. Example:
./build_csd.sh 1.0. The CSD version is independent of the parcel version.
- Copy the CSD jar (
LIVY-1.0.jar) to CM host at
/opt/cloudera/csd/and change the ownership of the jar to
- Restart CM
- Restart Cloudera Management Service from the CM UI.
- Install the Livy service with Add a service option in CM.
Note: Step 3 and 4 can be avoided by using the
api for installing CSDs without restarting CM with a
Livy has been defined as a javabased service in CSD which enables the basic monitoring and configuration relevant to java process through CM. Additional Livy parameters has also be added to the CSD to enable configuration management from CM. We are forcing logback as the logging implementation.
As of now, custom CSDs support declaring custom montitoring metrics but no way to publish them from the service.