Mirror of Apache livy (Incubating)
Scala Java Python Shell JavaScript HTML Other
Clone or download
Permalink
Failed to load latest commit information.
.github [MINOR] Adding a PR Template Oct 9, 2017
api [LIVY-434][BUILD] Update master to version 0.6.0-incubating-SNAPSHOT Jan 23, 2018
assembly [LIVY-448] Assembly format is duplicated in pom.xml and assembly.xml Mar 8, 2018
bin LIVY-364. livy-server writes one log file when set LIVY_MAX_LOG_FILE… Jul 12, 2017
client-common [LIVY-434][BUILD] Update master to version 0.6.0-incubating-SNAPSHOT Jan 23, 2018
client-http [LIVY-434][BUILD] Update master to version 0.6.0-incubating-SNAPSHOT Jan 23, 2018
conf [LIVY-475] Support of Hadoop CredentialProvider API May 31, 2018
core [HOTFIX] Fix Base64 not existed in jdk7 issue Jun 13, 2018
coverage [LIVY-434][BUILD] Update master to version 0.6.0-incubating-SNAPSHOT Jan 23, 2018
dev [MINOR][BUILD] Improve release-build script Feb 9, 2018
docs [LIVY-452] Differentiate FAILED and KILLED states May 21, 2018
examples [LIVY-434][BUILD] Update master to version 0.6.0-incubating-SNAPSHOT Jan 23, 2018
integration-test [LIVY-473][TEST] Minor refactor of integration test to remove legacy … May 29, 2018
python-api [LIVY-434][BUILD] Update master to version 0.6.0-incubating-SNAPSHOT Jan 23, 2018
repl [Security] Update to support pyspark and sparkr changes in Spark 2.3.1 Jun 13, 2018
rsc [LIVY-466][RSC] Fix RSCDriver exception during RPC shutdown May 3, 2018
scala-api [LIVY-446][BUILD] Livy to Support Spark 2.3 Mar 12, 2018
scala [LIVY-434][BUILD] Update master to version 0.6.0-incubating-SNAPSHOT Jan 23, 2018
server [LIVY-475] Support of Hadoop CredentialProvider API May 31, 2018
test-lib [LIVY-446][BUILD] Livy to Support Spark 2.3 Mar 12, 2018
.gitignore [MINOR][BUILD] Improve release-build script Feb 9, 2018
.rat-excludes [MINOR] Fix travis builds Apr 9, 2018
.travis.yml [LIVY-457][REPL] Fix SQLContext is not initialized correctly issue Apr 18, 2018
DISCLAIMER [MINOR] Add Incubating DISCLAIMER file Aug 4, 2017
LICENSE [LIVY-384][DOCS] Added ability to build Livy Docs Aug 4, 2017
NOTICE [LIVY-440] Update copyright year to 2018 Feb 6, 2018
README.md [LIVY-376][DOCS] Update README with new naming and links and move Doc… Jul 27, 2017
checkstyle-suppressions.xml LIVY-374. Update and unify all the license header to Apache Jul 4, 2017
checkstyle.xml LIVY-375. Change Livy code package name to org.apache.livy Jul 5, 2017
pom.xml [Security] Update to support pyspark and sparkr changes in Spark 2.3.1 Jun 13, 2018
scalastyle.xml LIVY-375. Change Livy code package name to org.apache.livy Jul 5, 2017

README.md

Apache Livy

Build Status

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in Apache Hadoop YARN.

  • Interactive Scala, Python and R shells
  • Batch submissions in Scala, Java, Python
  • Multiple users can share the same server (impersonation support)
  • Can be used for submitting jobs from anywhere with REST
  • Does not require any code change to your programs

Pull requests are welcomed! But before you begin, please check out the Contributing section on the Community page of our website.

Online Documentation

Guides and documentation on getting started using Livy, example code snippets, and Livy API documentation can be found at livy.incubator.apache.org.

Before Building Livy

To build Livy, you will need:

Debian/Ubuntu:

  • mvn (from maven package or maven3 tarball)
  • openjdk-7-jdk (or Oracle Java7 jdk)
  • Python 2.6+
  • R 3.x

Redhat/CentOS:

  • mvn (from maven package or maven3 tarball)
  • java-1.7.0-openjdk (or Oracle Java7 jdk)
  • Python 2.6+
  • R 3.x

MacOS:

  • Xcode command line tools
  • Oracle's JDK 1.7+
  • Maven (Homebrew)
  • Python 2.6+
  • R 3.x

Required python packages for building Livy:

  • cloudpickle
  • requests
  • requests-kerberos
  • flake8
  • flaky
  • pytest

To run Livy, you will also need a Spark installation. You can get Spark releases at https://spark.apache.org/downloads.html.

Livy requires at least Spark 1.6 and supports both Scala 2.10 and 2.11 builds of Spark, Livy will automatically pick repl dependencies through detecting the Scala version of Spark.

Livy also supports Spark 2.0+ for both interactive and batch submission, you could seamlessly switch to different versions of Spark through SPARK_HOME configuration, without needing to rebuild Livy.

Building Livy

Livy is built using Apache Maven. To check out and build Livy, run:

git clone https://github.com/apache/incubator-livy.git
cd livy
mvn package

By default Livy is built against Apache Spark 1.6.2, but the version of Spark used when running Livy does not need to match the version used to build Livy. Livy internally uses reflection to mitigate the gaps between different Spark versions, also Livy package itself does not contain a Spark distribution, so it will work with any supported version of Spark (Spark 1.6+) without needing to rebuild against specific version of Spark.