Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
- Hadoop 2.x
- Java 1.7 or higher as $JAVA_HOME
Automatic Installation via Chef
Run the Chef recipes located in the separate hiway-chef repository.
Manual Three-Step Installation
On all Hadoop worker nodes:
- Install Hi-WAY: Clone and build manually (against your installed Hadoop version), or download the latest build from Sonatype.
- Make Hi-WAY available to YARN Containers: Add the Hi-WAY folder and the lib folder within it to the yarn.application.classpath property in the yarn-site.xml configuration file in the $HADOOP_CONF_DIR.
- Configure Hi-WAY: Generate the hiway-site.xml as found in the hiway-chef repository and put it into the $HADOOP_CONF_DIR; make sure that the hiway.am.directory.cache property points to a HDFS directory in which both the user running yarn and the user running hiway have write access to.
Run an Example
Run an example workflow like the helloworld.cf Cuneiform workflow in the hiway-chef repository:
$HADOOP_HOME/bin/yarn jar $HIWAY_HOME/hiway-1.0.0-SNAPSHOT.jar -w helloworld.cf -s report.json.