Intelligent, Multi-Engine Resource Scheduler for Big Data Analytics Workflows
Before using IReS it should be configured properly. This setting consists of 3 main steps,
- Cloning IReS-Platform to the server. For a quick reference of how to use git, click here Git - the simple guide
- Running install.sh that
- builds IReS-Platform using Maven. A tutorial about Maven can be found here Maven Getting Started Guide.
- connects IReS to Hadoop YARN.
- updates configuration files and folders appropriately.
- Setting cluster resources and services monitoring
For demonstration reasons a Linux operating system like Ubuntu is assumed throughout this text. In Windows or other Linux distributions the equivalents should be done. The local home directory of the IReS-Platform project is depicted as
following bash script variable notation. Similarly, for Hadoop YARN the local home directory is denoted as
Open a terminal( Linux) and navigate to a desired directory (create it if does not exist) where IReS-Platform files will be cloned. In CloneIReS-Platform github page, under the green drop down list "Clone or download", the clone url can be found. Copy this url and execute in terminal the command,
git clone clone_url
After successful cloning of IReS-Platform inside the
$IRES_HOMEvarious folders and files can be found. Among them there exists
You can run install.sh from any directory you would like. Here for demnostration reasons is assumed that the current working directory is $IRES_HOME.
install.shis your friend!
will start building IReS-Platform. Upon successful building you will be prompted to provide the path where Hadoop YARN is located in your computer. By doing this, IReS gets connected with Hadoop YARN. You can skip this step and the installation will be finished.
- if you do not provide an existing YARN installation, then IReS will not be able to execute any workflow. Also, resources and cluster services monitoring will not be functioning.
- you can provide YARN installation path afterwards as it will be shown straight ahead.
Connecting IReS to Hadoop YARN
will make the connection of IReS and YARN, where $YARN_HOME and $IRES_HOME correspond to the absolute paths of YARN's and IReS's home folder.
./install.sh -c $YARN_HOME,$IRES_HOME
This step requires the connection of IReS with YARN. Assuming that this connection has been established, then the user should update the file
and more specifically the values of the following properties,
These properties and some others have been added during the connection of IReS and YARN to enable IReS run workflows over YARN and cluster resources and services monitoring. Although details about filling these values are provided into
$YARN_HOME/etc/hadoop/yarn-site.xml, roughly speaking,
yarn.nodemanager.services-running.per-nodeproperty describes the cluster services running per node. The property,
yarn.nodemanager.services-running.check-availabilityprovides the commands per service that "tell" if the relative service runs or not. Finally, the property
yarn.nodemanager.services-running.check-statushas the statuses per service that the corresponding service has when it runs.
Here are some tips to confirm IReS installation.
- If anything goes wrong during the build process of IReS, error messages will be print out and a log file will be provided.
- Run ASAP server by running the command
No exception should be raised. Also, the
./install.sh -r start
jpscommand should print a "Main" process running that corresponds to ASAP server.
- Run ASAP server web user interface at http://your_hostname:1323/web/main. IReS home page should be displayed.
- Run a workflow, for example run "hello_world" from "Abstrack Workflows" tab and see what happens not only in IReS web interface but also in YARN and HDFS web interfaces. Make sure that YARN has been started before running any workflow.
- Click on "Cockpit" tab to verify that the expected services to run are really running.
To see all available functionalities of Appendix
However, in this section some details about
install.shscript are given.
To begin with, install.sh supports can be run from user's working directory, it is not needed navigating to
IRES_HOME folder. Moreover, user can access server's logs by running
install.sh script using
To see all available functionalities of
Now assume that for some reason the IRES_HOME has been changed. For example you would like to rename IReS home folder or move it to another folder. In this case, all configuration files of IReS should be updated with the new value of $IRES_HOME. To do so, run the command
./install.sh -s NEW_IRES_HOME