Multi-tenant Hadoop cluster deployment with HA and Kerberos in 1 command
Shell Logos PHP CoffeeScript Python GCC Machine Description Other
Permalink
Failed to load latest commit information.
ambari ambari/agent: force name to hostname when registering Dec 4, 2016
bin package: ryba script Feb 14, 2016
cloudera-manager mecano: refactor registry Nov 20, 2016
commons uid_gid: update doc Nov 15, 2016
docs doc: replace development by setup Dec 20, 2016
druid trailing space Dec 20, 2016
elasticsearch Normalize log Dec 15, 2016
esdocker esdocker: remove \t Dec 20, 2016
falcon typos Jan 16, 2017
flume trailing space Dec 20, 2016
ganglia Normalize log Dec 15, 2016
graphite/carbon Normalize log Dec 15, 2016
hadoop hadoop/resources: fix service name template for namenode Jan 16, 2017
hbase typos Jan 16, 2017
hdp masson_2: refector zookeeper and hadoop Nov 15, 2016
hive typos Jan 16, 2017
hue add tmpfs pid dir creation for Centos/Redhat 7 Jan 6, 2017
huedocker huedocker: add tmpfs action instead of install for pid dir Jan 18, 2017
kafka kafka/consumer: typo mistake Jan 18, 2017
knox knox/resource: typo mistake Jan 18, 2017
lib capacity: isolate nn hdfs site Jan 11, 2017
mahout mecano: refactor registry Nov 20, 2016
mongodb refactor tmpfs to system.tmpfs Jan 6, 2017
nagios refactor tmpfs to system.tmpfs Jan 6, 2017
nagvis mecano: latest file.download and connection.wait Sep 25, 2016
nifi normalize and typos Jan 9, 2017
oozie typos Jan 10, 2017
opentsdb opentsdb: same conf for ranger admin ui than other components Jan 18, 2017
phoenix phoenix: service add restart Jan 10, 2017
pig pig: remove console.log Nov 17, 2016
ranger ranger/plugins/knox: add 0.6 configuration properties to authorise po… Jan 18, 2017
resources revert log4j last commit Jan 3, 2017
rexster normalize log Dec 19, 2016
shinken trailing space Dec 20, 2016
smartsense smartsense: migrate masson1.5 Jan 6, 2017
solr solr/cloud_docker: remove always restart command and pass ssl conf th… Jan 10, 2017
spark spark: rename deprecated options Jan 16, 2017
sqoop mecano: refactor registry Nov 20, 2016
test trailing space Dec 20, 2016
tez tez: implicit declaration of yarn and hdfs client Jan 3, 2017
titan normalize log Dec 19, 2016
zeppelin normalize log Dec 19, 2016
zookeeper zookeeper/client: add retry on check Jan 6, 2017
.gitignore Initial commit Sep 27, 2013
LICENSE.md license: title Apr 13, 2015
README.md readme: reformulate Apr 20, 2015
package.json package: latest dependencies Sep 25, 2016

README.md

Ryba boostraps and manages a full secured Hadoop cluster with one command. This is an Open-source software (OSS) project released under the new BSD license developed for one of the World largest utility company. It is now operationnal.

Ryba is our answer to DevOps integration need for product delivery and quality testing. It provides the flexibilty to answer the demand of your internal information technology (IT) operations team. It is written in JavaScript and CoffeeScript to facilitate and accelerate feature developments and maintenance releases. The language encourages self-documented code, look by yourself the source code deploying two HA namenodes.

Install Ryba locally or on a remote server and you are ready to go. It uses SSH to connect to each server of your cluster and will fully install all the components you wish. You don't need to prepare your cluster nodes as long as a minimal installation of RHEL or CentOS is installed with a root user or a user with sudo access.

Ryba motivations

  • Use secured comminication with SSH
  • No database used, full distribution across multiple servers relying on GIT
  • No agent or pre-installation required on your cluster nodes
  • Version control all your configuration and modifications with GIT and NPM, the Node.js Package Manager
  • Command-based to integrate with your Business Continuity Plan (BCP) and existing scripts
  • For developer, as simple as learning Node.js and not a new framework
  • Self-documented code written in Literate CoffeeScript
  • Idempotent and executable on a running cluster without any negative impact

Ryba features

  • Bootstrap the nodes from a fresh install
  • Configure proxy environment if needed
  • Optionnaly create a bind server (useful in Vagrant development environment)
  • Install OpenLDAP and Kerberos and/or integrate with your existing infrastructure
  • Deploy the latest Hortonworks Data Platform (HDP)
  • Setup High Availabity for HDFS
  • Integrate Kerberos with cross realm support
  • Set IPTables rules and startup scripts
  • Check the running components
  • Provide convenient utilities such as global start/stop/status commands, distributed shell execution, ...

Installation

Node.js

First download Node.js. You might need to adjust the name of the Node.js archive depending on the version you choose to install. Also, replace the path "/usr/local/node" to another location (eg "~/node") if you don't have the permission to write inside "/usr/local".

# Extract the Node.js package
tar xzf node-v0.10.28-linux-x64.tar.gz
# Move Node.js into its final destination
sudo mv node-v0.10.28-linux-x64 /usr/local/node
# Add path to Node.js binary
echo 'export PATH=/usr/local/node/bin:$PATH' >> ~/.bashrc
# Source the update profile
. ~/.bashrc
# Check if node is installed
node -v
# Clean up uploaded archive
rm -rf node-v0.10.28-linux-x64.tar.gz

If you are behind a proxy, configure the Node.js Pakage Manager (NPM) with the commands:

npm config set proxy http://proxy.company.com:8080
npm config set https-proxy http://proxy.company.com:8080

Ryba

Run npm install to download the project dependencies.

Contributors