Skip to content
Code for the deployment of Hadoop clusters, written in Bourne or Bourne Again shell.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
api
ca
director Fix discover_os() on RHEL. Jan 22, 2019
navencrypt Fix discover_os() on RHEL. Jan 22, 2019
services Fix discover_os() on RHEL. Jan 22, 2019
tls
utilities Fix discover_os() on RHEL. Jan 22, 2019
.gitattributes
.gitignore Added .gitignore. Jan 12, 2016
.travis.yml
CONTRIBUTING.md Update CONTRIBUTING.md to reference shellcheck. Aug 30, 2018
EVALUATE.md
LICENSE
README.md
ambari_prepare_database.sh
change_swappiness.sh
configure_javahome.sh
configure_tuned.sh
disable_iptables.sh Fix discover_os() on RHEL. Jan 22, 2019
disable_ipv6.sh
disable_selinux.sh Fix discover_os() on RHEL. Jan 22, 2019
disable_thp.sh
evaluate.sh
install_clouderamanageragent.sh Fix discover_os() on RHEL. Jan 22, 2019
install_clouderamanagerserver.sh
install_clouderanavigatorencrypt.sh Fix discover_os() on RHEL. Jan 22, 2019
install_entropy.sh
install_hortonworksambariagent.sh
install_hortonworksambariserver.sh
install_jce.sh Fix discover_os() on RHEL. Jan 22, 2019
install_jdbc.sh Upgrade MYSQL_VERSION to 5.1.46. Feb 23, 2019
install_jdbc_sqoop.sh Fix discover_os() on RHEL. Jan 22, 2019
install_jdk.sh
install_krb5.sh
install_lzo.sh
install_nscd.sh Fix discover_os() on RHEL. Jan 22, 2019
install_ntp.sh Fix discover_os() on RHEL. Jan 22, 2019
install_tools.sh
install_tuned.sh
link_openssl.sh
remove_tuned.sh
tune_kernel.sh Fix discover_os() on RHEL. Jan 22, 2019
upgrade_clouderamanager.sh Fix discover_os() on RHEL. Jan 22, 2019

README.md

hadoop-deployment-bash

Cloudera

These are shell scripts to deploy Cloudera Manager and related Cloudera encryption products to a cluster. The goal of these scripts are to be idempotent and to serve as a template for translation into other Configuration Management frameworks/languages.

  • Works with RHEL/CentOS 6 or 7 x86_64.
  • Works with Ubuntu Trusty 14.04 x86_64.
  • Allows for installation of Oracle JDK 7 from Cloudera, Oracle JDK 8 from Oracle, or OpenJDK 7 or 8.

This is an example of some of the functionality. Not everything is documented. Some scripts have arguments that can be passed to them to change their internal operation. Read the source to learn more.

Prep

This is needed for both the Evaluation and Example sections below.

Set the GITREPO variable to the local directory where you have cloned this repository and create a file with SSH login and hostname.

GITREPO=~/git/teamclairvoyant/bash

cat <<EOF >HOSTLIST
centos@ip-10-2-6-62.ec2.internal
centos@ip-10-2-6-5.ec2.internal
centos@ip-10-2-5-22.ec2.internal
EOF

Evaluation

Run the evaluation script to gather the configuration of all the nodes of the cluster. Save the output in the directory "evaluate-pre".

mkdir evaluate-pre
for HOST in `cat HOSTLIST`; do
  echo "*** $HOST"
  scp -p ${GITREPO}/evaluate.sh ${HOST}:
  ssh -qt $HOST './evaluate.sh' >evaluate-pre/${HOST}.out 2>evaluate-pre/${HOST}.err
done

Example

Copy several of the scripts to the nodes.

for HOST in `cat HOSTLIST`; do
  echo "*** $HOST"
  scp -p \
  ${GITREPO}/install_tools.sh \
  ${GITREPO}/change_swappiness.sh \
  ${GITREPO}/disable_iptables.sh \
  ${GITREPO}/disable_ipv6.sh \
  ${GITREPO}/disable_selinux.sh \
  ${GITREPO}/disable_thp.sh \
  ${GITREPO}/install_ntp.sh \
  ${GITREPO}/install_nscd.sh \
  ${GITREPO}/install_jdk.sh \
  ${GITREPO}/configure_javahome.sh \
  ${GITREPO}/install_jce.sh \
  ${GITREPO}/install_krb5.sh \
  ${GITREPO}/configure_tuned.sh \
  ${GITREPO}/install_entropy.sh \
  ${GITREPO}/install_jdbc.sh \
  ${GITREPO}/install_jdbc_sqoop.sh \
  ${GITREPO}/install_clouderamanageragent.sh \
  $HOST:
done

Run the scripts to prep the system for Cloudera Manager installation. Pin the version of Cloudera Manager to the value in $CMVER. Also deploy Oracle JDK 8.

#BOPT="-x"    # Turn on bash debugging.
#CMVER=5.9.1  # Set specific Cloudera Manager version, or ...
CMVER=5       # ... use major version 5.
for HOST in `cat HOSTLIST`; do
  echo "*** $HOST"
  ssh -t $HOST " \
  sudo bash $BOPT ./install_tools.sh; \
  sudo bash $BOPT ./change_swappiness.sh; \
  sudo bash $BOPT ./disable_iptables.sh; \
  sudo bash $BOPT ./disable_ipv6.sh; \
  sudo bash $BOPT ./disable_selinux.sh; \
  sudo bash $BOPT ./disable_thp.sh; \
  sudo bash $BOPT ./install_ntp.sh; \
  sudo bash $BOPT ./install_nscd.sh; \
  sudo bash $BOPT ./install_jdk.sh --jdktype oracle --jdkversion 8; \
  sudo bash $BOPT ./configure_javahome.sh; \
  sudo bash $BOPT ./install_jce.sh; \
  sudo bash $BOPT ./install_krb5.sh; \
  sudo bash $BOPT ./configure_tuned.sh; \
  sudo bash $BOPT ./install_entropy.sh"
done

Install the Cloudera Manager agent.

CMSERVER=ip-10-2-5-22.ec2.internal
for HOST in `cat HOSTLIST`; do
  echo "*** $HOST"
  ssh -t $HOST "sudo bash $BOPT ./install_clouderamanageragent.sh $CMSERVER $CMVER"
done

Install the Cloudera Manager server with the embedded PostgreSQL database.

scp -p ${GITREPO}/install_clouderamanagerserver.sh ${CMSERVER}:
ssh -t ${CMSERVER} "sudo bash $BOPT ./install_clouderamanagerserver.sh embedded $CMVER"

You can use the argument embedded, postgresql, mysql, or oracle.

Post Evaluation

Run the evaluation script again to gather the new configuration of all the nodes of the cluster. Save the output in the directory "evaluate-post".

mkdir evaluate-post
for HOST in `cat HOSTLIST`; do
  echo "*** $HOST"
  scp -p ${GITREPO}/evaluate.sh ${HOST}:
  ssh -qt $HOST './evaluate.sh' >evaluate-post/${HOST}.out 2>evaluate-pre/${HOST}.err
done

Hortonworks

These are shell scripts to deploy Hortonworks Ambari to a cluster. The goal of these scripts are to be idempotent and to serve as a template for translation into other Configuration Management frameworks/languages.

  • Works with RHEL/CentOS 6 or 7 x86_64.
  • Works with Ubuntu Trusty 14.04 x86_64.
  • Allows for installation of Oracle JDK 8 from Oracle or OpenJDK 7 or 8.

This is an example of some of the functionality. Not everything is documented. Some scripts have arguments that can be passed to them to change their internal operation. Read the source to learn more.

Prep

This is needed for both the Evaluation and Example sections below.

Set the GITREPO variable to the local directory where you have cloned this repository and create a file with SSH login and hostname.

GITREPO=~/git/teamclairvoyant/bash

cat <<EOF >HOSTLIST
centos@ip-10-2-6-62.ec2.internal
centos@ip-10-2-6-5.ec2.internal
centos@ip-10-2-5-22.ec2.internal
EOF

Evaluation

Run the evaluation script to gather the configuration of all the nodes of the cluster. Save the output in the directory "evaluate-pre".

mkdir evaluate-pre
for HOST in `cat HOSTLIST`; do
  echo "*** $HOST"
  scp -p ${GITREPO}/evaluate.sh ${HOST}:
  ssh -qt $HOST './evaluate.sh' >evaluate-pre/${HOST}.out 2>evaluate-pre/${HOST}.err
done

Example

Copy several of the scripts to the nodes.

for HOST in `cat HOSTLIST`; do
  echo "*** $HOST"
  scp -p \
  ${GITREPO}/install_tools.sh \
  ${GITREPO}/change_swappiness.sh \
  ${GITREPO}/disable_iptables.sh \
  ${GITREPO}/disable_ipv6.sh \
  ${GITREPO}/disable_selinux.sh \
  ${GITREPO}/disable_thp.sh \
  ${GITREPO}/install_ntp.sh \
  ${GITREPO}/install_nscd.sh \
  ${GITREPO}/install_jdk.sh \
  ${GITREPO}/configure_javahome.sh \
  ${GITREPO}/install_jce.sh \
  ${GITREPO}/install_krb5.sh \
  ${GITREPO}/configure_tuned.sh \
  ${GITREPO}/install_entropy.sh \
  ${GITREPO}/install_jdbc.sh \
  ${GITREPO}/install_jdbc_sqoop.sh \
  ${GITREPO}/install_hortonworksambariagent.sh \
  $HOST:
done

Run the scripts to prep the system for Hortonworks Ambari installation. Pin the version of Hortonworks Ambari to the value in $HAVER. Also deploy OpenJDK 8.

#BOPT="-x"    # Turn on bash debugging.
HAVER=2.5.2.0 # Set specific Hortonworks Ambari version
for HOST in `cat HOSTLIST`; do
  echo "*** $HOST"
  ssh -t $HOST " \
  sudo bash $BOPT ./install_tools.sh; \
  sudo bash $BOPT ./change_swappiness.sh; \
  sudo bash $BOPT ./disable_iptables.sh; \
  sudo bash $BOPT ./disable_ipv6.sh; \
  sudo bash $BOPT ./disable_selinux.sh; \
  sudo bash $BOPT ./disable_thp.sh; \
  sudo bash $BOPT ./install_ntp.sh; \
  sudo bash $BOPT ./install_nscd.sh; \
  sudo bash $BOPT ./install_jdk.sh --jdktype openjdk --jdkversion 8; \
  sudo bash $BOPT ./configure_javahome.sh; \
  sudo bash $BOPT ./install_jce.sh; \
  sudo bash $BOPT ./install_krb5.sh; \
  sudo bash $BOPT ./configure_tuned.sh; \
  sudo bash $BOPT ./install_entropy.sh"
done

Install the Hortonworks Ambari agent.

HASERVER=ip-10-2-5-22.ec2.internal
for HOST in `cat HOSTLIST`; do
  echo "*** $HOST"
  ssh -t $HOST "sudo bash $BOPT ./install_hortonworksambariagent.sh $HASERVER $HAVER"
done

Install the Hortonworks Ambari server with the embedded PostgreSQL database.

scp -p ${GITREPO}/install_hortonworksambariserver.sh ${HASERVER}:
ssh -t ${HASERVER} "sudo bash $BOPT ./install_hortonworksambariserver.sh embedded $HAVER"

You can use the argument embedded, postgresql, mysql, or oracle.

Post Evaluation

Run the evaluation script again to gather the new configuration of all the nodes of the cluster. Save the output in the directory "evaluate-post".

mkdir evaluate-post
for HOST in `cat HOSTLIST`; do
  echo "*** $HOST"
  scp -p ${GITREPO}/evaluate.sh ${HOST}:
  ssh -qt $HOST './evaluate.sh' >evaluate-post/${HOST}.out 2>evaluate-pre/${HOST}.err
done

Contributing to this project

Everyone is welcome to contribute. Please take a moment to review the guidelines for contributing.

License

Copyright (C) 2015 Clairvoyant, LLC.

Licensed under the Apache License, Version 2.0.

You can’t perform that action at this time.