Skip to content
Tool for provisioning GCE hadoop clusters.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
hadoop1_conf
hadoop2_conf
libexec
script_runner
.gitignore
LICENSE
README.md
__init__.py
cluster_config_sample
setup_env.py
zdgcutil.py
zdgsutil.py
zdutil.py

README.md

zdutil

Tool for provisioning GCE hadoop clusters.

Getting started

  • Install gevent
sudo pip install gevent
cp cluster_config_sample cluster_config
  • Edit the settings in the custom settings section of cluster_config

To setup a Hadoop cluster

python zdutil.py -c cluster_config -a setup

To teardown a Hadoop cluster

python zdutil.py -c cluster_config -a teardown

To setup a Hadoop cluster and run your own bash scripts on the namenode

python zdutil.py -c cluster_config -a setup -s <path_to_script1>,<path_to_script2>

To setup a Hadoop cluster, run your own bash scripts on the namenode, and teardown the cluster afterwards

python script_runner/script_runner.py -c cluster_config -z zdutil.py -s <path_to_script1>,<path_to_script2>

Details

Read more about zdutil at http://engineering.zulily.com/2014/12/03/google-compute-engine-hadoop-clusters-with-zdutil/

You can’t perform that action at this time.