Skip to content
A heat template to deploy Apache Hadoop on Rackspace Private Clouds
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
assets
LICENSE
README.md
env.yaml
hadoop_stack.yaml
minion_stack.yaml
wizard.yaml

README.md

Description

This is an OpenStack HEAT template to deploy Hadoop to multiple servers in an OpenStack cloud. Currently, only Hortonworks HDP 1.3 has been tested. Support for Apache Hadoop and Cloudera CDH distros and Hadoop YARN may be added in the future.

This template uses hadoop salt-formulas to configure the servers. It deploys a salt-master, and a number of salt-minions. One minion will be the Hadoop master node. Rest of the minions will be Hadoop data nodes. The number of data nodes can be specified during the stack creation.

For access to nodes in the Hadoop cluster, a floating ip will be assigned to the salt-master. Or a new server can be created in the same network as the Hadoop network.

Any changes to the Hadoop configuration can be done using Salt pillars on the Salt master. Adding or decommissioning datanodes can be done via Salt as well.

Requirements

  • A Heat provider that supports the following:

    • OS::Neutron::Net
    • OS::Neutron::Subnet
    • OS::Neutron::Router
    • OS::Neutron::RouterInterface
    • OS::Neutron::FloatingIP
    • OS::Neutron::FloatingIPAssociation
    • OS::Neutron::Port
    • OS::Heat::SoftwareConfig
    • OS::Heat::SoftwareDeployment
    • OS::Heat::RandomString
    • OS::Heat::ResourceGroup
    • OS::Nova::Server
    • OS::Nova::KeyPair
  • An Ubuntu image (12.04 or newer) preconfigured with heat-cfntools and heat config-script. Instructions for creating a heat-cfntools enabled image for use with Heat can be found [here] (http://docs.openstack.org/developer/heat/getting_started/jeos_building.html).

  • An OpenStack username, password, and tenant id.

  • python-heatclient >= v0.2.12:

pip install python-heatclient

Heat-client Usage

Here is an example of how to deploy this template using the python-heatclient:

heat stack-create hadoop-stack -f hadoop-stack.yaml \
  -e env.yaml -P flavor=m1.large;floating-network-id=<NET_ID>; \
  datanodes-count=<COUNT>;keyname=<KEYNAME>;image=<IMAGE_ID>

Using Horizon

You can also go to your Horizon Dashboard in your browser and create the Hadoop stack from under the Orchestration tab.

License

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
You can’t perform that action at this time.