Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
bin
 
 
ec2
 
 
hadoop @ 7ef4d94
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

docker-hadoop-cluster Build Status

Multiple node cluster on Docker for self development. docker-hadoop-cluster is suitable for testing your patch for Hadoop which has multiple nodes.

Build images from your Hadoop source code

Base image of hadoop service. This image includes JDK, hadoop package configurations etc. This image can include your self-build hadoop package. In order to bind, tar.gz package assumed be put under hadoop-base directory.

$ cd docker-hadoop-cluster
$ cp /path/to/hadoop-3.0.0-alpha3-SNAPSHOT.tar.gz hadoop-base
$ make

Once you build hadoop-base image, you can launch hadoop cluster by using docker-compose.

$ docker-compose up -d

or

$ make run

See http://localhost:9870 for NameNode or http://localhost:8088 for ResourceManager.

$ make down

shutdowns your cluster.

Build images from the latest trunk

docker-hadoop-cluster also uploads the latest image which refers HEAD of trunk. They are deployed on Docker Hub. If you want to try the trunk (though it can be unstable), docker-compose.yml like below is needed. It will launch 3 slave Hadoop cluster.

version: '2'

services:
  master:
    image: lewuathe/hadoop-master
    ports:
      - "9870:9870"
      - "8088:8088"
      - "19888:19888"
      - "8188:8188"
    container_name: "master"
  slave1:
    image: lewuathe/hadoop-slave
    container_name: "slave1"
    depends_on:
      - master
    ports:
      - "9901:9864"
      - "8041:8042"
  slave2:
    image: lewuathe/hadoop-slave
    container_name: "slave2"
    depends_on:
      - master
    ports:
      - "9902:9864"
      - "8042:8042"
  slave3:
    image: lewuathe/hadoop-slave
    container_name: "slave3"
    depends_on:
      - master
    ports:
      - "9903:9864"
      - "8043:8042"

Login cluster

$ docker exec -it master bash
bash-4.1# cd /usr/local/hadoop
bash-4.1# bin/hadoop version
Hadoop 3.0.0-SNAPSHOT
Source code repository git://git.apache.org/hadoop.git -r 0c7d3f480548745e9e9ccad1d318371c020c3003
Compiled by lewuathe on 2015-09-13T01:12Z
Compiled with protoc 2.5.0
From source with checksum 9174a352ac823cdfa576f525665e99
This command was run using /usr/local/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar

Deploy on EC2

We can run docker-hadoop-cluster on EC2 instance with ec2/ script.

$ python ec2/ec2.py -k <Key Name> -s <Security Group ID> -n <Subnet ID> launch

This script launch EC2 instance and prepare prerequisites to launch docker-hadoop-cluster.

Docker Hub

Image name Pulls Stars
lewuathe/hadoop-base hadoop-base hadoop-base
lewuathe/hadoop-master hadoop-master hadoop-master
lewuathe/hadoop-slave hadoop-slave hadoop-slave

License

Apache License Version2.0 This images are modified version of sequenceiq/hadoop-docker.

About

Multiple node cluster on Docker for self development.

Topics

Resources

License

Releases

No releases published

Packages

No packages published