Hadoop Stack

Prerequisite

Require: docker and docker-compose.

Create directory ./resources and download following resource into this directory

• server-jre-8u172-linux-x64.tar.gz
• hadoop-2.7.6.tar.gz
• spark-2.3.0-bin-hadoop2.7.tgz
• ojdbc6.jar
• mysql-connector-java-5.1.45-bin.jar
• apache-hive-2.3.3-bin.tar.gz

Once done build each image

hshekhar47/debian-jre8

debian oracle-server-jre8 passwordless ssh

Build

./build.sh build debian-jre8

hshekhar47/hadoop-core

hshekhar47/debian-jre8 hadoop-2.7.6 spark-2.3.0 sqoop-1.4.7

Build

./build.sh build hadoop-core

hshekhar47/hdfs-namenode

: hshekhar47/hadoop-core

Build

./build.sh build hdfs-namenode

hshekhar47/hdfs-datanode

hshekhar47/hadoop-core

Build

./build.sh build hdfs-datanode

Start

1. Run below command lauch hadoop-stack locally

docker compose -f hadoop-stack.yml up

References

Setup a hadoop cluster

Setup sqoop on hadoop cluster