Hortonworks Hadoop cluster deployment with Kubernetes

The point is to get the Hadoop cluster with needed components, running as docker container at Kubernetes cluster.

Files: Dockerfile ambari.repo bp.json map.json start.sh must be placed at accessible from hub.docker.com git repository.

1. Create Docker Automated Build.

1.1 Register at hub.docker.com and link your account to your git repository while creating Automated Build

Register at hub.docker.com and link account to github, Create Automated Build, setting git repo smaryn/hdpkub as a "Source Repository"

1.2 Go to the "Build Settings" tab and click "Trigger" button to run docker image building.

The build may take a while and should be finished successfully, you can always check the build status at Build Details tab

2. Prepare kubectl utility to be able to connect to Kubeletes master (198.18.8.161):

2.1 Download kubectl binary file from corresponding resource and place it somewhere in the $PATH:

For Linux workstation:

wget -c https://storage.googleapis.com/kubernetes-release/release/v1.1.8/bin/linux/amd64/kubectl -O /usr/local/bin/kubectl

For Mac OS X workstation:

wget -c https://storage.googleapis.com/kubernetes-release/release/v1.1.8/bin/darwin/amd64/kubectl -O /usr/local/bin/kubectl

2.2 Configure kubectl utility:

Set Kubeletes cluster:

kubectl config set-cluster ubuntu --server=http://198.18.8.161:8080 --insecure-skip-tls-verify=true

Set context:

kubectl config set-context ubuntu --cluster=ubuntu --user=ubuntu

Add user for created cluster and password:

kubectl config set-credentials ubuntu --username=admin --password=HAKlQaGHHmU6W0W

Inform kubectl to use desired context:

kubectl config use-context ubuntu

Check kubectl configuration:

kubectl config view

Check connection to configured cluster:

kubectl get nodes

3. Kubernetes pod and service by prepared yaml files

3.1 Create Kubernetes pod for Hadoop container(s) using hdp.yaml:

kubectl create -f kube/hdp.yaml

3.2 Create Kubernetes service for Hadoop using hdp-service.yaml

At this stage in the command output you should notice the TCP port which is assigned for Ambari server management:

kubectl create -f kube/hdp-service.yaml

3.3 Check created service status:

kubectl logs hdpkub

3.4 Check cluster deployment

Connect to Ambari Management Console

Open web browser and type Kubernetes node IP address and TCP port noticed at 2.3 stage. Use default username and password (admin) to see how cluster is deployed.

3. Cluster termination and Kubernetes pod and service removal

3.1 Delete Kubernetes service:

kubectl delete -f kube/hdp-service.yaml

3.2 Delete Kubernetes pod:

kubectl delete -f kube/hdp.yaml

3.3 Obtain the root shell at the Kubernetes node and remove unused Docker containers and images

List all containers:

docker ps -a

Delete unused containers by ID:

docker rm -f <ID1> <ID2> <...>

List all images:

docker images

Delete uneeded images by ID:

docker rmi -f <ID1> <ID2> <...>