Introduction

vagrant is a useful tool for managing vm's

this project is a simple hadoop setup with vagrant

Install

At first install vagrant

Then install Virtualbox

sudo apt install virtualbox

How to run

just run this command's

** note: you need to a stable internet to have a correct installation

sudo mkdir -p /etc/vbox/
echo "* 10.0.0.0/8 192.168.0.0/16"|sudo tee /etc/vbox/networks.conf
vagrant up

Note:

for Apple Silicon run this commands

docker_branch use docker as a provider instead of virtualbox

git checkout docker_provider
vagrant up

WebUI

• http://10.20.30.11:9870 #hdfs webUI
• http://10.20.30.11:8088 #yarn webUI

Note:

if you are using docker your webUI address will be changed because in docker version project use port_forwarding

• http://localhost:9870 #hdfs webUI
• http://localhost:8088 #yarn webUI

Some Useful Vagrant commands:

• delete vm's with their files

  vagrant destroy

• shutdown vm's

  vagrant halt

• ssh to a specific vm

  vagrant ssh <YOUR-VM-NAME>

  for example:

    vagrant ssh hadoop-master
    vagrant ssh hadoop-worker1

Some Useful Hadoop commands (in master or worker vm's):

after ssh change your unix user to hadoop

sudo su hadoop

now you are able to use below commands!

• ls in hdfs

  hdfs dfs -ls /your/path/to

• make directories in hdfs

  hdfs dfs -mkdir -p /your/path

• put file from your directory to hdfs

  hdfs dfs -put  /path/to/your-file /path/to/hdfs

Optional

run this command to set hosts

sudo sed -i '/hadoop/d' /etc/hosts
echo "
10.20.30.11 hadoop-master
10.20.30.21 hadoop-worker1
10.20.30.22 hadoop-worker2
10.20.30.23 hadoop-worker3
" | sudo tee -a /etc/hosts

References

• hadoop
• hadoop-core-configs
• hadoop-hdfs-configs
• hadoop-yarn-configs