[DEPRECATED] Boot Mesos with Vagrant
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
config-files
README.org
Vagrantfile
bootstrap-docker.sh
bootstrap-hadoop-setup.sh
bootstrap-master.sh
bootstrap-mesos-setup.sh
bootstrap-mesoshdfsnode.sh

README.org

Vagrant Mesos (2 master/2 slave setup)

This Vagrant installation sets up a minimal Big Data Europe platform with two masters and two slaves. This installation can be used for getting a feel of the platform without a lengthy installation procedure.

The Big Data Europe platform provides a generic platform for deploying Big Data pipelines. The instances providing the computation are offered through docker images. See http://github.com/big-data-europe/ for examples.

Requirements

You should ensure that you have the following available in order to use the platform:

Hardware requirements:

  • 10Gb RAM
  • 30Gb Disk space

Software requirements:

Starting the cluster

  • cd <clone directory>
  • vagrant up

After first boot: restart the Vagrant instances to correct the network configuration.

  • vagrant halt
  • vagrant up

You can ssh into the machines using the regular Vagrant commands:

vagrant ssh slave1

Standard configuration

The base cluster publishes the following nodes:

HOSTNAMEIPRAM
master1192.168.0.104GB
master2192.168.0.114GB
slave1192.168.0.52GB
slave2192.168.0.62GB

The Vagrantfile should be clear to understand and fairly easy to alter.

Playing with the cluster

The virtual cluster can be used for deploying applications and for testing the rigidity of the platform.

Deploying applications

See the relevant applications on http://github.com/big-data-europe/ for more information on how to deploy applications on this cluster.

Trolling the cluster

Big Data clusters are one of the few machines which like to be trolled. Feeling frustrated, why don’t you kill a slave node? Need a bigger up, kill the master. Clustered systems should stay alive in these cases. This minimal setup allows you to play with that usecase.

  • Visit http://192.168.0.10:5050/ which should show 2 actives slaves (shutting down a slave will leave to it becoming deactivated, etc.)
  • Start the system, access the link and check that 2 slaves are visible
  • Shut down one of the slaves (and watch it pass from deactivated to gone)
  • Restart the slave and check it becomes active again
  • Power down the master and check the other master continues

Starting optional Hadoop services

Hadoop services are not started on boot. In order to use HDFS or YARN, execute the following as hduser:

  • start-dfs.sh
  • start-yarn.sh
  • jps