Tool for setting up Metapipe on Spark Cluster
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
pouta-ansible-cluster
provision
templates
.gitignore
LICENSE
README.md
create.py
destroy.py
mmg-cluster.tf
run.py
stop.py

README.md

mmg-cluster-setup

This is tool for setting up Metapipe in OCCI enabled endpoint. Uses code from mmg-cluster-setup.
Currently in development.

Requirements:

File structure:

  • "mmg-cluster.tf": Contains Terraform configuration
  • "pouta-ansible-cluster": Folder containing Ansible playbooks
  • "provision": Folder containing installation scripts
  • "create.py": Python script creating and provisioning Metapipe environment
  • "run.py": Python script running Metapipe with custom job tag
  • "destroy.py": Python script destroying Metapipe environment
  • "stop.py": Python script stopping Metapipe on cluster

What it can do:

  • Create 1 master and x slave nodes with Terraform at OCCI endpoint (currently tested only in CESNET MetaCloud. Might not work at other endpoints due to different CentOS 7 images)
  • Provision all hosts with Ansible, copy necessary files to run Metapipe
  • Run Metapipe with custom job tag
  • Cleanup environment

What is missing:

  • Some functionality

How to use:

Recommended way to use Metapipe tool is with Docker