Tool for setting up Metapipe on Spark Cluster
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


This is tool for setting up Metapipe in OCCI enabled endpoint. Uses code from mmg-cluster-setup.
Currently in development.


File structure:

  • "": Contains Terraform configuration
  • "pouta-ansible-cluster": Folder containing Ansible playbooks
  • "provision": Folder containing installation scripts
  • "": Python script creating and provisioning Metapipe environment
  • "": Python script running Metapipe with custom job tag
  • "": Python script destroying Metapipe environment
  • "": Python script stopping Metapipe on cluster

What it can do:

  • Create 1 master and x slave nodes with Terraform at OCCI endpoint (currently tested only in CESNET MetaCloud. Might not work at other endpoints due to different CentOS 7 images)
  • Provision all hosts with Ansible, copy necessary files to run Metapipe
  • Run Metapipe with custom job tag
  • Cleanup environment

What is missing:

  • Some functionality

How to use:

Recommended way to use Metapipe tool is with Docker