TutorialAMQP

Hyperflow with AMQP executor tutorial

Installation

This tutorial is based on Ubuntu 14.04 preview release. Here we will assume that everything is installed on one node.

Prerequisites

• node.js and npm
  • sudo apt-get install nodejs nodejs-dev npm
  • Use nvm if package not available
• Redis
  • sudo apt-get install redis-server
  • Required version >= 2.8
• RabbitMQ
  • sudo apt-get install rabbitmq-server
• Ruby
  • sudo apt-get install ruby2.0 ruby2.0-dev build-essential libxml2-dev libxslt1-dev
  • Required version >= 2.0.0
  • Use rvm if package not available.
• Git
  • sudo apt-get install git

Install Hyperflow

# We will use development branch
git clone https://github.com/dice-cyfronet/hyperflow.git --depth 1 -b develop
cd hyperflow
npm install

Install executor

curl -O https://dl.dropboxusercontent.com/u/81819/hyperflow-amqp-executor.gem
sudo gem2.0 install --no-ri --no-rdoc hyperflow-amqp-executor.gem # or simply gem if you are using different distribution

Configuration

Redis and RabbitMQ servers

We will stay with defaults for Redis and RabbitMQ, that means no security or authentication. In real deployment you should really take care of that.

AMQP executor

Create executor_config.yml file with the following contents:

amqp_url: amqp://localhost
storage: local
threads: <%= Executor::cpu_count %>

Then you should be able to start executor:

$ hyperflow-amqp-executor executor_config.yml
I, [2014-02-18T17:29:17.069743 #26656]  INFO -- : Starting worker 5eae5b8a-4d73-4d18-b741-126fdcacede2
I, [2014-02-18T17:29:17.075426 #26656]  INFO -- : Running 1 worker threads
I, [2014-02-18T17:29:17.079106 #26656]  INFO -- : Connected to AMQP broker...
D, [2014-02-18T17:29:17.082622 #26656] DEBUG -- : Publishing event executor.ready

Montage workflow

• Install Montage binaries

  curl -O http://pegasus.isi.edu/montage/Montage_v3.3_patched_4.tar.gz
  tar zxvf Montage_v3.3_patched_4.tar.gz
  cd Montage_v3.3_patched_4
  make
  export PATH=$PATH:`pwd`/bin
  cd ..

• Download example data

  mkdir data
  cd data
  curl -O https://gist.github.com/kfigiela/9075623/raw/dacb862176e9d576c1b23f6a243f9fa318c74bce/bootstrap.sh && chmod +x bootstrap.sh
  ./bootstrap.sh 0.25

• Convert workflow DAX do JSON:

  nodejs ~/hyperflow/scripts/dax_convert_amqp.js 0.25/workdir/dag.xml > 0.25/workdir/dag.json

• Set data path Edit ~/hyperflow/functions/amqpCommand.config.js to look like this

  var AMQP_URL  = process.env.AMQP_URL ? process.env.AMQP_URL : "amqp://localhost:5672";
  
  exports.amqp_url = AMQP_URL;
  
  exports.options = {
      "storage": "local",
      "workdir": "/home/vagrant/data/0.25/input", // Remember to adjust path here
  }

• Run the workflow! nodejs ~/hyperflow/scripts/runwf.js -f ~/data/0.25/workdir/dag.json -s