Permalink
Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
204 lines (153 sloc) 6.25 KB

REANA example - "hello world"

About

This repository provides a simple "hello world" application example for REANA reusable research data analysis plaftorm. The example generates greetings for persons specified in an input file.

Analysis structure

Making a research data analysis reproducible basically means to provide "runnable recipes" addressing (1) where is the input data, (2) what software was used to analyse the data, (3) which computing environments were used to run the software and (4) which computational workflow steps were taken to run the analysis. This will permit to instantiate the analysis on the computational cloud and run the analysis to obtain (5) output results.

1. Input dataset

The input file is a text file containing person names:

$ cat data/names.txt
Jane Doe
Joe Bloggs

2. Analysis code

A simple "hello world" Python code for illustration:

The program takes an input file with the names of the persons to greet and an optional sleeptime period and produces an output file with the greetings:

$ python code/helloworld.py \
    --inputfile data/names.txt
    --outputfile results/greetings.txt
    --sleeptime 0
$ cat results/greetings.txt
Hello Jane Doe!
Hello Joe Bloggs!

3. Compute environment

In order to be able to rerun the analysis even several years in the future, we need to "encapsulate the current compute environment", for example to freeze the Jupyter notebook version and the notebook kernel that our analysis was using. We shall achieve this by preparing a Docker container image for our analysis steps.

For example:

$ cat environments/python/Dockerfile
FROM python:2.7

Since we don't need any additional Python packages for this simple example to work, we can directly rely on the python image from the Docker Hub. The trailing :2.7 makes sure we are using Python 2.7.

4. Analysis workflow

This analysis is very simple because it consists basically of running only the Python executable which will produce a text file.

The workflow can be represented as follows:

            START
            |
            |
            V
+-------------------------------+
|                               | <-- inputfile=data/names.txt
|    $ python helloworld.py ... | <-- sleeptime=2
+-------------------------------+
            |
            | otuputfile=results/greetings.txt
            V
            STOP

Note that you can also use CWL or Yadage workflow specifications:

5. Output results

The example produces a file greeting all names included in the names.txt file.

Hello Jane Doe!
Hello Joe Bloggs!

Running the example on REANA cloud

We are now ready to run this example and on the REANA cloud.

First we need to create a reana.yaml file describing the structure of our analysis with its inputs, the code, the runtime environment, the computational workflow steps and the expected outputs:

version: 0.3.0
inputs:
  files:
    - code/helloworld.py
    - data/names.txt
  parameters:
    sleeptime: 2
    inputfile: data/names.txt
    helloworld: code/helloworld.py
    outputfile: results/greetings.txt
workflow:
  type: serial
  specification:
    steps:
      - environment: 'python:2.7'
        commands:
          - python "${helloworld}" --sleeptime ${sleeptime} --inputfile "${inputfile}" --outputfile "${outputfile}"
outputs:
  files:
   - results/greetings.txt

In case you are using CWL or Yadage workflow specifications:

We can now install the REANA command-line client, run the analysis and download the resulting file:

$ # create new virtual environment
$ virtualenv ~/.virtualenvs/myreana
$ source ~/.virtualenvs/myreana/bin/activate
$ # install REANA client
$ pip install reana-client
$ # connect to some REANA cloud instance
$ export REANA_SERVER_URL=https://reana.cern.ch/
$ export REANA_ACCESS_TOKEN=XXXXXXX
$ # create a new workflow
$ reana-client create -n my-analysis
$ export REANA_WORKON=my-analysis
$ # upload input code and data to the workspace
$ reana-client upload ./data ./code
$ reana-client list
$ # start computational workflow
$ reana-client start
$ # should take about a minute
$ reana-client status
$ # list workspace files
$ reana-client list
$ # download output results
$ reana-client download results/greetings.txt

Please see the REANA-Client documentation for more detailed explanation of typical reana-client usage scenarios.

Contributors

The list of contributors in alphabetical order: