Web API for dynamic compute
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
example
templates
.gitignore
.travis.yml
.zenodo.json
CITATION.cff
LICENSE
README.md
onebuttoncompute.py
requirements.txt
settings.cfg-dist

README.md

One button compute

Build Status DOI

One button compute is aweb site that runs a workflow.

Feature/Limitations

  • Workflow is a single file in Common Workflow format
  • Workflow must take single input file (--input option) and generates a single output file (--output option)
  • Web application runs workflow on directory of input files
  • The workflow, the directory with input files is downloaded from a remote storage server
  • The directory with output files is uploaded to a remote storage server

The remote storage server can be WebDAV or S3 or Swift.

Requirements

  • Python2
  • Docker
  • Read/write access to a remote storage server. Can be a WebDAV or S3 or Swift server/account.

Install

1. Redis server

Redis server is used to perform computation asynchronous from http request.

Use Docker to start a redis server

docker run -d -p 6379:6379 redis

Note!: When Celery workers are going to be run on different machines make sure they can connect to the redis server.

2. Install dependencies

Install the Python dependencies with

pip install -r requirements.txt

3. Configure the application

cp settings.cfg-dist settings.cfg

Configure remote storage type, location and credentials in settings.cfg.

S3 development server (optional)

A Minio server can be started with

mkdir -p minio/export
docker run -d --name obc-minio -p 9000:9000 -v $PWD/minio:/root minio/minio /root/export
docker logs obc-minio

The log output contains the credentials, urls and access instructions.

To use known credentials from settings.cfg start it with

docker run -d --name obc-minio -p 9000:9000 -v $PWD/minio:/root -e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" \
  -e "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" minio/minio /root/export

Use mc (https://docs.minio.io/docs/minio-client-quickstart-guide) as CLI client.

WebDAV development server (optional)

A WebDAV server can be started with

docker run -d nlesc/xenon-webdav

Read/write can be done in ~/xenon/uploads path with xenon:javagat credentials.

Use cadaver (http://www.webdav.org/cadaver/) as CLI client.

Reverse proxy (optional)

Configure Nginx as reverse proxy for the flask app port 5000.

location / {
  proxy_pass http://localhost:5000;
}

Auto start (optional)

Automatically start one-button-compute on boot with upstart file

cat /etc/init/onebuttoncompute.conf
# Running on port 5000

description "One button compute"

start on filesystem or runlevel [2345]
stop on runlevel [!2345]

script
  cd /opt/one-button-compute
  python onebuttoncompute.py
end script

Run

Start Celery worker and web server with

celery worker -A onebuttoncompute.celery &
python onebuttoncompute.py

Usage

Add a CWL workflow and input files to remote storage. See example/ sub-directory for an example workflow.

Go to http://localhost:5000/ (or http://<server-name>/ when reverse proxy is setup) to submit a computation.