Skip to content
This repository has been archived by the owner on May 26, 2018. It is now read-only.

Commit

Permalink
merge
Browse files Browse the repository at this point in the history
  • Loading branch information
ikit committed May 2, 2017
2 parents ee6d594 + 364d8ea commit e26b873
Show file tree
Hide file tree
Showing 7 changed files with 228 additions and 30 deletions.
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,6 @@

Pirus is an application launcher service which allow user to run pipeline code from a remote REST client. Each pipeline are provided as plugin made by the community. Pirus runs code in a LXC container to keep your server safe.

The latest version of the installation guide is at [docs/installation_guide.md](docs/installation_guide.md).

[![Documentation Status](https://readthedocs.org/projects/pirus/badge/?version=latest)](http://pirus.readthedocs.io/en/latest/?badge=latest)
139 changes: 139 additions & 0 deletions docs/installation_guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
## Run Pirus

You can run Pirus on a fresh install of Ubuntu Xenial either on bare metal or in a container (see below).
The following commands starting with a `#` have to be run as root.

Install Pirus dependencies:

# apt update && apt upgrade
# apt install git ca-certificates nginx rabbitmq-server mongodb lxd build-essential libssl-dev libffi-dev python3-dev virtualenv

Setup lxd for Pirus containers (FIXME). `newgrp` permet d'ajouter un groupe à l'utilisateur courant (et non pas de créer un groupe).

# newgrp lxd
# lxd init

You have to configure LXD with `lxd init`:
* Name of the storage backend to use (dir or zfs): dir
* Would you like LXD to be available over the network (yes/no): no
* Do you want to configure the LXD bridge: yes
* Would you like to setup a network bridge for LXD containers now? Yes
* Bridge interface name: (keep default)
* Would you like to setup an IPv4 subnet? Yes
* IPv4 address: (keep default)
* IPv4 CIDR mask: (keep default)
* First DHCP address: (keep default)
* Last DHCP address: (keep default)
* Max number of DHCP clients: (keep default)
* Do you want to NAT the IPv4 traffic? Yes
* Do you want to setup an IPv6 subnet? No

Add an user account for Pirus and allow it to use lxd:

# useradd pirus --create-home
# usermod -a -G lxd pirus

Create Pirus directories:

# mkdir -p /var/regovar/pirus/{cache,downloads,files,databases,pipelines,runs}
# chown -R pirus:pirus /var/regovar/pirus

Launch a LXD container to get an Ubuntu Xenial image. This will generate a client certificate and make the subsequent container creations faster.

# su pirus
$ lxc launch images:ubuntu/xenial firstContainerToInitLxd
$ lxc delete firstContainerToInitLxd --force

Get the Pirus source code and assets:

$ git clone https://github.com/REGOVAR/Pirus.git ~/Pirus
$ cd ~/Pirus

Create a virtual environment to use Python 3.5 without conflicting with other setups:

$ virtualenv -p /usr/bin/python3.5 venv

Activate the virtual environment:

$ source venv/bin/activate

Install Pirus Python dependencies:

$ pip install -r requirements.txt

Launch Celery to keep track of jobs:

$ cd pirus
$ make cel &!

Launch Pirus itself:

$ make app &!

Leave the pirus user session (Celery and the Pirus application are still running); this leaves the virtual environment at the same time:

$ exit

Configure nginx as a reverse proxy for the Pirus application and static assets:

# echo 'upstream aiohttp_pirus
{
server 127.0.0.1:8200 fail_timeout=0;
}
server
{
listen 80;
listen [::]:80;

location / {
# Need for websockets
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_pass http://aiohttp_pirus;
}

location /static {
root /var/regovar/pirus;
}
}' > /etc/nginx/sites-available/pirus

Disable the "Welcome to nginx!" page on port 80:

# rm /etc/nginx/sites-enabled/default

Enable the pirus site in nginx on port 80:

# ln -s /etc/nginx/sites-available/pirus /etc/nginx/sites-enabled

Restart nginx:

# service nginx restart

## Run Pirus in a container (optional and experimental)

### Run containers inside containers
You only to do this step once when you want to install Pirus for the first time.

$ echo 'lxc.mount.auto = cgroup
lxc.aa_profile = lxc-container-default-with-nesting' >> ~/.config/lxc/default.conf

"The first will cause the cgroup manager socket to be bound into the container, so that lxc inside the container is able to administer cgroups for its nested containers. The second causes the container to run in a looser Apparmor policy which allows the container to do the mounting required for starting containers. Note that this policy, when used with a privileged container, is much less safe than the regular policy or an unprivileged container." See [LXC documentation on Ubuntu help](https://help.ubuntu.com/lts/serverguide/lxc.html).

### Create a lxc container and start it
You need to do these steps every time you want to install Pirus in a container.

$ lxc-create -n regovar_pirus -t download -- -d ubuntu -r xenial -a amd64
$ lxc-start -n regovar_pirus
$ lxc-attach -n regovar_pirus

### Restart a stopped container
If you have stopped a container either manually or by stopping the host computer, you can restart it.

$ lxc-start -n regovar_pirus
$ lxc-attach -n regovar_pirus
57 changes: 57 additions & 0 deletions docs/installation_script
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
Optional, if you want to wrap Pirus into a lxc container:
$ echo 'lxc.mount.auto = cgroup
lxc.aa_profile = lxc-container-default-with-nesting' >> ~/.config/lxc/default.conf
$ lxc-create -n regovar_pirus -t download -- -d ubuntu -r xenial -a amd64
$ lxc-start -n regovar_pirus
$ lxc-attach -n regovar_pirus

Installation script for Pirus on a fresh Ubuntu Xenial:
# apt update && apt upgrade
# apt install git ca-certificates nginx rabbitmq-server mongodb lxd build-essential libssl-dev libffi-dev python3-dev virtualenv
# newgrp lxd
# lxd init
# useradd pirus --create-home
# sudo usermod -a -G lxd pirus
# mkdir -p /var/regovar/pirus/{cache,downloads,files,databases,pipelines,runs}
# chown -R pirus:pirus /var/regovar/pirus
# su pirus
$ lxc launch images:ubuntu/xenial firstContainerToInitLxd
$ lxc delete firstContainerToInitLxd --force
$ git clone https://github.com/REGOVAR/Pirus.git ~/Pirus
$ cd ~/Pirus
$ virtualenv -p /usr/bin/python3.5 venv
$ source venv/bin/activate
$ pip install -r requirements.txt
$ cd pirus
$ make cel &!
$ make app &!
$ exit
# echo 'upstream aiohttp_pirus
{
server 127.0.0.1:8200 fail_timeout=0;
}
server
{
listen 80;
listen [::]:80;

location / {
# Need for websockets
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_pass http://aiohttp_pirus;
}

location /static {
root /var/regovar/pirus;
}
}' > /etc/nginx/sites-available/pirus
# rm /etc/nginx/sites-enabled/default
# ln -s /etc/nginx/sites-available/pirus /etc/nginx/sites-enabled
# /etc/init.d/nginx restart
8 changes: 4 additions & 4 deletions examples/pipelines/PirusBasic/form.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"description": "Sélectionner lequel est le `file1` parmis ceux que vous avez sélectionné pour le run.",
"type": "string",
"enum": "__PIRUS_INPUT_FILES__",
"required": true
"required": false
},
"file2":
{
Expand All @@ -25,14 +25,14 @@
"description": "Choisissez votre génome de référence parmis les base de données disponible",
"type": "string",
"enum": "__PIRUS_DB_ALL__",
"required": true
"required": false
},
"confirm":
{
"title": "Vous êtes sûr ?",
"description": "En cochant cette case vous vous engagez blablabla.",
"type": "boolean",
"default": true
"default": false
},
"witherror":
{
Expand All @@ -46,7 +46,7 @@
"title": "Durée du run",
"description": "Indiquez le nombre de seconde que va durer le run.",
"type": "integer",
"required": true,
"required": false,
"default" : 100
},
"outfilename":
Expand Down
26 changes: 13 additions & 13 deletions examples/pipelines/PirusBasic/readme.md
Original file line number Diff line number Diff line change
@@ -1,44 +1,44 @@
# PirusBasic Pipeline

This document explain you how to build a simple pipeline image for Pirus.
This document explain you how to build a simple pipeline image for Pirus.

## Requirement
* You need LXD on your computer to create it
* You should read the official doc of Pirus

##Instructions
## Instructions

# create a container
lxc launch images:ubuntu/xenial pirus
# configure it
lxc exec pirus -- /bin/bash

# following directories are mandatory
mkdir /pipeline/
mkdir /pipeline/run
mkdir /pipeline/inputs
mkdir /pipeline/outputs
mkdir /pipeline/logs
mkdir /pipeline/db

# need curl if you want to notify server with the progress of your run
apt install curl jq nano --fix-missing

# Create the script run.sh. this will be the "entry point" of your run
# An example can be found on github (https://github.com/REGOVAR/Pirus/blob/master/examples/pipelines/PirusBasic/run.sh)
nano /pipeline/run/run.sh
chmod +x /pipeline/run/run.sh
# To allow users to configure your pipeline, you shall put in your container a form.json file

# To allow users to configure your pipeline, you shall put in your container a form.json file
# that will describe a form to set parameter for your pipe.
# An example can be found on github (https://github.com/REGOVAR/Pirus/blob/master/examples/pipelines/PirusBasic/form.json)
nano /pipeline/form.json

# You can also put a a custom logo (png or jpeg file) in your pipeline.

# exit the container
exit

# stop it and create an image
lxc stop pirus
lxc publish pirus --alias=PirusSimple
Expand All @@ -49,13 +49,13 @@ This document explain you how to build a simple pipeline image for Pirus.
## Export image as file and edit image conf to create a piruse package installable on any pirus server

lxc image export PirusSimple
# following command shall be done as root to avoid image corruption
# following command shall be done as root to avoid image corruption
# (as it will try to create symlink to computer resource in /dev folder by example)
sudo tar xf <the_name_of_lxc_export_something_like_a8d44d24fcs...8fzef54e5>.tar.gz

# add folowing informations into the metadata.yaml file
sudo nano metadata.yaml

# if json
"pirus":
{
Expand Down Expand Up @@ -89,7 +89,7 @@ This document explain you how to build a simple pipeline image for Pirus.
form: "/pipeline/form.json"
icon: "/pipeline/logo.png"


# You can repackage the image in tar.xz, to save space
sudo tar cfJ PirusSimple.tar.xz metadata.yaml rootfs templates
sudo rm -fr metadata.yaml rootfs templates
Expand Down
22 changes: 11 additions & 11 deletions pirus/api_rest/routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@


# Routes
app.router.add_route('GET', "/", website.home)
app.router.add_route('GET', "/", website.home)
app.router.add_route('GET', "/www", website.home)
app.router.add_route('GET', "/config", website.get_config)
app.router.add_route('GET', "/db", website.get_db)
Expand Down Expand Up @@ -64,18 +64,18 @@
app.router.add_route('GET', "/run/{run_id}/monitoring", runHdl.get_monitoring)
#app.router.add_route('GET', "/run/{run_id}/{filename}", fileHdl.dl_run_file)

app.router.add_route('GET', "/file", fileHdl.get)
app.router.add_route('DELETE', "/file/{file_id}", fileHdl.delete)
app.router.add_route('PUT', "/file/{file_id}", fileHdl.edit_infos)
app.router.add_route('GET', "/file/{file_id}", fileHdl.get_details)
app.router.add_route('POST', "/file/upload", fileHdl.tus_upload_init)
app.router.add_route('OPTIONS',"/file/upload", fileHdl.tus_config)
app.router.add_route('HEAD', "/file/upload/{file_id}", fileHdl.tus_upload_resume)
app.router.add_route('PATCH', "/file/upload/{file_id}", fileHdl.tus_upload_chunk)
app.router.add_route('DELETE', "/file/upload/{file_id}", fileHdl.tus_upload_delete)
app.router.add_route('GET', "/v1/file", fileHdl.get)
app.router.add_route('DELETE', "/v1/file/{file_id}", fileHdl.delete)
app.router.add_route('PUT', "/v1/file/{file_id}", fileHdl.edit_infos)
app.router.add_route('GET', "/v1/file/{file_id}", fileHdl.get_details)
app.router.add_route('POST', "/v1/file/upload", fileHdl.tus_upload_init)
app.router.add_route('OPTIONS',"/v1/file/upload", fileHdl.tus_config)
app.router.add_route('HEAD', "/v1/file/upload/{file_id}", fileHdl.tus_upload_resume)
app.router.add_route('PATCH', "/v1/file/upload/{file_id}", fileHdl.tus_upload_chunk)
app.router.add_route('DELETE', "/v1/file/upload/{file_id}", fileHdl.tus_upload_delete)

# Websockets / realtime notification
app.router.add_route('POST', "/run/notify/{run_id}", runHdl.update_status)
app.router.add_route('POST', "/v1/run/notify/{run_id}", runHdl.update_status)


# DEV/DEBUG - Routes that should be manages directly by NginX
Expand Down
4 changes: 2 additions & 2 deletions pirus/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@

# HOST (internal)
HOST = "127.0.0.1"
PORT = "8090"
PORT = "8200"
VERSION = "v1"
HOSTNAME = HOST + ":" + PORT + "/" + VERSION

# HOST (public)
HOST_P = "pirus.absolumentg.fr"
HOST_P = HOSTNAME

RANGE_DEFAULT = 20
RANGE_MAX = 1000
Expand Down

0 comments on commit e26b873

Please sign in to comment.