Skip to content
This repository has been archived by the owner on May 26, 2018. It is now read-only.

Commit

Permalink
fix doc and demo
Browse files Browse the repository at this point in the history
  • Loading branch information
ikit committed Mar 14, 2017
2 parents ede3446 + 4cc45eb commit 2491cd5
Show file tree
Hide file tree
Showing 15 changed files with 376 additions and 176 deletions.
1 change: 1 addition & 0 deletions docs/devguide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Solution organisation
* One branch by release; with the version number as name (by example branch "v1.0.0" for the v1.0.0)
* Discussion :
* https://regovar.slack.com/
* dev@regovar.org



Expand Down
54 changes: 54 additions & 0 deletions docs/quickguide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,60 @@ Enable this virtual host by creating a symbolic link ::
sudo /etc/init.d/nginx restart

Installing genomic databases
----------------------------
According to the config file of the Pirus application, you will install databases in the folder /var/pirus/databases by examples. You have to put in this directory all heavy databases used by pipes. The organisation shall stay simple, one folder by reference ::

/var/pirus/databases
/hg19
hg19.fa
1000g.vcf.gz
1000g.vcf.gz.tbi
... <- all other files that could be used by pipelines
/hg38
hg38.fa
...
Below the command to get all files for hg19 from the gatk public repository ::

mkdir -p /var/pirus/databases/hg19
cd /var/pirus/databases/hg19
nohup wget ftp://gsapubftp-anonymous@ftp.broadinstitute.org/bundle/hg19/ -r &
# nohup allow the long task to run without bocking your shell session
# you can follow the execution by looking the log
tail -f nohup.out # Ctrl+C to quit
# meanwhile, downloading hg19.fa ref
wget http://hgdownload.soe.ucsc.edu/goldenPath/hg19/bigZips/hg19.2bit
md5sum hg19.2bit # compare fingerprint with online md5
# TODO : choice 1 : installing bioinfo tools on the server or choice 2 : download all file from a directory ?
twoBitToFa hg19.2bit hg19.fa
samtools faidx hg19.fa
bwa index hg19.fa
# when all other downloads are completed (look into nohup.out file)
mv ftp.broadinstitute.org/bundle/hg19/* .
# If you don't trust your connection, you can check if file are not corrupted...
# unfortunately, md5 provided by the broad institute are not for the good files :P
cat *.md5 >> all.md5
sed -i 's/humgen\/gsa-scr1\/pub\/bundle\/2.8\/hg19/var\/pirus\/databases\/hg19/' all.md5
md5sum -c all.md5
# Unfortunately -again-, all gz file in the gatk ftp are not in bzip format... so, to be used
# by bioinformatic's pipelines, we need to redo compression with the good algorithm
# To get the bzip tool, you need to get and compile Htslib (https://github.com/samtools/htslib)
gzip -d *.vcf.gz
rm *.vcf.idx.gz
rm *.md5
for i in `ls -L *.vcf`; bgzip $i;
for i in `ls -L *.vcf.gz`; tabix -p vcf $i;

Run pirus
---------

Expand Down
52 changes: 19 additions & 33 deletions docs/restapi.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ REST Api
########




Ressources
==========

Expand Down Expand Up @@ -63,42 +65,26 @@ Endpoints
=========
Runs
----
``GET /run/``
Description of the get run
``GET /run``
Generic request to return a list of runs. Accepted (and all optional) query parameters :
- ``range`` : start-end :
- ``fields`` : the list (comma separedted) of fields that shall be loaded/return
- ``order_by`` : the list (comma separated) of fields in which the results shall be ordered
- ``sorted_by`` : this list shall be used accordingly with order_by to precise if the fields shall ordered ASC or DESC
- ``filter`` : [string] a value that will be used to filter on all fields. only elemnt that contain this string will be returned

Response example:
.. code-block:: python
{
data:
{
progress:
{
value: "100",
label: "100 / 100",
max: "100",
min: "0"
},
end: null,
name: "t2",
config: "<json serialized>",
inputs: [
"580897120e95cb2d349fd3ce"
],
id: "5808a28b0e95cb328c35c5c3",
status: "DONE",
outputs: null,
pipe_id: "5808a25a0e95cb328c35c5c2",
start: "1476960907.384298"
},
success: true
}
``POST /run/``
Description of the get run


``POST /run``
Init and start a run with provided information. The run is automatically start if there is enough resources; otherwise it will wait for that.
POST data must be JSON dictionary with the 3 requiered fields :
- ``pipeline_id`` : [string] the id of the pipeline to use for the run
- ``config`` : [dict] the config key-values to configure the run
- ``inputs`` : [list] the list of file ids that shall be used as input by the run


``GET /run/{run_id}``
Description of the get run
**Description of the get run
``GET /run/{run_id}/progress``
Description of the get run
Expand Down
62 changes: 62 additions & 0 deletions examples/pipelines/PirusBasic/form.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{
"$schema": "http://json-schema.org/draft-03/schema#",
"type": "object",
"properties":
{
"file1":
{
"title": "Fichier 1",
"description": "Sélectionner lequel est le `file1` parmis ceux que vous avez sélectionné pour le run.",
"type": "string",
"enum": "__PIRUS_INPUT_FILES__",
"required": true
},
"file2":
{
"title": "Fichier 2",
"description": "Sélectionner lequel est le `file2` parmis ceux que vous avez sélectionné pour le run.",
"type": "string",
"enum": "__PIRUS_INPUT_FILES__",
"required": false
},
"ref":
{
"title": "Génome de référence",
"description": "Choisissez votre génome de référence parmis les base de données disponible",
"type": "string",
"enum": "__PIRUS_DB_ALL__",
"required": true
},
"confirm":
{
"title": "Vous êtes sûr ?",
"description": "En cochant cette case vous vous engagez blablabla.",
"type": "boolean",
"default": true
},
"witherror":
{
"title": "Crash ?",
"description": "Cochez cette case si vous souhaitez que le run crash... pour voir comment ça se passe.",
"type": "boolean",
"default": false
},
"duration":
{
"title": "Durée du run",
"description": "Indiquez le nombre de seconde que va durer le run.",
"type": "integer",
"required": true,
"default" : 100
},
"outfilename":
{
"title": "Fichier résultat",
"description": "Comment souhaitez vous nommer le fichier résultat que génera ce run.",
"type": "string",
"required": true,
"default":"result.txt"
}
}
}

72 changes: 72 additions & 0 deletions examples/pipelines/PirusBasic/metadata.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
{
"architecture": "x86_64",
"creation_date": 1475725740,
"properties": {
"architecture": "x86_64",
"description": "ubuntu xenial x86_64 (default) (20161006_03:49)",
"name": "ubuntu-xenial-x86_64-default-20161006_03:49",
"os": "ubuntu",
"release": "xenial",
"variant": "default"
},
"templates": {
"/etc/hostname": {
"template": "hostname.tpl",
"when": [
"create"
]
},
"/etc/hosts": {
"template": "hosts.tpl",
"when": [
"create"
]
},
"/etc/init/console.override": {
"template": "upstart-override.tpl",
"when": [
"create"
]
},
"/etc/init/tty1.override": {
"template": "upstart-override.tpl",
"when": [
"create"
]
},
"/etc/init/tty2.override": {
"template": "upstart-override.tpl",
"when": [
"create"
]
},
"/etc/init/tty3.override": {
"template": "upstart-override.tpl",
"when": [
"create"
]
},
"/etc/init/tty4.override": {
"template": "upstart-override.tpl",
"when": [
"create"
]
}
},
"pirus":
{
"name" : "Pirus Simple",
"description" : "Test pipeline for pirus",
"version": "1.0.0",
"pirus_api": "1.0.0",
"license" : "AGPL",
"developers" : ["Olivier GUEUDELOT"],
"run" : "/pipeline/run/run.sh",
"inputs" : "/pipeline/inputs",
"outputs" : "/pipeline/outputs",
"databases" : "/pipeline/db",
"logs" : "/pipeline/logs",
"form" : "/pipeline/form.json"
}
}

55 changes: 32 additions & 23 deletions examples/pipelines/PirusBasic/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,20 +14,28 @@ This document explain you how to build a simple pipeline image for Pirus.
lxc exec pirus -- /bin/bash

# following directories are mandatory
mkdir /pipeline/
mkdir /pipeline/run
mkdir /pipeline/inputs
mkdir /pipeline/outputs
mkdir /pipeline/logs
mkdir /pipeline/db

# need curl if you want to notify server with the progress of your run
apt install curl jq nano
apt install curl jq nano --fix-missing

# the script run.sh is the "entry point" of your run
echo "curl ${NOTIFY}50" > /pipeline/run/run.sh
echo "ls -l /pipeline/database/db > /pipeline/outputs/result.txt" >> /pipeline/run/run.sh
# Create the script run.sh. this will be the "entry point" of your run
# An example can be found on github (https://github.com/REGOVAR/Pirus/blob/master/examples/pipelines/PirusBasic/run.sh)
nano /pipeline/run/run.sh
chmod +x /pipeline/run/run.sh

# To allow users to configure your pipeline, you shall put in your container a form.json file
# that will describe a form to set parameter for your pipe.
# An example can be found on github (https://github.com/REGOVAR/Pirus/blob/master/examples/pipelines/PirusBasic/form.json)
nano /pipeline/form.json

# You can also put a a custom logo (png or jpeg file) in your pipeline.

# exit the container
exit

Expand All @@ -38,47 +46,48 @@ This document explain you how to build a simple pipeline image for Pirus.
# Your Pipeline is ready to use on your server


## TODO : export image as file and edit image conf to create a piruse package installable on any pirus server
## Export image as file and edit image conf to create a piruse package installable on any pirus server

lxc image export PirusSimple
# following command shall be done as root to avoid image corruption
# (as it will try to create symlink to computer resource in /dev folder by example)
sudo tar xf a847ed7......3c4e2987e75.tar.gz
sudo tar xf <the_name_of_lxc_export_something_like_a8d44d24fcs...8fzef54e5>.tar.gz

# add folowing informations into the metadata.yaml file
sudo nano metadata.yaml

# si json
# if json
"pirus":
{
"name" : "Pirus Simple",
"description" : "Test pipeline for pirus",
"version": "1.0.0",
"pirus_api": "1.0.0",
"license" : "AGPLv3",
"developers" : ["Olivier GUEUDELOT"],
"run" : "/pipeline/run/run.sh",
"inputs" : "/pipeline/inputs",
"outputs" : "/pipeline/outputs",
"databases" : "/pipeline/db",
"logs" : "/pipeline/logs",
"form" : "/pipeline/form.json",
"icon" : "/pipeline/logo.png"
"name" : "Pirus Simple", # required : the name of your pipe
"description" : "Test pipeline", # optional : the purpose of your pipe
"version": "1.0.0", # optional : the version of your pipe
"pirus_api": "1.0.0", # optional : the pirus api version
"license" : "AGPLv3", # optional : the license of your pipe
"developers" : ["Olivier GUEUDELOT"], # optional : a list of name
"run" : "/pipeline/run/run.sh", # required : the command command that shall be execute to run your pipe (use absolute path)
"inputs" : "/pipeline/inputs", # optional : absolute path to the folder (in the container) where inputs files for the pipe shall be put
"outputs" : "/pipeline/outputs", # optional : absolute path to the folder (in the container) where ouputs files of the pipe will be put
"databases" : "/pipeline/db", # optional : absolute path to the folder (in the container) where tierce databases (hg19 by example) shall be put
"logs" : "/pipeline/logs", # optional : absolute path to the folder (in the container) where log of the run will be put
"form" : "/pipeline/form.json", # optional : absolute path to the json file that describe the form for the user to configure the run of the pipe
"icon" : "/pipeline/logo.png" # optional : absolute path to the image that shall be used as logo for the pipe
}
# si yaml
# if yaml
pirus:
name: "Pirus Simple"
name: "Pirus Simple" # required
description: "Test pipeline for pirus"
version : "1.0.0"
pirus_api: "1.0.0"
license: "AGPLv3"
developers: ["Olivier GUEUDELOT"]
run: "/pipeline/run/run.sh"
run: "/pipeline/run/run.sh" # required
logs: "/pipeline/logs"
inputs: "/pipeline/inputs"
outputs: "/pipeline/outputs"
databases: "/pipeline/db"
form: "/pipeline/form.json"
icon: "/pipeline/logo.png"


# You can repackage the image in tar.xz, to save space
Expand Down

0 comments on commit 2491cd5

Please sign in to comment.