Building workflows with Pim

[mgalland]

Description

Building workflows with the Common Workflow Language.

There are several workflow tools for bioinformatics. Interoperability (the I in FAIR) is however often still a problem. In the context of creating a workflow that can easily migrate an open source workflow from Naturalis (see internship: https://www.rijkzwaancareers.com/careers/joint-internship-bioinformatics-rijk-zwaan-and-naturalis ) we're going to try out the CWL standard by creating a workflow with the Rabix Composer GUI and then running it locally with the CWL reference tool. Since CWL has multiple interpreters (toil, nextflow), it should afterwards be easier to run this workflow anywhere.

Check out the preparation guidelines below

Time and Place

Where: Room D1.113, Science Park 904
When: Tuesday, January 9th, 2018 at 17:00 PM

Required Preparation

Prior knowledge required

• Linux command line

Software Dependencies

• Linux laptop.
• terminal (bash).
• virtualenv.

Cwltool

( see https://github.com/common-workflow-language/cwltool )

virtualenv -p python3 cwlvenv
source cwlvenv/bin/activate
pip install cwltool
Optional test if it worked ( needs node.js ):
pip install pytest mock
py.test --ignore cwltool/schemas/ --pyarg cwltool

Rabix composer:

• Go to http://rabix.io/
• Download the rabix composer zip file
  https://github.com/rabix/composer/releases/download/v1.0.0-rc.2/rabix-composer-1.0.0-rc.2.zip
  (* note: there is also an installer for Mac in case you have a Mac laptop)

mkdir rabix
mkdir rabix-workspace
mv Downloads/rabix-composer-1.0.0-rc.2.zip rabix
cd rabix
unzip rabix-composer-1.0.0-rc.2.zip rabix
./rabix-composer

A graphical interface will open

Step-by-step activities for the hour

Getting started

• Short introduction to bioinformatics at Rijk Zwaan.
• Common Workflow Language
• Double-check who managed to get Rabix and cwltool installed.

Creating a test file, initialize our Rabix workspace and a first tool wrapper

• cd rabix-workspace

• seq 100 120 > lines.txt

• start Rabix composer (see above)

• Top left: "My Project" .. "+ Open a Project"

• "Local" .. "Select a folder"

• Choose "rabix-workspace" .. "Add to workspace"

• Double-click on "Local files" at the left pane, the folder should appear.

• Press the "+" on the top of the screen to go to the "new file" tab

• "New command line tool"

• App Name: ReverseLinesStep, CWL 1.0, choose a local folder, your rabix workspace, create

• You'll get a new tab

• Base command: /usr/bin/rev

• Input ports: Add an Input:

• required: yes;

• ID: input-for-reverselines,

• Type: File;

• Include in command line: no

• Save!

• Output ports: Add an Output:

• required: yes

• ID: output

• Glob: output-from-reverselines.*

• Save!

• Other: stdout redirect: output-from-reverse-lines.txt

• Save!

• Run the output file in cwltool

Now try eSciencecenter's scriptcwl

• make a virtualenv
• pip install scriptcwl
• download https://github.com/NLeSC/scriptcwl/archive/master.zip
• unzip it
• Copy the example script from http://scriptcwl.readthedocs.io/en/latest/ into a test.py file
• change the directory to the example files
• run test.py to generate the cwl
• use cwltool to run the cwl file

Additional reading material

A nice tutorial

[ps-account]

Thanks for adding me to the schedule, Marc! Just to give some context of a CWL project:

If anyone knows BSc or MSc students interested in this topic they can apply for a paid internship in on CWL-ing an open source workflow together with Naturalis! Contact me at p.schravendijk@rijkzwaan.nl if there are questions! https://www.rijkzwaancareers.com/careers/joint-internship-bioinformatics-rijk-zwaan-and-naturalis

[ps-account]

Marc: can you add an installed Docker daemon to the requirements? I will add more requirements later.

[mgalland]

Hey Pim.
You can actually modify the note yourself. The syntax follows that of Markdown basics for links, etc.
More on Markdown with Github here

[ps-account]

Hi Marc, I don’t have an edit option! Maybe you can figure out why? Otherwise I can send a screenshot.

[mgalland]

I think I found the problem. As a temporary solution, you've became a full member so you should have read/write rights. When we're done, I'll try to convert you an outside collaborator with read/write privileges.
Let me know how it goes!