Skip to content
Tomoya Tanjo edited this page Sep 12, 2019 · 12 revisions

Workflow

Authors with ORCIDs(!)

In the pre-print process we should emphasize the structure of a (pre-publishable) subdocument. For the 2017 report we stuck to:

Contents

See How to write.

Category: Workflow, Workflow Execution Service

  • Leads: Manabu Ishii & Michael R. Crusoe
  • Members: Hirotaka Suetake, Michael R. Crusoe, Manabu Ishii, Tazro Ohta, Tomoya Tanjo, Alex Kanitz, Ralf Schmidt, Kozo Nishida, Naohisa Goto, Masahiro Kasahara, Takeshi Fujino, Norio Shinkai
  • One-line description: All about workflows including workflow managers, workflow engines, applications, and development tools. Help people who want to write workflows.

Goals

There are several subgoals such as:

Workflow Managers

We are working on workflow managers that support GA4GH Cloud Work Stream.

WES-Elixir (Alex Kanitz, Ralf Schmidt)

  1. State the problem you worked on
  2. Give the state-of-the art/plan
  3. Describe what you have done/results starting with The working group created...
  4. Write a conclusion
  5. Write up any future work

SAPPORO (Hirotaka Suetake)

https://github.com/suecharo/SAPPORO

  1. State the problem you worked on
  • There are some bugs.
  1. Give the state-of-the art/plan
  • Bug fix.
  1. Describe what you have done/results starting with
  • Maintenance
    • SAPPORO-service
    • SAPPORO-fileserver
  1. Write a conclusion
  • Some maintenaces are done.
  1. Write up any future work
  • Django + jQuery -> Flask + vue.js
  • Use and connect to another WES

Workflow Engines

  • Workflow managers described above finally invoke workflow engines that actually execute given workflow.
  • Developing workflow engines and related tools

We are working on workflow engines that includes:

CWL (Tanjo)

  1. [State the problem you worked on]

    • There are a lot of workflow engines that support CWL. However, it is difficult for users to compare their supported features.
    • Currently there are mainly two ways to achieve it: reading the documents of workflow engines to be compared, or running the conformance test of CWL (that is provided by CWL project) for workflow engines and comparing the result for each features.
    • The former is time comsuming task and is not realistic.
    • The latter is more reasonable option but it is hard for users to install and to compare all the engines.
      • The best way is that each workflow engine provides the result of the conformance test.
  2. [Give the state-of-the art/plan]

    • CWL project provides the conformance test as well as a testing framework for it.
    • Also, CWL project has Jenkins servers for Continuous Integration (CI) and they run the conformance test for several workflow engines.
    • (However, we have to check the outputs of the conformance test to know how much each workflow engine supports some features.)
    • Before BioHackathon, we add a summarization feature to the testing framework. This feature generates a JSON file that shows how much a given workflow engine passes tests for each feature. Also, we enabled this feature for the conformance test for cwltool, which is a reference runner of CWL, in CI servers.
  3. [Describe what you have done/results starting with The working group created...]

    • We made CI servers to store JSON files that are generated by the conformance test for cwltool in common-workflow-language/conformance repository. JSON files are used to show conformance badges, which show how a given workflow engine supports a specific feature of CWL.
    • In addition to that, we update the website of cwltool to show conformance badges for cwltool.
      • Here is an example of the conformance badge that shows how cwltool supports container feature: DockerRequirement
  4. [Write a conclusion]

    • We added conformance badges that show how cwltool, a reference runner of CWL, supports features of CWL.
    • The badge generation feature can be applicable to other workflow engines that support CWL.
  5. [Write up any future work]

    • We will add conformance badges to websites of other workflow engines.
    • In the long term plan, we will make a website in which users can easily choose appropriate workflow engines by checking their conformance badges.

cwltool (Michael R. Crusoe)

  1. State the problem you worked on
  1. Give the state-of-the art/plan
  1. Describe what you have done/results starting with
  • The working group created...
  1. Write a conclusion
  1. Write up any future work

toil (Miguel Boland, Michael R. Crusoe)

  1. State the problem you worked on
  1. Give the state-of-the art/plan
  1. Describe what you have done/results starting with
  • The working group created...
  1. Write a conclusion
  1. Write up any future work

TCE / lpmx (Masahiro Kasahara, Takeshi Fujino)

  1. State the problem you worked on
  2. Give the state-of-the art/plan
  3. Describe what you have done/results starting with The working group created...
  4. Write a conclusion
  5. Write up any future work

Applications

  • Developing workflows for various applications

We developed workflows for:

BioRuby (Naohisa Goto, Kozo Nishida)

  1. (State the problem you worked on) Because BioRuby is a library, the usage is not very clear especially for new users, although tutorial and sample scripts are available.
  2. (Give the state-of-the art/plan) In order to clarify the usage of BioRuby with reproducible way, we determined to develop CWL workfolows for BioRuby sample scripts.
  3. (Describe what you have done/results starting with The working group created...) As a first step, we wrote CWL workflows for 3 sample scripts in BioRuby.
  4. (Write a conclusion) With the power of CWL, those who unfamiliar with Ruby can have chance to understand the usage of BioRuby and its sample scripts. Users can also develop much useful workflows to combine them.
  5. (Write up any future work) We will continue to add more CWL workflows for BioRuby scripts. We will also add more sample BioRuby scripts. Documentation including tutorials should also be added more, by using Jupyter Notebook.

DASH-Bio (Kozo Nishida)

Dashboarding pathway network with Dash-Cytoscape

  1. State the problem you worked on
  2. Give the state-of-the art/plan
  • Created a Wikipathways2Cytoscape.js data converter (github.com/ecell/wp2cyjs)
  • Created a sample Dash app for Wikipathways (WP3604)
  1. Describe what you have done/results starting with
  1. Write a conclusion
  2. Write up any future work

Development Tools and documentation

Tonkaz (Hirotaka Suetake)

  1. State the problem you worked on
  2. Give the state-of-the art/plan
  3. Describe what you have done/results starting with The working group created...
  4. Write a conclusion
  5. Write up any future work

Deveopment template to write CWL tool and workflow definitions (Tanjo)

  1. State the problem you worked on
    • There are several materials to learn CWL such as the specification, User Guide and existing CWL definitions provided by the community.
    • However, preparing development environment is not easy task for users (Note: it is not limited to CWL). For example, we can use pip or conda command to install cwltool because it is written in Python. But it is not clear which is better. Furthermore, in Python ecosystem, there are too many tools that are related to environment construction, such as package managers, python version managers and virtual environment construction tools: pyenv, venv, virtualenv, pipenv, conda and pip, etc.
    • It is hard for users to choose good or appropriate development tool(s) to construct the environment.
      • It would be nice if there is an environment in which the development tools are already installed.
      • But it is not realistic to develop yet another development tools to solve the problem. It will cause further fragmentation of development tools.
  2. Give the state-of-the art/plan
    • Recently, Microsoft developed a remote development extension for Visual Studio Code. It connects to remote machines, virtual machines, or docker containers and provides users with connected development environment. We can develop something just like we develop with our local machine.
    • Especially, in the case of the remote development extension for containers, we can customize the container by providing Dockerfile or/and docker-compese.yml.
    • It would be nice if there is a repository that includes an appropriate Dockerfile and docker-compose.yml that provides a container in which development tools such as a reference workflow engine, a testing framework, and docker are already installed.
  3. Describe what you have done/results starting with The working group created...
    • We developed a template repository to develop CWL tool and workflow definitions for a remote container extension for Visual Studio Code. Once users clone it to the local machine and open it by using Remote-Contaiers: Open Folder in Container..., Visual Studio Code automatically builds a container that includes all necessary development tools according to the Dockerfile provided by the repository and configures the docker-in-docker by using the docker-compose.yml provided by the repository.
    • It provides the following tools:
      • cwltool: A reference runner of CWL
      • cwltest: A testing framework for CWL
      • benten: A language server for CWL that provides users with enhanced development environment
    • Also, it provides template configurations of the following testing frameworks and services
      • test.yml for cwltest: By adding a new test entry to it, users can check the developed CWL files pass the test by using cwltest
      • Configurations for the following CI services. By integrating the corresponding services, they automatically check the developed CWL files pass the test by using cwltest for each commit:
  4. Write a conclusion
    • We developed a repository that provides users with a development environment that includes development tools for CWL.
    • Also, we provide templates for testing framework and CI services.
    • The developed repository encourages people to write more CWL tool and workflow definitions as well as to write tests for them.
  5. Write up any future work
    • It is a promising approach to provide a development environment as well as the development materials such as source codes and documents in the repository. We will apply this approach to other CWL related tools.

Help people who want to write workflows(everyone)

  1. State the problem you worked on
  • People have question about workflows
  1. Give the state-of-the art/plan
  • Answer to people
  1. Describe what you have done/results starting with
  1. Write a conclusion
  • Some people start to write CWL description
  • Setup SAPPORO and WES-Elixir various environments.
  • Create new article for begginers.
  1. Write up any future work
  • Write documentation to write easy
  • Write documentation to setup easy
  • More interoperability WES and TES.

Live coding session. Write documentations and tips.

LAST Workflow (Norio Shinkai)

  1. State the problem you worked on
    • Try to build valiant calling workflow with last work on cwl
  2. Give the state-of-the art/plan
    • build several last-tool-cwl (lastdb,lastal,last-pair-probs)
    • merge these cwl-tool to one workflow
  3. Describe what you have done/results starting with
    • make several last-tool-cwl
      • lastal
      • lastdb
      • last-pair-probs
      • maf-convert
    • merge them to one workflow
      • fastq-to-sam
  4. Write a conclusion
    • Some tool is hard to make cwl-tool (because of secondary-file-problem).But finished.
    • CWL-workflow(with last-docker-container) worked with test-data.
  5. Write up any future work
    • build workflow fastq-to-bam, and connect to GATK-variantcalling-process
    • try to run on cluster-server-system(with no-superuser)

Template (replace project) (Name)

  1. State the problem you worked on
  2. Give the state-of-the art/plan
  3. Describe what you have done/results starting with The working group created...
  4. Write a conclusion
  5. Write up any future work

Project pages

Rfam non-coding RNA annotation pipeline

Category: Workflow Lead: Members:

Interested: Josiah Seaman, Hidemasa Bono, Takeya Kasukawa One-line description: Automating the Rfam non-coding RNA annotation pipeline using CWL Goal: Convert a collection of python scripts to a functioning CWL workflow Link to the project page (if any): http://rfam.org Slack channel: ?


How to write

  1. State the problem you worked on
  2. Give the state-of-the art/plan
  3. Describe what you have done/results starting with The working group created...
  4. Write a conclusion
  5. Write up any future work

We suggest to use either github markdown (wiki or a README.md) or the https://osf.io/ wiki pages which you can create when you open an account with ORCID. Make sure we can find this information by adding a group page on the wiki of this biohackathon.

In addition provide a figure with caption (if applicable) and Also provide a URL to your Jupyter notebook/github repo and data (as research objects)

The general idea is that we will compile a metadocument as the biohackathon 2019 report. When every project sticks to this layout the reader knows what to expect when reading about every project. This is about the reader experience rather than the author experience! So please stick with this 'flow'.

Example:

Making the Sequence Read Archive public and searchable

(State the problem) Information from next-generation sequencing experiments is separately archived to different databases including: the Sequence Read Archive (SRA) (Leinonen et al., 2010), the National Center for Biotechnology Information’s (NCBI) BioProject (Barrett et al., 2011), and BioSample (Courtot et al., 2018). (State of the art/plan) The working group aimed to make information in the SRA public and searchable (Figure 7). (Work executed) The following tasks were accomplished during the BH17: 1) customized the system to follow DNA Databank of Japan (DDBJ) (Stoesser, 2004) requests; 2) defined specifications for a brand new Database Center for Life Science (DBCLS) SRA (Leinonen et al., 2010)) Web user interface; 3) converted BioProject and BioSample data to RDF; and 4) developed a prototype method for searching reads by expression pattern. (Future work) In the future, the working group will integrate the SRA with BioProject and BioSample metadata as well as develop a Web user interface.

(figure and references)

Make sure references are added in the format:

Authors (surname, letters) (year). Title. Journal ref. DOI.

    Adachi, T., Kawamura, K., Furusawa, Y., Nishizaki, Y., Imanishi, N., Umehara,
    S., et al. (2017). Japan’s 827 initiative on rare and undiagnosed diseases
    (IRUD): towards an end to the diagnostic odyssey. Eur. J. 828 Hum. Genet. 25,
    1025

Clone this wiki locally