Repository navigation
Workflows
- Michael R. Crusoe
- Manabu Ishii
- Hirotaka Suetake
- Tazro Ohta
- Tomoya Tanjo
- Alexander Kanitz
- Ralf Schmidt
- Kozo Nishida
- Naohisa Goto
- Masahiro Kasahara
- Takeshi Fujino
- Ioanna Kalvari
- Miguel Boland
- Norio Shinkai
In the pre-print process we should emphasize the structure of a (pre-publishable) subdocument. For the 2017 report we stuck to:
See How to write.
- Leads: Manabu Ishii & Michael R. Crusoe
- Members: Hirotaka Suetake, Michael R. Crusoe, Manabu Ishii, Tazro Ohta, Tomoya Tanjo, Alex Kanitz, Ralf Schmidt, Kozo Nishida, Naohisa Goto, Masahiro Kasahara, Takeshi Fujino, Norio Shinkai
- One-line description: All about workflows including workflow managers, workflow engines, applications, and development tools. Help people who want to write workflows.
There are several subgoals such as:
- Importance of workflow managers
- GA4GH Cloud Work Stream
- Developing workflow managers
We are working on workflow managers that support GA4GH Cloud Work Stream.
- State the problem you worked on
- Give the state-of-the art/plan
- Describe what you have done/results starting with The working group created...
- Write a conclusion
- Write up any future work
https://github.com/suecharo/SAPPORO
- State the problem you worked on
- There are some bugs.
- Give the state-of-the art/plan
- Bug fix.
- Describe what you have done/results starting with
- Maintenance
- SAPPORO-service
- SAPPORO-fileserver
- Write a conclusion
- Some maintenaces are done.
- Write up any future work
- Django + jQuery -> Flask + vue.js
- Use and connect to another WES
- Workflow managers described above finally invoke workflow engines that actually execute given workflow.
- Developing workflow engines and related tools
We are working on workflow engines that includes:
-
[State the problem you worked on]
- There are a lot of workflow engines that support CWL. However, it is difficult for users to compare their supported features.
- Currently there are mainly two ways to achieve it: reading the documents of workflow engines to be compared, or running the conformance test of CWL (that is provided by CWL project) for workflow engines and comparing the result for each features.
- The former is time comsuming task and is not realistic.
- The latter is more reasonable option but it is hard for users to install and to compare all the engines.
- The best way is that each workflow engine provides the result of the conformance test.
-
[Give the state-of-the art/plan]
- CWL project provides the conformance test as well as a testing framework for it.
- Also, CWL project has Jenkins servers for Continuous Integration (CI) and they run the conformance test for several workflow engines.
- (However, we have to check the outputs of the conformance test to know how much each workflow engine supports some features.)
- Before BioHackathon, we add a summarization feature to the testing framework. This feature generates a JSON file that shows how much a given workflow engine passes tests for each feature. Also, we enabled this feature for the conformance test for cwltool, which is a reference runner of CWL, in CI servers.
-
[Describe what you have done/results starting with The working group created...]
- We made CI servers to store JSON files that are generated by the conformance test for cwltool in common-workflow-language/conformance repository. JSON files are used to show
conformance badges, which show how a given workflow engine supports a specific feature of CWL. - In addition to that, we update the website of cwltool to show conformance badges for cwltool.
- We made CI servers to store JSON files that are generated by the conformance test for cwltool in common-workflow-language/conformance repository. JSON files are used to show
-
[Write a conclusion]
- We added conformance badges that show how cwltool, a reference runner of CWL, supports features of CWL.
- The badge generation feature can be applicable to other workflow engines that support CWL.
-
[Write up any future work]
- We will add conformance badges to websites of other workflow engines.
- In the long term plan, we will make a website in which users can easily choose appropriate workflow engines by checking their conformance badges.
- State the problem you worked on
- Give the state-of-the art/plan
- Describe what you have done/results starting with
- The working group created...
- Write a conclusion
- Write up any future work
- State the problem you worked on
- Give the state-of-the art/plan
- Describe what you have done/results starting with
- The working group created...
- Write a conclusion
- Write up any future work
- State the problem you worked on
- Give the state-of-the art/plan
- Describe what you have done/results starting with The working group created...
- Write a conclusion
- Write up any future work
- Developing workflows for various applications
We developed workflows for:
- (State the problem you worked on) Because BioRuby is a library, the usage is not very clear especially for new users, although tutorial and sample scripts are available.
- (Give the state-of-the art/plan) In order to clarify the usage of BioRuby with reproducible way, we determined to develop CWL workfolows for BioRuby sample scripts.
- (Describe what you have done/results starting with The working group created...) As a first step, we wrote CWL workflows for 3 sample scripts in BioRuby.
- (Write a conclusion) With the power of CWL, those who unfamiliar with Ruby can have chance to understand the usage of BioRuby and its sample scripts. Users can also develop much useful workflows to combine them.
- (Write up any future work) We will continue to add more CWL workflows for BioRuby scripts. We will also add more sample BioRuby scripts. Documentation including tutorials should also be added more, by using Jupyter Notebook.
Dashboarding pathway network with Dash-Cytoscape
- State the problem you worked on
- Give the state-of-the art/plan
- Created a Wikipathways2Cytoscape.js data converter (github.com/ecell/wp2cyjs)
- Created a sample Dash app for Wikipathways (WP3604)
- Describe what you have done/results starting with
-
Binder for Wikipathways visualization dashboard
- wp2cyjs (Convert Wikipathways XML[GPML] to Cytoscape.js json[cyjs])
- ↓
- bridgedbpy (Convert HMDB IDs to KEGG IDs with BridgeDB webservices)
- ↓
- Dash-Cytoscape (Visualize pathway network in Jupyter Notebook)
- Write a conclusion
- Write up any future work
- Packaging Wikipathways Dashboard as "Dash component"
- Escher map to Cyjs
- mnxref (Python package for MetaNetX SPARQL endpoint)
- State the problem you worked on
- Give the state-of-the art/plan
- Describe what you have done/results starting with The working group created...
- Write a conclusion
- Write up any future work
- State the problem you worked on
- There are several materials to learn CWL such as the specification, User Guide and existing CWL definitions provided by the community.
- However, preparing development environment is not easy task for users (Note: it is not limited to CWL). For example, we can use
piporcondacommand to install cwltool because it is written in Python. But it is not clear which is better. Furthermore, in Python ecosystem, there are too many tools that are related to environment construction, such as package managers, python version managers and virtual environment construction tools: pyenv, venv, virtualenv, pipenv, conda and pip, etc. - It is hard for users to choose good or appropriate development tool(s) to construct the environment.
- It would be nice if there is an environment in which the development tools are already installed.
- But it is not realistic to develop yet another development tools to solve the problem. It will cause further fragmentation of development tools.
- Give the state-of-the art/plan
- Recently, Microsoft developed a remote development extension for Visual Studio Code. It connects to remote machines, virtual machines, or docker containers and provides users with connected development environment. We can develop something just like we develop with our local machine.
- Especially, in the case of the remote development extension for containers, we can customize the container by providing
Dockerfileor/anddocker-compese.yml. - It would be nice if there is a repository that includes an appropriate
Dockerfileanddocker-compose.ymlthat provides a container in which development tools such as a reference workflow engine, a testing framework, and docker are already installed.
- Describe what you have done/results starting with
The working group created...
- We developed a template repository to develop CWL tool and workflow definitions for a remote container extension for Visual Studio Code. Once users clone it to the local machine and open it by using
Remote-Contaiers: Open Folder in Container..., Visual Studio Code automatically builds a container that includes all necessary development tools according to the Dockerfile provided by the repository and configures the docker-in-docker by using the docker-compose.yml provided by the repository. - It provides the following tools:
- Also, it provides template configurations of the following testing frameworks and services
-
test.ymlfor cwltest: By adding a new test entry to it, users can check the developed CWL files pass the test by using cwltest - Configurations for the following CI services. By integrating the corresponding services, they automatically check the developed CWL files pass the test by using cwltest for each commit:
-
- We developed a template repository to develop CWL tool and workflow definitions for a remote container extension for Visual Studio Code. Once users clone it to the local machine and open it by using
- Write a conclusion
- We developed a repository that provides users with a development environment that includes development tools for CWL.
- Also, we provide templates for testing framework and CI services.
- The developed repository encourages people to write more CWL tool and workflow definitions as well as to write tests for them.
- Write up any future work
- It is a promising approach to provide a development environment as well as the development materials such as source codes and documents in the repository. We will apply this approach to other CWL related tools.
- State the problem you worked on
- People have question about workflows
- Give the state-of-the art/plan
- Answer to people
- Describe what you have done/results starting with
- Answer to How to write workflows.
- Answer to How to setup workflow execution engine.
- Live Coding Session to show how to write CWL.
- Create some documetations
- Japanese
- CWLを docker in docker で使いたいときに気をつけること - Qiita
- CWLで、ツールが複数のファイルを必要とするときなど、secondaryFiles について - Qiita
- CWLで、location と path なら、location を使ったほうが良い - Qiita
- cwltoolで、[Errno 28] No space left on device がでるときの対処方法 - Qiita
- CWLで空白を含むファイルを処理したいとき - Qiita
- CWLで、arguments が意図せずクォートで囲われてしまうケースについて - Qiita
- CWLで気をつけたい文字"ハイフン"について - Qiita
- English (Translate from Japanese Article)
- Japanese
- Write a conclusion
- Some people start to write CWL description
- Setup SAPPORO and WES-Elixir various environments.
- Create new article for begginers.
- Write up any future work
- Write documentation to write easy
- Write documentation to setup easy
- More interoperability WES and TES.
Live coding session. Write documentations and tips.
- State the problem you worked on
- Try to build valiant calling workflow with last work on cwl
- Give the state-of-the art/plan
- build several last-tool-cwl (lastdb,lastal,last-pair-probs)
- merge these cwl-tool to one workflow
- Describe what you have done/results starting with
- make several last-tool-cwl
- lastal
- lastdb
- last-pair-probs
- maf-convert
- merge them to one workflow
- fastq-to-sam
- make several last-tool-cwl
- Write a conclusion
- Some tool is hard to make cwl-tool (because of secondary-file-problem).But finished.
- CWL-workflow(with last-docker-container) worked with test-data.
- Write up any future work
- build workflow fastq-to-bam, and connect to GATK-variantcalling-process
- try to run on cluster-server-system(with no-superuser)
- State the problem you worked on
- Give the state-of-the art/plan
- Describe what you have done/results starting with The working group created...
- Write a conclusion
- Write up any future work
-
WES-Elixir
-
- SAPPORO is a workflow and individual task execution system. It is also useful for continuous testing of workflows.
-
- Testing framework for workflow
-
- Open source bioinformatics library for Ruby
-
DASH-Bio (keggscape branch)
- Dashboard component for bioinformatics
-
TCE
-
Slack channel: #cwl , #wes-elixir
-
- Variant calling workflow with LAST on CWL
Category: Workflow Lead: Members:
Interested: Josiah Seaman, Hidemasa Bono, Takeya Kasukawa One-line description: Automating the Rfam non-coding RNA annotation pipeline using CWL Goal: Convert a collection of python scripts to a functioning CWL workflow Link to the project page (if any): http://rfam.org Slack channel: ?
- State the problem you worked on
- Give the state-of-the art/plan
- Describe what you have done/results starting with The working group created...
- Write a conclusion
- Write up any future work
We suggest to use either github markdown (wiki or a README.md) or the https://osf.io/ wiki pages which you can create when you open an account with ORCID. Make sure we can find this information by adding a group page on the wiki of this biohackathon.
In addition provide a figure with caption (if applicable) and Also provide a URL to your Jupyter notebook/github repo and data (as research objects)
The general idea is that we will compile a metadocument as the biohackathon 2019 report. When every project sticks to this layout the reader knows what to expect when reading about every project. This is about the reader experience rather than the author experience! So please stick with this 'flow'.
Making the Sequence Read Archive public and searchable
(State the problem) Information from next-generation sequencing experiments is separately archived to different databases including: the Sequence Read Archive (SRA) (Leinonen et al., 2010), the National Center for Biotechnology Information’s (NCBI) BioProject (Barrett et al., 2011), and BioSample (Courtot et al., 2018). (State of the art/plan) The working group aimed to make information in the SRA public and searchable (Figure 7). (Work executed) The following tasks were accomplished during the BH17: 1) customized the system to follow DNA Databank of Japan (DDBJ) (Stoesser, 2004) requests; 2) defined specifications for a brand new Database Center for Life Science (DBCLS) SRA (Leinonen et al., 2010)) Web user interface; 3) converted BioProject and BioSample data to RDF; and 4) developed a prototype method for searching reads by expression pattern. (Future work) In the future, the working group will integrate the SRA with BioProject and BioSample metadata as well as develop a Web user interface.
(figure and references)
Make sure references are added in the format:
Authors (surname, letters) (year). Title. Journal ref. DOI.
Adachi, T., Kawamura, K., Furusawa, Y., Nishizaki, Y., Imanishi, N., Umehara,
S., et al. (2017). Japan’s 827 initiative on rare and undiagnosed diseases
(IRUD): towards an end to the diagnostic odyssey. Eur. J. 828 Hum. Genet. 25,
1025