Skip to content

Commit

Permalink
Update docs on pipeline interface
Browse files Browse the repository at this point in the history
  • Loading branch information
nsheff committed Jan 23, 2017
1 parent 4281cdb commit a00c9ef
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 6 deletions.
8 changes: 4 additions & 4 deletions doc/source/config-files.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
Configuration files
=========================

Looper uses `YAML <http://www.yaml.org/>`_ configuration files to describe a project. Looper is a very modular system, so there are few different YAML files. Here's an explanation of each. Which ones you need to know about will depend on whether you're a pipeline user (running pipelines on your project) or a pipeline developer (building a new pipeline).
Looper uses `YAML <http://www.yaml.org/>`_ configuration files to describe a project. Looper is a very modular system, so there are few different YAML files. Here's an explanation of each. Which ones you need to know about will depend on whether you're a pipeline user (running pipelines on your project) or a pipeline developer (building your own pipeline).

Pipeline users
*****************

Users (non-developers) of the system only need to be aware of one YAML file:
Users (non-developers) of pipelines only need to be aware of one YAML file:

- :ref:`project config file <project-config-file>`: This file is specific to each project and contains information about the project's metadata, where the processed files should be saved, and other variables that allow to configure the pipelines specifically for this project.

Expand All @@ -27,5 +27,5 @@ If you want to add a new pipeline to looper, then there are two YAML files that

Finally, if you're using Pypiper to develop pipelines, it uses a pipeline-specific configuration file (detailed in the Pypiper documentation):

- :ref:`pipeline-config-file`: Each pipeline may have a configuration file describing where software is, and parameters to use for tasks within the pipeline.

- `Pypiper pipeline config file <http://pypiper.readthedocs.io/en/latest/advanced.html#pipeline-config-files>`_: Each pipeline may have a configuration file describing where software is, and parameters to use for tasks within the pipeline
.pypiper
12 changes: 10 additions & 2 deletions doc/source/pipeline-interface.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,16 @@
Pipeline interface YAML
**************************************************

The pipeline interface file describes how the looper, which submits jobs, knows what resources to request for a pipeline, and what arguments to pass to the
pipeline. For each pipeline (defined by the filename of the script itself), you specify three components: ``name``, ``arguments``, and ``resources``.
The pipeline interface file describes how looper, which submits jobs, knows what arguments to pass to the pipeline and (possibly) what resources to request. For each pipeline (defined by the filename of the script itself), you specify some optional and required variables:

- **name (recommended)**: Name of the pipeline
- **arguments (required)**: List of key-value pairs of arguments, and attribute sources to pass to the pipeline (details below).
- **resources (required)**: A section outlining how much memory, CPU, and clock time to request, modulated by input file size (details below)
- **required_input_files (optional)**: A list of sample attributes (annotation sheets column names) that will point to input files that must exist.
- **all_input_files (optional)**: A list of sample attributes (annotation sheets column names) that will point to input files that are not required, but if they exist, should be counted in the total size calculation for requesting resources.
- **ngs_input_files (optional)**: A list of sample attributes (annotation sheets column names) that will point to input files to be used for automatic detection of read_length and read_type.

Example:

.. code-block:: yaml
Expand Down

0 comments on commit a00c9ef

Please sign in to comment.