Skip to content

Commit

Permalink
Update docs to PEPENV
Browse files Browse the repository at this point in the history
  • Loading branch information
nsheff committed Jun 13, 2017
1 parent a8b6fda commit e0e559f
Show file tree
Hide file tree
Showing 3 changed files with 6 additions and 4 deletions.
2 changes: 2 additions & 0 deletions doc/source/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ Changelog

- Changed

- Changed LOOPERENV environment variable to PEPENV, generalizing it to generic models

- Changed name of ``run`` column to ``toggle``, since ``run`` can also refer to a sequencing run.

- Relaxes many constraints (like resources sections, pipelines_dir columns), making project
Expand Down
6 changes: 3 additions & 3 deletions doc/source/cluster-computing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Cluster computing

By default, looper will build a shell script for each sample and then run each sample serially on the local computer. But where looper really excels is in large projects that require submitting these jobs to a cluster resource manager (like SLURM, SGE, LFS, etc.). Looper handles the interface to the resource manager so that projects and pipelines can be moved to different environments with ease.

To configure looper to use cluster computing, all you have to do is tell looper a few things about your cluster setup: you create a configuration file (`compute_config.yaml`) and point an environment variable (``LOOPERENV``) to this file, and that's it! Complete, step-by-step instructions and examples are available in the looperenv repository at https://github.com/epigen/looperenv.
To configure looper to use cluster computing, all you have to do is tell looper a few things about your cluster setup: you create a configuration file (`compute_config.yaml`) and point an environment variable (``PEPENV``) to this file, and that's it! Complete, step-by-step instructions and examples are available in the pepenv repository at https://github.com/pepkit/pepenv.

Compute config overview
****************************************
Expand All @@ -31,12 +31,12 @@ The sub-sections below ``compute`` each define a "compute package" that can be a

There are two or three sub-parameters for a compute package:

- **submission_template** is a (relative or absolute) path to the template submission script. Templates are described in more detail in the `looperenv readme <https://github.com/epigen/looperenv>`_.
- **submission_template** is a (relative or absolute) path to the template submission script. Templates are described in more detail in the `pepenv readme <https://github.com/pepkit/pepenv>`_.
- **submission_command** is the command-line command that `looper` will prepend to the path of the produced submission script to actually run it (`sbatch` for SLURM, `qsub` for SGE, `sh` for localhost, etc).
- **partition** is necessary only if you need to specify a queue name


Submission templates
****************************************
A template uses variables (encoded like `{VARIABLE}`), which will be populated independently for each sample as defined in `pipeline_interface.yaml`. The one variable ``{CODE}`` is a reserved variable that refers to the actual python command that will run the pipeline. Otherwise, you can use any variables you define in your `pipeline_interface.yaml`. In `Templates <https://github.com/epigen/looperenv/tree/master/templates>`__ are examples for submission templates for `SLURM <https://github.com/epigen/looperenv/blob/master/templates/slurm_template.sub>`__, `SGE <https://github.com/epigen/looperenv/blob/master/templates/sge_template.sub>`__, and `local runs <https://github.com/epigen/looperenv/blob/master/templates/localhost_template.sub>`__. You can also create your own templates, giving looper ultimate flexibility to work with any compute infrastructure in any environment.
A template uses variables (encoded like `{VARIABLE}`), which will be populated independently for each sample as defined in `pipeline_interface.yaml`. The one variable ``{CODE}`` is a reserved variable that refers to the actual python command that will run the pipeline. Otherwise, you can use any variables you define in your `pipeline_interface.yaml`. In `Templates <https://github.com/pepkit/pepenv/tree/master/templates>`__ are examples for submission templates for `SLURM <https://github.com/pepkit/pepenv/blob/master/templates/slurm_template.sub>`__, `SGE <https://github.com/pepkit/pepenv/blob/master/templates/sge_template.sub>`__, and `local runs <https://github.com/pepkit/pepenv/blob/master/templates/localhost_template.sub>`__. You can also create your own templates, giving looper ultimate flexibility to work with any compute infrastructure in any environment.

2 changes: 1 addition & 1 deletion doc/source/project-config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ The above specification will now pass '--flavor=simple' and '--flag' whenever rr

Project config section: compute
"""""""""""""""""""""""""""""""""""""""""""""""
You can specify project-specific compute settings in a ``compute`` section. However, you're better off specifying this globally using a ``looperenv`` environment configuration. Instructions are at the `looperenv repository <https://github.com/epigen/looperenv>`_. If you do need project-specific control over compute settings (like submitting a certain project to a certain resource account), you can do this by specifying variables in a project config ``compute`` section, which will override global looperenv values for that project only.
You can specify project-specific compute settings in a ``compute`` section. However, you're better off specifying this globally using a ``pepenv`` environment configuration. Instructions are at the `pepenv repository <https://github.com/pepkit/pepenv>`_. If you do need project-specific control over compute settings (like submitting a certain project to a certain resource account), you can do this by specifying variables in a project config ``compute`` section, which will override global pepenv values for that project only.


.. code-block:: yaml
Expand Down

0 comments on commit e0e559f

Please sign in to comment.