Skip to content

Commit

Permalink
Merge pull request #305 from nf-core/dev
Browse files Browse the repository at this point in the history 
Prepare nf-core tools release 1.6
  • Loading branch information
@sven1103
sven1103 committed Apr 9, 2019
2 parents 1855d34 + 95624ed commit 4ab80b8
Show file tree
Hide file tree
Showing 31 changed files with 455 additions and 545 deletions.
6 changes: 3 additions & 3 deletions .github/CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ We try to manage the required tasks for nf-core/tools using GitHub issues, you p

However, don't be put off by this template - other more general issues and suggestions are welcome! Contributions to the code are even more welcome ;)

> If you need help using or developing nf-core/tools then the best place to go is the Gitter chatroom where you can ask us questions directly: https://gitter.im/nf-core/Lobby
> If you need help using or developing nf-core/tools then the best place to ask is the nf-core `tools` channel on [Slack](https://nf-core-invite.herokuapp.com/).
## Contribution workflow
If you'd like to write some code for nf-core/tools, the standard workflow
Expand All @@ -24,7 +24,7 @@ If you're not used to this workflow with git, you can start with some [basic doc

## Style guide
Google provides an excellent [style guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md), which
is an best practise extension of [PEP](https://www.python.org/dev/peps/), the Python Enhancement Proposals. Have a look at the
is a best practise extension of [PEP](https://www.python.org/dev/peps/), the Python Enhancement Proposals. Have a look at the
[docstring](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings) section, which is in particular
important, as nf-core tool's code documentation is generated out of these automatically.

Expand Down Expand Up @@ -68,4 +68,4 @@ nf-core lint nf-core-testpipeline
```

## Getting help
For further information/help, please consult the [nf-core/tools documentation](https://github.com/nf-core/tools#documentation) and don't hesitate to get in touch on [Gitter](https://gitter.im/nf-core/Lobby)
For further information/help, please consult the [nf-core/tools documentation](https://github.com/nf-core/tools#documentation) and don't hesitate to get in touch on the nf-core `tools` channel on [Slack](https://nf-core-invite.herokuapp.com/).
5 changes: 4 additions & 1 deletion .travis.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,13 @@ install:
- sudo apt-get install npm && npm install -g markdownlint-cli # For documentation linting

script:
# Test the main code in this repo
- pytest --cov=nf_core .
- markdownlint . -c ${TRAVIS_BUILD_DIR}/.github/markdownlint.yml
# Test the pipeline template code
- nf-core create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface"
- nf-core lint nf-core-testpipeline
- markdownlint . -c ${TRAVIS_BUILD_DIR}/.github/markdownlint.yml
- markdownlint nf-core-testpipeline -c nf-core-testpipeline/.github/markdownlint.yml

after_success:
- codecov --rcfile=${TRAVIS_BUILD_DIR}/.github/coveragerc
Expand Down
33 changes: 33 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,37 @@
# nf-core/tools: Changelog

## v1.6

#### Syncing
* Code refactoring to make the script more readable
* No travis build failure anymore on sync errors
* More verbose logging

#### Template pipeline
* awsbatch `work-dir` checking moved to nextflow itself. Removed unsatisfiable check in main.nf template.
* Fixed markdown linting
* Tools CI testing now runs markdown lint on compiled template pipeline
* Migrated large portions of documentation to the [nf-core website](https://github.com/nf-core/nf-co.re/pull/93)
* Removed Gitter references in `.github/` directories for `tools/` and pipeline template.
* Changed `scrape_software_versions.py` to output `.csv` file
* Added `export_plots` parameter to multiqc config

#### Tools helper code
* Drop [nf-core/rnaseq](https://github.com/nf-core/rnaseq]) from `blacklist.json` to make template sync available
* Updated main help command to sort the subcommands in a more logical order
* Updated readme to describe the new `nf-core launch` command
* Fix bugs in `nf-core download`
* The _latest_ release is now fetched by default if not specified
* Downloaded pipeline files are now properly executable.
* Fixed bugs in `nf-core list`
* Sorting now works again
* Output is partially coloured (better highlighting out of date pipelines)
* Improved documentation
* Fixed bugs in `nf-core lint`
* The order of conda channels is now correct, avoiding occasional erroneous errors that packages weren't found ([#207](https://github.com/nf-core/tools/issues/207))
* Add reporting of ignored errored process
* As a solution for [#103](https://github.com/nf-core/tools/issues/103))

## [v1.5](https://github.com/nf-core/tools/releases/tag/1.5) - 2019-03-13 Iron Shark

#### Template pipeline
Expand All @@ -14,6 +46,7 @@
* Completion email now includes MultiQC report if not too big
* `params.genome` is now checked if set, to ensure that it's a valid iGenomes key
* Together with nf-core/configs, helper function now checks hostname and suggests a valid config profile
* `awsbatch` executor requires the `tracedir` not to be set to an `s3` bucket.

#### Tools helper code
* New `nf-core launch` command to interactively launch nf-core pipelines from command-line
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@ LABEL authors="phil.ewels@scilifelab.se,alexander.peltzer@qbic.uni-tuebingen.de"
description="Docker image containing base requirements for the nfcore pipelines"

# Install procps so that Nextflow can poll CPU usage
RUN apt-get update && apt-get install -y procps && apt-get clean -y
RUN apt-get update && apt-get install -y procps && apt-get clean -y
RUN conda install conda=4.6.7
174 changes: 141 additions & 33 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,18 @@ A python package with helper tools for the nf-core community.

## Table of contents

* [Installation](#installation)
* [Listing pipelines](#listing-pipelines) (`nf-core list`)
* [Downloading pipelines for offline use](#downloading-pipelines-for-offline-use) (`nf-core download`)
* [Listing software licences](#pipeline-software-licences): List software licences for a given workflow (`nf-core licences`)
* [Creating a new workflow](#creating-a-new-workflow) (`nf-core create`)
* [Checking a pipeline against nf-core guidelines](#linting-a-workflow) (`nf-core lint`)
* [Bumping a pipeline version number](#bumping-a-pipeline-version-number) (`nf-core bump-version`)
* [`nf-core` tools installation](#installation)
* [`nf-core list` - List available pipelines](#listing-pipelines)
* [`nf-core launch` - Run a pipeline with interactive parameter prompts](#launch-a-pipeline)
* [`nf-core download` - Download pipeline for offline use](#downloading-pipelines-for-offline-use)
* [`nf-core licences` - List software licences in a pipeline](#pipeline-software-licences)
* [`nf-core create` - Create a new workflow from the nf-core template](#creating-a-new-workflow)
* [`nf-core lint` - Check pipeline code against nf-core guidelines](#linting-a-workflow)
* [`nf-core bump-version` - Update nf-core pipeline version number](#bumping-a-pipeline-version-number)


The nf-core tools package is written in Python and can be imported and used within other packages.
For documentation of the internal Python functions, please refer to the [Tools Python API docs](https://nf-co.re/tools-docs/).

## Installation

Expand All @@ -25,6 +30,12 @@ You can install `nf-core/tools` from [PyPI](https://pypi.python.org/pypi/nf-core
pip install nf-core
```

Alternatively, it can be installed using [bioconda](https://bioconda.github.io/recipes/nf-core/README.html):

```bash
conda install -c bioconda nf-core
```

If you would like the development version instead, the command is:

```bash
Expand All @@ -50,7 +61,7 @@ The command `nf-core list` shows all available nf-core pipelines along with thei

An example of the output from the command is as follows:

```bash
```console
$ nf-core list

,--./,-.
Expand All @@ -60,23 +71,27 @@ $ nf-core list
`._,._,'


Name Version Published Last Pulled Default local is latest release?
----------------- --------- ----------- ------------- ----------------------------------
nf-core/hlatyping 1.1.0 5 days ago 9 minutes ago Yes
nf-core/methylseq 1.1 1 week ago 2 months ago No
nf-core/chipseq dev - - No
nf-core/eager dev - - No
nf-core/exoseq dev - - No
nf-core/mag dev - - No
nf-core/rnaseq dev - - No
nf-core/smrnaseq dev - - No
nf-core/vipr dev - - No
Name Version Released Last Pulled Have latest release?
------------------------- --------- ------------ -------------- ----------------------
nf-core/rnaseq 1.3 4 days ago 27 minutes ago Yes
nf-core/hlatyping 1.1.4 3 weeks ago 1 months ago No
nf-core/eager 2.0.6 3 weeks ago - -
nf-core/mhcquant 1.2.6 3 weeks ago - -
nf-core/rnafusion 1.0 1 months ago - -
nf-core/methylseq 1.3 1 months ago 3 months ago No
nf-core/ampliseq 1.0.0 3 months ago - -
nf-core/deepvariant 1.0 4 months ago - -
nf-core/atacseq dev - 1 months ago No
nf-core/bacass dev - - -
nf-core/bcellmagic dev - - -
nf-core/chipseq dev - 1 months ago No
nf-core/clinvap dev - - -
```

To narrow down the list, supply one or more additional keywords to filter the pipelines based on matches in titles, descriptions and topics:

```bash
nf-core list rna rna-seq
```console
$ nf-core list rna rna-seq

,--./,-.
___ __ __ __ ___ /,-._.--~\
Expand All @@ -85,16 +100,109 @@ nf-core list rna rna-seq
`._,._,'


Name Version Published Last Pulled Default local is latest release?
---------------- --------- ------------ ------------- ----------------------------------
nf-core/rnaseq 1.0 20 hours ago - No
nf-core/smrnaseq dev - - No
Name Version Released Last Pulled Have latest release?
----------------- --------- ------------ -------------- ----------------------
nf-core/rnaseq 1.3 4 days ago 28 minutes ago Yes
nf-core/rnafusion 1.0 1 months ago - -
nf-core/lncpipe dev - - -
nf-core/smrnaseq dev - - -
```

You can sort the results by latest release (default), name (alphabetical) or number of GitHub stars using the `-s`/`--stars` option.
You can sort the results by latest release (`-s release`, default),
when you last pulled a local copy (`-s pulled`),
alphabetically (`-s name`),
or number of GitHub stars (`-s stars`).

```console
$ nf-core list -s stars

,--./,-.
___ __ __ __ ___ /,-._.--~\
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'


Name Stargazers Version Released Last Pulled Have latest release?
------------------------- ------------ --------- ------------ -------------- ----------------------
nf-core/rnaseq 81 1.3 4 days ago 30 minutes ago Yes
nf-core/methylseq 22 1.3 1 months ago 3 months ago No
nf-core/ampliseq 21 1.0.0 3 months ago - -
nf-core/chipseq 20 dev - 1 months ago No
nf-core/deepvariant 15 1.0 4 months ago - -
nf-core/eager 14 2.0.6 3 weeks ago - -
nf-core/rnafusion 14 1.0 1 months ago - -
nf-core/lncpipe 9 dev - - -
nf-core/exoseq 8 dev - - -
nf-core/mag 8 dev - - -
```

Finally, to return machine-readable JSON output, use the `--json` flag.

## Launch a pipeline
Some nextflow pipelines have a considerable number of command line flags that can be used.
To help with this, the `nf-core launch` command uses an interactive command-line wizard tool to prompt you for
values for running nextflow and the pipeline parameters.

If the pipeline in question has a `parameters.settings.json` file following the [nf-core parameter JSON schema](https://nf-co.re/parameters.schema.json), parameters will be grouped and have associated description text and variable typing.

Nextflow `params` variables are saved in to a JSON file called `nfx-params.json` and used by nextflow with the `-params-file` flag.
This makes it easier to reuse these in the future.

It is not essential to run the pipeline - the wizard will ask you if you want to launch the command at the end.
If not, you finish with the `params` JSON file and a nextflow command that you can copy and paste.

```console
$ nf-core launch rnaseq

,--./,-.
___ __ __ __ ___ /,-._.--~\
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'


INFO: Launching nf-core/rnaseq
Main nextflow options

Config profile to use
-profile [standard]: docker

Unique name for this nextflow run
-name [None]: test_run

Work directory for intermediate files
-w [./work]:

Resume a previous workflow run
-resume [y/N]:

Release / revision to use
-r [None]: 1.3


Parameter group: Main options
Do you want to change the group's defaults? [y/N]: y

Input files
Specify the location of your input FastQ files.
--reads ['data/*{1,2}.fastq.gz']: '/path/to/reads_*{R1,R2}.fq.gz'

[..truncated..]

Nextflow command:
nextflow run nf-core/rnaseq -profile "docker" -name "test_run" -r "1.3" --params-file "/Users/ewels/testing/nfx-params.json"


Do you want to run this command now? [y/N]: y

INFO: Launching workflow!
N E X T F L O W ~ version 19.01.0
Launching `nf-core/rnaseq` [evil_engelbart] - revision: 37f260d360 [master]

[..truncated..]
```


## Downloading pipelines for offline use
Sometimes you may need to run an nf-core pipeline on a server or HPC system that has no internet connection. In this case you will need to fetch the pipeline files first, then manually transfer them to your system.
Expand All @@ -103,7 +211,7 @@ To make this process easier and ensure accurate retrieval of correctly versioned

By default, the pipeline will just download the pipeline code. If you specify the flag `--singularity`, it will also download any singularity image files that are required.

```bash
```console
$ nf-core download methylseq --singularity

,--./,-.
Expand All @@ -124,7 +232,7 @@ INFO: Downloading 1 singularity container
nf-core-methylseq-1.0.simg [762.28MB] [####################################] 780573/780572
```

```bash
```console
$ tree -L 2 nf-core-methylseq-1.0/

nf-core-methylseq-1.0/
Expand All @@ -150,7 +258,7 @@ nf-core-methylseq-1.0/
## Pipeline software licences
Sometimes it's useful to see the software licences of the tools used in a pipeline. You can use the `licences` subcommand to fetch and print the software licence from each conda / PyPI package used in an nf-core pipeline.

```bash
```console
$ nf-core licences rnaseq

,--./,-.
Expand Down Expand Up @@ -191,7 +299,7 @@ With a given pipeline name, description and author, it makes a starter pipeline
After creating the files, the command initialises the folder as a git repository and makes an initial commit. This first "vanilla" commit which is identical to the output from the templating tool is important, as it allows us to keep your pipeline in sync with the base template in the future.
See the [nf-core syncing docs](http://nf-co.re/sync) for more information.

```bash
```console
$ nf-core create

,--./,-.
Expand All @@ -217,7 +325,7 @@ INFO: Done. Remember to add a remote and push to GitHub:
Once you have run the command, create a new empty repository on GitHub under your username (not the `nf-core` organisation, yet).
On your computer, add this repository as a git remote and push to it:

```bash
```console
git remote add origin https://github.com/ewels/nf-core-nextbigthing.git
git push --set-upstream origin master
```
Expand All @@ -235,7 +343,7 @@ This is the same test that is used on the automated continuous integration tests

For example, the current version looks something like this:

```bash
```console
$ cd path/to/my_pipeline
$ nf-core lint .

Expand Down Expand Up @@ -268,7 +376,7 @@ The command uses results from the linting process, so will only work with workfl

Usage is `nf-core bump-version <pipeline_dir> <new_version>`, eg:

```bash
```console
$ cd path/to/my_pipeline
$ nf-core bump-version . 1.0

Expand Down
1 change: 0 additions & 1 deletion bin/blacklist.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
"exoseq",
"nascent",
"neutronstar",
"rnaseq",
"smrnaseq",
"vipr"
]
Expand Down
Loading

0 comments on commit 4ab80b8

Please sign in to comment.