Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev -> Master for 2.5 release #315

Merged
merged 70 commits into from
Jul 13, 2022
Merged
Show file tree
Hide file tree
Changes from 68 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
aea5c83
Bump pipeline version to 2.5dev
drpatelh Mar 4, 2022
12833c4
Merge pull request #293 from drpatelh/bump
drpatelh Mar 10, 2022
98aba35
Template update for nf-core/tools version 2.3
nf-core-bot Mar 15, 2022
edd8bb9
Template update for nf-core/tools version 2.3.1
nf-core-bot Mar 23, 2022
54a3225
Template update for nf-core/tools version 2.3.2
nf-core-bot Mar 24, 2022
8bb920e
Fix merge conflicts
drpatelh Mar 31, 2022
c7e62dd
Update CHANGELOG after merging template updates
drpatelh Mar 31, 2022
5fa2191
Fix template merge bug
drpatelh Mar 31, 2022
bd19710
Remove TODO statement in README
drpatelh Apr 1, 2022
952d10a
Remove retry logic from AWS actions
drpatelh Apr 1, 2022
3d5c560
Add .prettierignore file
drpatelh Apr 1, 2022
408b31e
Fix ECLint
drpatelh Apr 1, 2022
eb6f554
Merge pull request #301 from drpatelh/sync
drpatelh Apr 1, 2022
8ed4fa6
Merge branch 'dev' of github.com:BU-ISCIII/viralrecon into dev
saramonzon Apr 12, 2022
3ef2937
start reformat
saramonzon Apr 12, 2022
293664e
draft reformat
saramonzon Apr 12, 2022
fe8adb2
vcf conversion and strand bias working
saramonzon Apr 12, 2022
2f14d06
fixed bugs, finished first debugging
saramonzon Apr 13, 2022
1514898
black linting
saramonzon Apr 13, 2022
780be42
comments and function reordering
saramonzon Apr 13, 2022
a4067ec
fix variants iterator
saramonzon Apr 13, 2022
a5a0a40
fixed bug in format field and last lines handling
saramonzon Apr 13, 2022
9bff8a5
minor fixed comment
saramonzon Apr 13, 2022
62d3302
updated changelog
saramonzon Apr 14, 2022
7c787e0
Merge pull request #304 from BU-ISCIII/dev
drpatelh Apr 14, 2022
5d0d14b
added fasta arg and contig field to vcf header
saramonzon Apr 19, 2022
70dceae
fixed filter field
saramonzon Apr 19, 2022
68a05d4
installed bcftools sort module
saramonzon Apr 19, 2022
04fceef
added bcftools sort step in ivar_variants subworkflow
saramonzon Apr 19, 2022
eb9cf95
removed sorting with awk in ivar_variants_to_vcf module
saramonzon Apr 19, 2022
e5e8aa0
fixed import paths in modules
saramonzon Apr 19, 2022
4ccd89d
add fasta as input for ivar_variants_to_vcf module
saramonzon Apr 19, 2022
91ec8ab
black lint
saramonzon Apr 19, 2022
aa22edf
updated mulled container for ivar_variants_to_vcf module
saramonzon Apr 22, 2022
9136bd9
fixed bcftools sort in variants_ivar subworkflow and config
saramonzon Apr 22, 2022
9eb97c4
Replace vanilla Linux Biocontainer with Ubuntu
drpatelh Apr 25, 2022
870bd01
Replace vanilla Linux Biocontainer with Ubuntu
drpatelh Apr 25, 2022
ffa8ac2
Merge pull request #307 from drpatelh/sync
drpatelh Apr 25, 2022
5bdb329
Merge branch 'nf-core:dev' into dev
saramonzon Apr 26, 2022
423a840
Updated changelog
saramonzon Apr 26, 2022
46e1928
prettier lint modules.json
saramonzon Apr 26, 2022
eba9969
Template update for nf-core/tools version 2.4
nf-core-bot May 16, 2022
483f115
Fix merge conflicts
drpatelh May 17, 2022
02a8487
Merge branch 'dev' into dev
saramonzon Jun 4, 2022
9d193d0
Merge pull request #308 from BU-ISCIII/dev
drpatelh Jul 7, 2022
2919d30
Follow up fix for PR #308
drpatelh Jul 7, 2022
5c07e17
Run nf-core modules update
drpatelh Jul 7, 2022
8049c25
Intentionally ignore expected nf-core warnings
drpatelh Jul 7, 2022
f106faf
Re-install nf-core modules after updating tool versions
drpatelh Jul 7, 2022
7561671
Re-install nf-core modules after updating tool versions
drpatelh Jul 7, 2022
f770367
Fix tests after updating tool versions for all modules
drpatelh Jul 8, 2022
92a0ce3
Revert version bumps for SnpEff and SnpSift
drpatelh Jul 8, 2022
734a4da
Fix incorrect container definitions for SnpEff and SnpSift
drpatelh Jul 8, 2022
f2c5adf
Fix ECLint
drpatelh Jul 8, 2022
a1f810b
Fix #292
drpatelh Jul 8, 2022
6a0bc31
Fix #292 again
drpatelh Jul 8, 2022
af9cfe8
Fix config bug - see issue 1722 on MultiQC repo
drpatelh Jul 11, 2022
95c0cc2
Add print statement back into ivar_variants_to_vcf.py scrip
drpatelh Jul 11, 2022
7464f88
Fix Pangolin and Nextclade lineage parsing issues
drpatelh Jul 11, 2022
678e235
Fix publishDir bug introduced in PR 308
drpatelh Jul 11, 2022
cb7efb7
Fix #311
drpatelh Jul 11, 2022
81e03d5
Fix #234
drpatelh Jul 11, 2022
ab88946
Bump pipeline version to 2.5
drpatelh Jul 11, 2022
cf4e8df
Fix publishing config relative to v2.4.1
drpatelh Jul 11, 2022
62b73dc
Update CHANGELOG
drpatelh Jul 11, 2022
1a8a3c0
Add manual version definition for artic modules
drpatelh Jul 12, 2022
96c5d3e
Update docs to update default Nextclade version being used
drpatelh Jul 12, 2022
429a5ff
Merge pull request #314 from drpatelh/sync
drpatelh Jul 12, 2022
f7f8165
Fix #316
drpatelh Jul 13, 2022
74ee2b6
Merge pull request #318 from drpatelh/sync
drpatelh Jul 13, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions .editorconfig
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,9 @@ trim_trailing_whitespace = true
indent_size = 4
indent_style = space

[*.{yml,yaml}]
[*.{md,yml,yaml,html,css,scss,js}]
indent_size = 2

[*.json]
insert_final_newline = unset

# These files are edited and tested upstream in nf-core/modules
[/modules/nf-core/**]
charset = unset
Expand Down
15 changes: 7 additions & 8 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@ Contributions to the code are even more welcome ;)

If you'd like to write some code for nf-core/viralrecon, the standard workflow is as follows:

1. Check that there isn't already an issue about your idea in the [nf-core/viralrecon issues](https://github.com/nf-core/viralrecon/issues) to avoid duplicating work
* If there isn't one already, please create one so that others know you're working on this
1. Check that there isn't already an issue about your idea in the [nf-core/viralrecon issues](https://github.com/nf-core/viralrecon/issues) to avoid duplicating work. If there isn't one already, please create one so that others know you're working on this
2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [nf-core/viralrecon repository](https://github.com/nf-core/viralrecon) to your GitHub account
3. Make the necessary changes / additions within your forked repository following [Pipeline conventions](#pipeline-contribution-conventions)
4. Use `nf-core schema build` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10).
Expand Down Expand Up @@ -49,9 +48,9 @@ These tests are run both with the latest available version of `Nextflow` and als

:warning: Only in the unlikely and regretful event of a release happening with a bug.

* On your own fork, make a new branch `patch` based on `upstream/master`.
* Fix the bug, and bump version (X.Y.Z+1).
* A PR should be made on `master` from patch to directly this particular bug.
- On your own fork, make a new branch `patch` based on `upstream/master`.
- Fix the bug, and bump version (X.Y.Z+1).
- A PR should be made on `master` from patch to directly this particular bug.

## Getting help

Expand All @@ -73,7 +72,7 @@ If you wish to contribute a new step, please use the following coding standards:
6. Add sanity checks and validation for all relevant parameters.
7. Perform local tests to validate that the new code works as expected.
8. If applicable, add a new test command in `.github/workflow/ci.yml`.
9. Update MultiQC config `assets/multiqc_config.yaml` so relevant suffixes, file name clean up and module plots are in the appropriate order. If applicable, add a [MultiQC](https://https://multiqc.info/) module.
9. Update MultiQC config `assets/multiqc_config.yml` so relevant suffixes, file name clean up and module plots are in the appropriate order. If applicable, add a [MultiQC](https://https://multiqc.info/) module.
10. Add a description of the output files and if relevant any appropriate images from the MultiQC report to `docs/output.md`.

### Default values
Expand All @@ -92,8 +91,8 @@ The process resources can be passed on to the tool dynamically within the proces

Please use the following naming schemes, to make it easy to understand what is going where.

* initial process channel: `ch_output_from_<process>`
* intermediate and terminal channels: `ch_<previousprocess>_for_<nextprocess>`
- initial process channel: `ch_output_from_<process>`
- intermediate and terminal channels: `ch_<previousprocess>_for_<nextprocess>`

### Nextflow version bumping

Expand Down
1 change: 0 additions & 1 deletion .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ name: Bug report
description: Report something that is broken or incorrect
labels: bug
body:

- type: markdown
attributes:
value: |
Expand Down
5 changes: 2 additions & 3 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,13 @@ Remember that PRs should be made against the dev branch, unless you're preparing

Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-core/viralrecon/tree/master/.github/CONTRIBUTING.md)
-->
<!-- markdownlint-disable ul-indent -->

## PR checklist

- [ ] This comment contains a description of changes (with reason).
- [ ] If you've fixed a bug or added code that should be tested, add tests!
- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-core/viralrecon/tree/master/.github/CONTRIBUTING.md)
- [ ] If necessary, also make a PR on the nf-core/viralrecon _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-core/viralrecon/tree/master/.github/CONTRIBUTING.md)
- [ ] If necessary, also make a PR on the nf-core/viralrecon _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
- [ ] Make sure your code lints (`nf-core lint`).
- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
- [ ] Usage Documentation in `docs/usage.md` is updated.
Expand Down
8 changes: 1 addition & 7 deletions .github/workflows/awsfulltest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,20 +18,14 @@ jobs:
platform: ["illumina", "nanopore"]
steps:
- name: Launch workflow via tower
uses: nf-core/tower-action@v2

uses: nf-core/tower-action@v3
with:
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
compute_env: ${{ secrets.TOWER_COMPUTE_ENV }}
pipeline: ${{ github.repository }}
revision: ${{ github.sha }}
workdir: s3://${{ secrets.AWS_S3_BUCKET }}/work/viralrecon/work-${{ github.sha }}
parameters: |
{
"outdir": "s3://${{ secrets.AWS_S3_BUCKET }}/viralrecon/results-${{ github.sha }}/platform_${{ matrix.platform }}"
}
profiles: test_full_${{ matrix.platform }},aws_tower
nextflow_config: |
process.errorStrategy = 'retry'
process.maxRetries = 3
9 changes: 2 additions & 7 deletions .github/workflows/awstest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,21 +10,16 @@ jobs:
if: github.repository == 'nf-core/viralrecon'
runs-on: ubuntu-latest
steps:
# Launch workflow using Tower CLI tool action
- name: Launch workflow via tower
uses: nf-core/tower-action@v2

uses: nf-core/tower-action@v3
with:
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
compute_env: ${{ secrets.TOWER_COMPUTE_ENV }}
pipeline: ${{ github.repository }}
revision: ${{ github.sha }}
workdir: s3://${{ secrets.AWS_S3_BUCKET }}/work/viralrecon/work-${{ github.sha }}
parameters: |
{
"outdir": "s3://${{ secrets.AWS_S3_BUCKET }}/viralrecon/results-test-${{ github.sha }}"
}
profiles: test,aws_tower
nextflow_config: |
process.errorStrategy = 'retry'
process.maxRetries = 3
1 change: 0 additions & 1 deletion .github/workflows/branch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ jobs:
run: |
{ [[ ${{github.event.pull_request.head.repo.full_name }} == nf-core/viralrecon ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]


# If the above check failed, post a comment on the PR explaining the failure
# NOTE - this doesn't currently work if the PR is coming from a fork, due to limitations in GitHub actions secrets
- name: Post PR comment
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,20 +14,20 @@ env:

jobs:
test:
name: Run workflow tests
name: Run pipeline with test data
# Only run on push if this is the nf-core dev branch (merged PRs)
if: ${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-core/viralrecon') }}
if: "${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-core/viralrecon') }}"
runs-on: ubuntu-latest
strategy:
matrix:
# Nextflow versions
include:
# Test pipeline minimum Nextflow version
- NXF_VER: '21.10.3'
NXF_EDGE: ''
- NXF_VER: "21.10.3"
NXF_EDGE: ""
# Test latest edge release of Nextflow
- NXF_VER: ''
NXF_EDGE: '1'
- NXF_VER: ""
NXF_EDGE: "1"
steps:
- name: Check out pipeline code
uses: actions/checkout@v2
Expand Down
55 changes: 55 additions & 0 deletions .github/workflows/fix-linting.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
name: Fix linting from a comment
on:
issue_comment:
types: [created]

jobs:
deploy:
# Only run if comment is on a PR with the main repo, and if it contains the magic keywords
if: >
contains(github.event.comment.html_url, '/pull/') &&
contains(github.event.comment.body, '@nf-core-bot fix linting') &&
github.repository == 'nf-core/viralrecon'
runs-on: ubuntu-latest
steps:
# Use the @nf-core-bot token to check out so we can push later
- uses: actions/checkout@v3
with:
token: ${{ secrets.nf_core_bot_auth_token }}

# Action runs on the issue comment, so we don't get the PR by default
# Use the gh cli to check out the PR
- name: Checkout Pull Request
run: gh pr checkout ${{ github.event.issue.number }}
env:
GITHUB_TOKEN: ${{ secrets.nf_core_bot_auth_token }}

- uses: actions/setup-node@v2

- name: Install Prettier
run: npm install -g prettier @prettier/plugin-php

# Check that we actually need to fix something
- name: Run 'prettier --check'
id: prettier_status
run: |
if prettier --check ${GITHUB_WORKSPACE}; then
echo "::set-output name=result::pass"
else
echo "::set-output name=result::fail"
fi

- name: Run 'prettier --write'
if: steps.prettier_status.outputs.result == 'fail'
run: prettier --write ${GITHUB_WORKSPACE}

- name: Commit & push changes
if: steps.prettier_status.outputs.result == 'fail'
run: |
git config user.email "core@nf-co.re"
git config user.name "nf-core-bot"
git config push.default upstream
git add .
git status
git commit -m "[automated] Fix linting with Prettier"
git push
84 changes: 13 additions & 71 deletions .github/workflows/linting.yml
Original file line number Diff line number Diff line change
@@ -1,49 +1,14 @@
name: nf-core linting
# This workflow is triggered on pushes and PRs to the repository.
# It runs the `nf-core lint` and markdown lint tests to ensure that the code meets the nf-core guidelines
# It runs the `nf-core lint` and markdown lint tests to ensure
# that the code meets the nf-core guidelines.
on:
push:
pull_request:
release:
types: [published]

jobs:
Markdown:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-node@v2
- name: Install markdownlint
run: npm install -g markdownlint-cli
- name: Run Markdownlint
run: markdownlint .

# If the above check failed, post a comment on the PR explaining the failure
- name: Post PR comment
if: failure()
uses: mshick/add-pr-comment@v1
with:
message: |
## Markdown linting is failing

To keep the code consistent with lots of contributors, we run automated code consistency checks.
To fix this CI test, please run:

* Install `markdownlint-cli`
* On Mac: `brew install markdownlint-cli`
* Everything else: [Install `npm`](https://www.npmjs.com/get-npm) then [install `markdownlint-cli`](https://www.npmjs.com/package/markdownlint-cli) (`npm install -g markdownlint-cli`)
* Fix the markdown errors
* Automatically: `markdownlint . --fix`
* Manually resolve anything left from `markdownlint .`

Once you push these changes the test should pass, and you can hide this comment :+1:

We highly recommend setting up markdownlint in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help!

Thanks again for your contribution!
repo-token: ${{ secrets.GITHUB_TOKEN }}
allow-repeats: false

EditorConfig:
runs-on: ubuntu-latest
steps:
Expand All @@ -55,47 +20,24 @@ jobs:
run: npm install -g editorconfig-checker

- name: Run ECLint check
run: editorconfig-checker -exclude README.md $(git ls-files | grep -v test)
run: editorconfig-checker -exclude README.md $(find .* -type f | grep -v '.git\|.py\|.md\|json\|yml\|yaml\|html\|css\|work\|.nextflow\|build\|nf_core.egg-info\|log.txt\|Makefile')

YAML:
Prettier:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: actions/setup-node@v2
- name: Install yaml-lint
run: npm install -g yaml-lint
- name: Run yaml-lint
run: yamllint $(find ${GITHUB_WORKSPACE} -type f -name "*.yml" -o -name "*.yaml") -c ${GITHUB_WORKSPACE}/.yamllint.yml

# If the above check failed, post a comment on the PR explaining the failure
- name: Post PR comment
if: failure()
uses: mshick/add-pr-comment@v1
with:
message: |
## YAML linting is failing

To keep the code consistent with lots of contributors, we run automated code consistency checks.
To fix this CI test, please run:

* Install `yaml-lint`
* [Install `npm`](https://www.npmjs.com/get-npm) then [install `yaml-lint`](https://www.npmjs.com/package/yaml-lint) (`npm install -g yaml-lint`)
* Fix the markdown errors
* Run the test locally: `yamllint $(find . -type f -name "*.yml" -o -name "*.yaml") -c ./.yamllint.yml`
* Fix any reported errors in your YAML files
- uses: actions/checkout@v2

Once you push these changes the test should pass, and you can hide this comment :+1:
- uses: actions/setup-node@v2

We highly recommend setting up yaml-lint in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help!
- name: Install Prettier
run: npm install -g prettier

Thanks again for your contribution!
repo-token: ${{ secrets.GITHUB_TOKEN }}
allow-repeats: false
- name: Run Prettier --check
run: prettier --check ${GITHUB_WORKSPACE}

nf-core:
runs-on: ubuntu-latest
steps:

- name: Check out pipeline code
uses: actions/checkout@v2

Expand All @@ -106,10 +48,10 @@ jobs:
wget -qO- get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/

- uses: actions/setup-python@v1
- uses: actions/setup-python@v3
with:
python-version: '3.6'
architecture: 'x64'
python-version: "3.6"
architecture: "x64"

- name: Install dependencies
run: |
Expand Down
18 changes: 9 additions & 9 deletions .gitpod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@ image: nfcore/gitpod:latest

vscode:
extensions: # based on nf-core.nf-core-extensionpack
- codezombiech.gitignore # Language support for .gitignore files
# - cssho.vscode-svgviewer # SVG viewer
- davidanson.vscode-markdownlint # Markdown/CommonMark linting and style checking for Visual Studio Code
- eamodio.gitlens # Quickly glimpse into whom, why, and when a line or code block was changed
- EditorConfig.EditorConfig # override user/workspace settings with settings found in .editorconfig files
- Gruntfuggly.todo-tree # Display TODO and FIXME in a tree view in the activity bar
- mechatroner.rainbow-csv # Highlight columns in csv files in different colors
- codezombiech.gitignore # Language support for .gitignore files
# - cssho.vscode-svgviewer # SVG viewer
- esbenp.prettier-vscode # Markdown/CommonMark linting and style checking for Visual Studio Code
- eamodio.gitlens # Quickly glimpse into whom, why, and when a line or code block was changed
- EditorConfig.EditorConfig # override user/workspace settings with settings found in .editorconfig files
- Gruntfuggly.todo-tree # Display TODO and FIXME in a tree view in the activity bar
- mechatroner.rainbow-csv # Highlight columns in csv files in different colors
# - nextflow.nextflow # Nextflow syntax highlighting
- oderwat.indent-rainbow # Highlight indentation level
- streetsidesoftware.code-spell-checker # Spelling checker for source code
- oderwat.indent-rainbow # Highlight indentation level
- streetsidesoftware.code-spell-checker # Spelling checker for source code
16 changes: 0 additions & 16 deletions .markdownlint.yml

This file was deleted.

12 changes: 5 additions & 7 deletions .nf-core.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
repository_type: pipeline
lint:
files_unchanged:
- .markdownlint.yml
- assets/email_template.html
- assets/email_template.txt
- lib/NfcoreTemplate.groovy
- .github/ISSUE_TEMPLATE/bug_report.yml
- .github/PULL_REQUEST_TEMPLATE.md
- .github/workflows/branch.yml
- .github/workflows/linting_comment.yml
- .github/workflows/linting.yml
- lib/NfcoreSchema.groovy
files_exist:
- assets/multiqc_config.yml
- conf/igenomes.config
- lib/WorkflowViralrecon.groovy
Loading