Skip to content

Cc nf corrections#650

Open
MattWellie wants to merge 5 commits intomainfrom
cc_nf_corrections
Open

Cc nf corrections#650
MattWellie wants to merge 5 commits intomainfrom
cc_nf_corrections

Conversation

@MattWellie
Copy link
Copy Markdown
Collaborator

Fixes

  • Various issues identified through analysis with claude.
  • Some are soft inconsistencies (mixed use of case, missing shebangs, missing nextflow.enable.dsl=2 etc.)
  • Some are hard bugs (bcftools output written using wrong flag)
  • Some are incomplete changes (mix of workflow publishing and output directives, addressed by unmerged Workflow outputs only #645)
  • Some are functional but incorrect (Memory setting of "1 GB" instead of 1.GB - same result, but though string parsing instead of plain Groovy
  • Some are ideal but not essential (set -euo pipefail added to all script blocks), rm -f to prevent failure if files (which are definitely written) stop being written
  • Some corrections (indexing files during correct stage and passing VCF with index, instead of indexing an input, so the index would be lost in the working directory)
  • Adds missing script: directives to some files

Proposed Changes

  • Solve all of the above. Test in progress

Checklist

  • ChangeLog Updated
  • Version Bumped
  • Related Issue created
  • Tests covering new change
  • Linting checks pass

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR cleans up and standardizes the Talos Nextflow pipelines by moving further toward DSL2 + workflow outputs, tightening bash execution behavior across modules, and correcting several workflow/module inconsistencies.

Changes:

  • Standardize Nextflow DSL2 usage and workflow outputs (removing publishDir usage across many processes).
  • Harden module script execution (set -euo pipefail), adjust indexing/IO wiring for some VCF flows, and fix a bcftools output mode bug.
  • Add a nextflow_schema.json for parameter documentation/validation and update docs to reflect outputDir usage.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
talos_only.nf Adds DSL2/shebang and adjusts how cohort MT inputs are discovered for TALOS-only runs.
src/talos/cpg_internal_scripts/cpgflow_jobs/make_config.py Comment wording/grammar tweak.
nextflow_schema.json Introduces a Nextflow params JSON schema for the pipeline.
nextflow/modules/talos/ValidateMOI/main.nf Removes publishDir; adds stricter bash options.
nextflow/modules/talos/UnifiedPanelAppParser/main.nf Removes publishDir; adds missing script: + stricter bash options.
nextflow/modules/talos/StartupChecks/main.nf Removes publishDir comment; strengthens bash error handling.
nextflow/modules/talos/RunHailFiltering/main.nf Removes publishDir; adds stricter bash options.
nextflow/modules/talos/HPOFlagging/main.nf Removes publishDir; adds missing script: + stricter bash options.
nextflow/modules/talos/CreateTalosHTML/main.nf Removes publishDir; adds stricter bash options.
nextflow/modules/prep/ResummariseRawSubmissions/main.nf Emits VCF+index as a tuple, adds tabix indexing, and hardens cleanup.
nextflow/modules/prep/ParseManeIntoJson/main.nf Adds stricter bash options.
nextflow/modules/prep/ParseAlphaMissense/main.nf Adds stricter bash options.
nextflow/modules/prep/MakeClinvarbitrationPm5/main.nf Adds stricter bash options and makes cleanup tolerant.
nextflow/modules/prep/EncodeAlphaMissense/main.nf Adds stricter bash options.
nextflow/modules/prep/DownloadPanelApp/main.nf Adds stricter bash options.
nextflow/modules/prep/DownloadClinVarFiles/main.nf Adds stricter bash options in shell: block.
nextflow/modules/prep/CreateRoiFromGff3/main.nf Adds stricter bash options; minor comment adjustment.
nextflow/modules/prep/ConvertSpliceVarDb/main.nf Adds stricter bash options.
nextflow/modules/prep/AnnotateClinvarWithBcftools/main.nf Updates input signature to include VCF index; adds stricter bash options.
nextflow/modules/annotation/SplitVcf/main.nf Adds stricter bash options.
nextflow/modules/annotation/NormaliseAndRegionFilterVcf/main.nf Adds stricter bash options and indexes the input VCF.
nextflow/modules/annotation/MergeVcfsWithBcftools/main.nf Adds stricter bash options and fixes bcftools output mode to match .vcf.bgz.
nextflow/modules/annotation/AnnotatedVcfIntoMatrixTable/main.nf Adds stricter bash options; makes checkpoint cleanup tolerant.
nextflow/modules/annotation/AnnotateWithEchtvar/main.nf Adds stricter bash options.
nextflow/modules/annotation/AnnotateCsqWithBcftools/main.nf Adds stricter bash options and indexes before csq.
nextflow/annotation.nf Fixes Channel.fromPath capitalization/usage consistency.
nextflow.config Removes outdir, normalizes resource units to X.GB, and updates process resources.
main.nf Explicitly enables DSL2.
docs/NextflowConfiguration.md Removes outdir docs; documents outputDir instead.
README.md Updates output path documentation to use {outputDir}.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.map { row -> tuple(
row.cohort,
files("${params.outdir}/${row.cohort}_outputs/*.mt", type: 'dir'),
Channel.fromPath("${outputDir}/${row.cohort}_outputs/*.mt", checkIfExists: true),
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Channel.fromPath(...) inside the .map { row -> tuple(...) } produces a nested channel, but downstream TALOS inputs expect a concrete path/collection (e.g., the list of *.mt directories), not a Channel. This will break channel typing/tuple structure at runtime. Build the mts value as a path collection (e.g., via files(...) with type:'dir') and avoid creating a new channel per row; also prefer workflow.outputDir here (the bare outputDir identifier is not consistently used elsewhere in this script).

Suggested change
Channel.fromPath("${outputDir}/${row.cohort}_outputs/*.mt", checkIfExists: true),
files("${workflow.outputDir}/${row.cohort}_outputs/*.mt", type: 'dir'),

Copilot uses AI. Check for mistakes.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants