nextflow.config not detected when pulling from repository #1032

cimendes · 2019-02-13T17:01:02Z

Bug report

Expected behavior and actual behavior

My nextflow pipeline (https://github.com/assemblerflow/DEN-IM) contains various config files included in the nextflow.config through the "includeConfig" parameter. This pipeline works fine when cloned. When pulled directly through nextflow, it fails with the following error:

$ nextflow run assemblerflow/DEN-IM --fastq fastq/*_R{1,2}*.fastq.gz
N E X T F L O W  ~  version 19.01.0
Pulling assemblerflow/DEN-IM ...
Not a valid Nextflow project -- The repository `https://github.com/assemblerflow/DEN-IM` must contain a the script `main.nf` or the file `nextflow.config`

Steps to reproduce the problem

nextflow run assemblerflow/DEN-IM

Program output

Pulling assemblerflow/DEN-IM ...
Not a valid Nextflow project -- The repository https://github.com/assemblerflow/DEN-IM must contain a the script main.nf or the file nextflow.config
nextflow.log

Environment

Nextflow version: 19.01.0
Java version: openjdk version "1.8.0_181"
Operating system: Arch Linux

Additional context

Pipelines with only one config file pull without any issue for me, so my idea is that this is related to the multiple config file approach. The manifest information, including the mainScript field, is included in the manifest.config file.

The text was updated successfully, but these errors were encountered:

cimendes · 2019-02-13T17:42:00Z

After some digging in nextflow's code, I saw that by default the IncludeConfigs are not included in the validation of the pipeline before the pull (

nextflow/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy

Line 431 in ae04fc3

def config = new ConfigParser().setIgnoreIncludes(true).parse(text)

) and in my case it's failing the verification because in the nextflow.config it cannot find the manifest stating the scriptname, and the nextflow file isn't called main.nf. Any reason for this?

pditommaso · 2019-02-13T20:00:12Z

Not 100% sure, maybe the setIgnoreIncludes(true) is a legacy requirement that can be removed. Need to check.

huguesfontenelle · 2019-02-25T09:17:07Z

I can confirm this is a bug.

Here I made a minimum example, that defines a dictionary in the nextflow.config. Since the configuration is not read, it results in a "Access to undefined parameter" error.

Note that this breaks the nf-core pipelines, since they are written with this kind of configuration.
It also blocks my Kubernetes implementation.

Also, respectfully, I'd set this one higher than pri/low, what do you think?

Steps to reproduce the problem

nextflow run huguesfontenelle/nextflow-bug-config

Program output

N E X T F L O W  ~  version 19.01.0
Launching `huguesfontenelle/nextflow-bug-config` [marvelous_wing] - revision: e4b985abff [dev]
WARN: Access to undefined parameter `genome` -- Initialise it to a default value eg. `params.genome = some_value`
ERROR ~ Cannot get property 'genomeFile' on null object

 -- Check script 'main.nf' at line: 4 or see '.nextflow.log' file for more details

pditommaso · 2019-02-26T07:08:05Z

@huguesfontenelle your script is wrong because it's using params.genome that is not define, actually as warned by NF

WARN: Access to undefined parameter `genome` -- Initialise it to a default value eg. `params.genome = some_value`

pditommaso · 2019-02-26T10:49:36Z

@cimendes I'm unable to replicate the problem (tho is failing for an apparent application issue):

nextflow run https://github.com/assemblerflow/DEN-IM
N E X T F L O W  ~  version 19.01.0
Launching `assemblerflow/DEN-IM` [soggy_visvesvaraya] - revision: 619d70ea6d [master]

============================================================
                F L O W C R A F T
============================================================
Built using flowcraft v1.4.0

 Input FastQ                 : 0
 Input samples               : 0
 Reports are found in        : ./reports
 Results are found in        : ./results
 Profile                     : standard

Starting pipeline at Tue Feb 26 11:48:41 CET 2019

ERROR ~ .treeDag.json (No such file or directory)

huguesfontenelle · 2019-02-26T14:32:07Z

Yes, I had not managed to make a minimal example...

The new one should work:

remote run

nextflow run huguesfontenelle/nextflow-bug-config --genome smallGRCh37

N E X T F L O W  ~  version 19.01.0
Launching `huguesfontenelle/nextflow-bug-config` [sick_wozniak] - revision: 88fad15780 [master]
WARN: Access to undefined parameter `genomes` -- Initialise it to a default value eg. `params.genomes = some_value`
ERROR ~ Genome smallGRCh37 not found in configuration

 -- Check '.nextflow.log' file for details

local run

nextflow run pipelines/mapping.nf --genome smallGRCh37

N E X T F L O W  ~  version 19.01.0
Launching `pipelines/mapping.nf` [nauseous_cuvier] - revision: 329ad7c4ac
[max_cpus:2, max_memory:7 GB, max_time:1h, outDir:/Users/hugues/learn/nextflow-bug-config, out-dir:/Users/hugues/learn/nextflow-bug-config, genomes:[smallGRCh37:[cosmic:b37_cosmic_v74.noCHR.sort.4.1.small.vcf.gz, cosmicIndex:b37_cosmic_v74.noCHR.sort.4.1.small.vcf.gz.tbi, dbsnp:dbsnp_138.b37.small.vcf.gz, dbsnpIndex:dbsnp_138.b37.small.vcf.gz.tbi, genomeFile:human_g1k_v37_decoy.small.fasta, bwaIndex:human_g1k_v37_decoy.small.fasta.{amb,ann,bwt,pac,sa}, genomeDict:human_g1k_v37_decoy.small.dict, genomeIndex:human_g1k_v37_decoy.small.fasta.fai, knownIndels:{1000G_phase1,Mills_and_1000G_gold_standard}.indels.b37.small.vcf, knownIndelsIndex:{1000G_phase1,Mills_and_1000G_gold_standard}.indels.b37.small.vcf.idx]], genome:smallGRCh37]

(as expected for the local run)

But then it doesn't have to do with nextflow.config not detected. It is something else ...

EDIT I created a new bug report, since this seems different. #1053

cimendes · 2019-02-27T12:02:54Z

Apologizes, I added the manifest field to the nextflow.config file in the master branch of my repository and the problem that you bumped into is unrelated to this issue. Here's the corrected steps to reproduce my problem:

Steps to reproduce the problem

nextflow run assemblerflow/DEN-IM -r manifesto

Program output

Pulling assemblerflow/DEN-IM ...
Not a valid Nextflow project -- The repository https://github.com/assemblerflow/DEN-IM must contain a the script main.nf or the file nextflow.config
nextflow.log

Environment

Nextflow version: 19.01.0
Java version: openjdk version "1.8.0_181"
Operating system: Arch Linux

Additional context

Pipelines with only one config file pull without any issue for me, so my idea is that this is related to the multiple config file approach. The manifest information, including the mainScript field, is included in the manifest.config file.

pditommaso · 2019-03-13T14:46:23Z

Still getting

Launching `assemblerflow/DEN-IM` [pedantic_mccarthy] - revision: 20f4d98fde [manifesto]

============================================================
                F L O W C R A F T
============================================================
Built using flowcraft v1.4.0

 Input FastQ                 : 0
 Input samples               : 0
 Reports are found in        : ./reports
 Results are found in        : ./results
 Profile                     : standard

Starting pipeline at Wed Mar 13 15:42:56 CET 2019

ERROR ~ .treeDag.json (No such file or directory)

Closing this because I'm not able to replicate the problem. Please report a self-contained issue if you continue to experience the problem.

srikarchamala · 2019-08-24T20:02:52Z

Apologizes, I added the manifest field to the nextflow.config file in the master branch of my repository and the problem that you bumped into is unrelated to this issue. Here's the corrected steps to reproduce my problem:

Steps to reproduce the problem

nextflow run assemblerflow/DEN-IM -r manifesto

Program output

Pulling assemblerflow/DEN-IM ...
Not a valid Nextflow project -- The repository https://github.com/assemblerflow/DEN-IM must contain a the script main.nf or the file nextflow.config
nextflow.log

Environment

Nextflow version: 19.01.0

Java version: openjdk version "1.8.0_181"

Operating system: Arch Linux

Additional context

Pipelines with only one config file pull without any issue for me, so my idea is that this is related to the multiple config file approach. The manifest information, including the mainScript field, is included in the manifest.config file.

I am facing same issue. How did you fix this?

cimendes · 2019-08-24T20:34:07Z

Hello. You need to have the manifest scope in your nextflow.config file :) it can't be imported from an external config file through the includeConfig

pditommaso added the pri/low label Feb 13, 2019

cimendes mentioned this issue Feb 18, 2019

generated pipelines incompatible with nextflow's pipeline sharing assemblerflow/flowcraft#194

Closed

huguesfontenelle mentioned this issue Feb 27, 2019

_undefined parameter_ even though it is defined in a config file. #1053

Closed

pditommaso closed this as completed Mar 13, 2019

evanroyrees mentioned this issue Mar 11, 2021

Nextflow implementation template KwanLab/Autometa#118

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nextflow.config not detected when pulling from repository #1032

nextflow.config not detected when pulling from repository #1032

cimendes commented Feb 13, 2019

cimendes commented Feb 13, 2019

pditommaso commented Feb 13, 2019

huguesfontenelle commented Feb 25, 2019 •

edited

Loading

pditommaso commented Feb 26, 2019

pditommaso commented Feb 26, 2019

huguesfontenelle commented Feb 26, 2019 •

edited

Loading

cimendes commented Feb 27, 2019

pditommaso commented Mar 13, 2019

srikarchamala commented Aug 24, 2019

Steps to reproduce the problem

Program output

Environment

Additional context

cimendes commented Aug 24, 2019

nextflow.config not detected when pulling from repository #1032

nextflow.config not detected when pulling from repository #1032

Comments

cimendes commented Feb 13, 2019

Bug report

Expected behavior and actual behavior

Steps to reproduce the problem

Program output

Environment

Additional context

cimendes commented Feb 13, 2019

pditommaso commented Feb 13, 2019

huguesfontenelle commented Feb 25, 2019 • edited Loading

Steps to reproduce the problem

Program output

pditommaso commented Feb 26, 2019

pditommaso commented Feb 26, 2019

huguesfontenelle commented Feb 26, 2019 • edited Loading

remote run

local run

cimendes commented Feb 27, 2019

Steps to reproduce the problem

Program output

Environment

Additional context

pditommaso commented Mar 13, 2019

srikarchamala commented Aug 24, 2019

Steps to reproduce the problem

Program output

Environment

Additional context

cimendes commented Aug 24, 2019

huguesfontenelle commented Feb 25, 2019 •

edited

Loading

huguesfontenelle commented Feb 26, 2019 •

edited

Loading