An input channel evaluating to null results in a misleading error message

[ewels]

Working test case

minimal.nf:

params.download_fasta = false
process downloadFASTA {
    input: val dl_url from params.download_fasta
    when: params.download_fasta
    script:
    """
    curl -O -L $dl_url
    """
}

nextflow.config:

params.bar = 'foo'
process {
  foo = params.bar
}

Works as expected:

$ nextflow run minimal.nf
N E X T F L O W  ~  version 0.22.4
Launching `minimal.nf` [agitated_hodgkin] - revision: b3b3101f7c
[warm up] executor > local

Breaking test case

Make this change tonextflow.config:

// params.bar = 'foo'
process {
  foo = params.bar
}

Now breaks:

$ nextflow run minimal.nf
N E X T F L O W  ~  version 0.22.4
Launching `minimal.nf` [tiny_nobel] - revision: b3b3101f7c
[warm up] executor > local
ERROR ~ No such variable: dl_url

 -- Check script 'minimal.nf' at line: 2 or see '.nextflow.log' file for more details

Working again:

Weirdly, changing the syntax as follows fixes it again:
nextflow.config:

// params.bar = 'foo'
process.foo = params.bar

$ nextflow run minimal.nf
N E X T F L O W  ~  version 0.22.4
Launching `minimal.nf` [compassionate_kalam] - revision: b3b3101f7c
[warm up] executor > local

Any ideas what's going on here? @viklund also testing, will probably follow with his own minimal example..

[viklund]

I've managed to get it down to one file:

$ cat test.nf
//params.value = "hola"
process buggy {
    input:
        val v from params.value

    script:
    """
    echo $v
    """
}

The example requires running these without anything specified on the command line (that is, no --value 5).

When running this fails. It works with the params.value line not commented. The error message is the same as @ewels got:

$ nextflow run test.nf
N E X T F L O W  ~  version 0.22.4
Launching `test.nf` [pensive_nobel] - revision: 71e67de276
[warm up] executor > local
ERROR ~ No such variable: v

 -- Check script 'test.nf' at line: 2 or see '.nextflow.log' file for more details

If nothing else, the error message should be referring to params.value.

I can also get this error message if I refer to a params var in the process scope of the config file. To me it looks like the problem is with an uninitialized params data structure.

[pditommaso]

OK, it can be reduced to this

process buggy {
    input:
    val v from null

    script:
    """
    echo $v
    """
}

[pditommaso]

This brings to an interesting question? Should a not defined params return null or throw an exception?

[ewels]

But does your example also cover my failure above where I'm not changing anything to do with the process? There params.download_fasta is not null, it's false...

[ewels]

Same error message, but I suspect that the mode of failure is different (probably why we found testing this so confusing this morning). Could it be anything to do with #48?

[ewels]

And in answer to your question - I think throwing an exception is fine, as long as when: returns false when the input is null so that the process is skipped. That was the behaviour that I expected and was trying to achieve...

[pditommaso]

But does your example also cover my failure above where I'm not changing anything to do with the process? There params.download_fasta is not null, it's false

This is slight different. The main problem is that the following snippet is assigning to foo a non existing parameter. For some causes that I'm still investigating that corrupts the process definition.

I've added a more strict check warning on that, however the general advice is to define always the parameter before use them.

I've added a new snapshot, you may want to check if solve your problem. You can try it by using:

NXF_VER=0.22.5-SNAPSHOT nextflow run ... etc

[ewels]

ok, so the thing to remember is to define param variables in config files before using them.. Makes sense I think. The use case I had was as follows:

process {
  clusterOptions: "-A $params.project"
}

So I should do the following:

params {
  project = false
}
process {
  clusterOptions: "-A $params.project"
}

I'll try this now. But specifying --project on the command line should still work in this case? When is clusterOptions evaluated?

[pditommaso]

Yes, parameter priority is the following (from higher to lower)

1. command line
2. config file
3. script file

[pditommaso]

Included in version 0.22.5

[ewels]

To follow up, my example above was a little wrong as I was missing {} around the clusterOptions. Once I had those, it was evaluated later in the pipeline after params.project had been updated by other config files, and it worked.