(Preserve old feature) - use of Channel.create() to separate Channel logic from process logic

[micans]

Channel.create() allows a different way of organising the code, separating all channel code from all process code (see below). I am not 100% convinced by the argument that Channel.create() is misused; there are also cases where it is useful, and I am now running a Pipeline That Shall Not Be Named where the authors do all sorts of hell-raising tricks with most of the NF language, e.g.

    beforeScript "rsync -az ${outdir}/* ."
    afterScript "rsync -az ./* ${outdir}/"

(Bad code can be written in any language). I'll end with a small example of the channel/process separation; would this still be supported in DSL2 in some other way? create() can be avoided by rearranging the code but the point is that by grouping all the channel logic together the flow logic of the program is easier to grasp (imagine this scaling to 10 processes with as least as many channels between them).

This use case is quite possibly not on anyone else's radar and I don't want to make it into a huge thing. I encountered it only very recently and think it's cool. I'll try to link to a more fully fledge example ASAP.

// This is a variant on http://nextflow-io.github.io/patterns/index.html#_problem_19
params.includeB = false

ch_input = Channel.create()
ch_BC = Channel.create()

ch_input.flatMap().map { f -> [f.text.trim(), f] }.view()
  .tap { ch_AC }
  .until { !params.includeB }
  .set { ch_AB }

ch_AC.until { params.includeB }.mix(ch_BC).set{ ch_C }

// Above is all the channel plumbing, below are all the process definitions.

process processA {        // Create a bunch of files, each with its ow sample ID.
  output: file('*.txt') into ch_input
  script: 'for i in {1..7}; do echo "sample_$i" > f$i.txt; done'
}

process processB {
  input:  set val(sampleid), file(thefile) from ch_AB
  output: set val(sampleid), file('out.txt') into ch_BC
  script: "(echo 'B process'; cat $thefile; md5sum $thefile) > out.txt"
}

process processC {
  publishDir "results"
  input: set val(sampleid), file(a) from ch_C.view()
  output: file('*.out')
  script: "(echo 'C process'; cat $a) > ${sampleid}.out"
}

[pditommaso]

In the above example, it's enough to move the snippet

ch_input.flatMap().map { f -> [f.text.trim(), f] }.view()
  .tap { ch_AC }
  .until { !params.includeB }
  .set { ch_AB }

after processA and the ch_input = Channel.create() is not needed any more.

The main problem with create is that with the new syntax the channel emission needs to be deferred after the script evaluation to infer the complete full datatflow network and allow the implicit forking of multi-read channels.

The create would give the user the ability to break this model causing the execution to stall.

However, the idea is to not break existing code without bothering users who do not want to move to the new syntax. Therefore I've removed the deprecation messages for create and close methods 454e78f.

[micans]

Well this particular example is too small to be convincing, but anyway, it's great if these methods will not be deprecated!

[pditommaso]

@JohnHadish can you provide an example of how you are Channel.create() and why the code could not be refactored without using it?