A few notes on static typing

[xsvato01]

Hi,

Over the past few months I've been following the plans to migrate Nextflow toward a fully statically typed language. After reading the migration docs, I can't shake the feeling that while static typing is clean and principled from a software-engineering standpoint, it also introduces a significant amount of boilerplate.

In my view, part of Nextflow's popularity comes from the fact that pipeline code is slim and quick to develop. I'm worried that this advantage is eroding, and I'd like to understand how the team sees these trade-offs. I have three concrete questions about how to reconcile real-world pipelines with the static-typing direction.

1. Is piping of outputs going away?

Consider this workflow, which is very readable thanks to piping:

workflow PloidyAnalyzer {
    take:
    sortedBowTie2Bams
    main:
    ploidyResults = SamtoolsMpileup(sortedBowTie2Bams)
        | MAFTableXlsx
        | CalcPloidyTablePerSample
        | map { meta, file -> [[name: meta.runName, method: meta.method], file] }
        | groupTuple
        | map { runMeta, paths ->
            def sortedByNumberName = paths.sort { path ->
                path.getName().split('-')[0]
            }
            return [runMeta, sortedByNumberName]
        }
        | MergePloidyTables
        | FilterPloidyTable
        | SummarizePloidyTable
        | LegendaryPloidyTable
        | FormatPloidyTable
    emit:
    ploidyResults
}

Does the static-typing policy imply that each output must be assigned to an explicitly typed variable that then feeds the next process? If so, this would inflate the code with many intermediate variables that exist only to be passed straight into the next step. Is piping still supported, and if not, is there a recommended pattern that preserves this readability?

2. What's the recommended way to use .map() and grouping?

I frequently work with tuples containing nested objects, and I often need to group records by specific fields of those objects. Today I do this with a workaround:

map { meta, panel, file -> [[meta.name, panel.name], meta, panel, file] }
| groupTuple()

Is there a cleaner approach under the new model — for example a groupBy that takes key extractors directly, like groupBy(meta.name, panel.name)?

3. How should deeply nested config objects be handled?

In params I encode per-process settings keyed by sample metadata. A single case can end up looking like this (showing only the tail end of the structure):

...
						autosomes: [
							minEntryZscore: 2.5,
							maxEntryZscore: -2.5,
							maxGlobalFreq: 0.10,
							maxInternalFreq: 0.10,
							maxRatio: -0.45,
							minRatio: 0.45,
							maxCopies: 1,
							minCopies: 3,
							maxSegMean: -0.4,
							minSegMean: 0.4,
							maxPredLargeSeg: 1.5,
							minPredLargeSeg: 3,
							maxZscore: -5,
							minZscore: 5,
						],
						gonosomes: [
							males: [
								minEntryZscore: 2.5,
								maxEntryZscore: -2.5,
								maxGlobalFreq: 0.10,
								maxInternalFreq: 0.10,
								maxRatio: -0.45,
								minRatio: 0.45,
								maxCopies: 0.5,
								minCopies: 2,
								maxSegMean: -0.4,
								minSegMean: 0.4,
								maxPredLargeSeg: 1.5,
								minPredLargeSeg: 3,
								maxZscore: -5,
								minZscore: 5,
							],
							females: [
								minEntryZscore: 2.5,
								maxEntryZscore: -2.5,
								maxGlobalFreq: 0.10,
								maxInternalFreq: 0.10,
								maxRatio: -0.45,
								minRatio: 0.45,
								maxCopies: 1,
								minCopies: 3,
								maxSegMean: -0.4,
								minSegMean: 0.4,
								maxPredLargeSeg: 1.5,
								minPredLargeSeg: 3,
								maxZscore: -5,
								minZscore: 5,
							],
						],
					],
				],
			],
		],

I'm fine with declaring a type for every integer/file in the config itself. My concern is what happens downstream: this nested object travels with each sample through the channel, and many processes consume it. Would I need to declare the full nested type as a typed input for every process that touches this metadata? That type definition alone would run close to a hundred lines and would be repeated across every process that uses it.

I can't flatten or rewrite how the parameters are consumed inside the processes, because these settings are applied combinatorially — which is their intended use.

I'd really appreciate guidance on the recommended patterns here, and I think clarifying these cases in the migration docs would help a lot of pipeline authors in a similar position. Thanks for all the work on Nextflow.

[bentsherman]

Thanks for the questions @xsvato01 . I will answer each one separately so we can discuss

Is piping of outputs going away?

Sadly yes. Pipes were a cool idea, and I actually like them and originally wanted to extend them to be more useful (see #3243). But in talking to people across the community, I concluded that pipes just aren't worth keeping.

Many people actually find it harder to read and understand (those intermediate variables help you remember what is being passed between each step). It creates "yet another way to do the same thing", which is already a big problem for Nextflow IMO. Channels and operators are what users struggle with most in Nextflow because they are the least familiar, and having multiple syntax variants (set op vs assignment, .out vs assignment, pipes vs assignment...) only makes that worse.

Your example is an extreme case where pipes help the most, but the vast majority of code does not seem to benefit much from pipes. In thinking about the community as a whole, I concluded it was better to have a simpler syntax that leads to more consistent code across the board, even if some of that code ends up being more verbose.

Syntax shortcuts like pipes can make some code snippets look nicer but it makes the language more complex, less consistent, harder for people to learn.

See also: https://github.com/nextflow-io/nextflow/blob/master/adr/20260310-typed-workflows.md#dataflow-syntax

[bentsherman]

These syntax restrictions are also discussed in the migration tutorial: https://docs.seqera.io/nextflow/tutorials/static-types#preparing-for-static-typing

[bentsherman]

What's the recommended way to use .map() and grouping?

With groupBy you could implement it like this:

// tuples
channel.of( tuple([:], 'panel1', 'file1'), /* ... */ )
  .map { meta, panel, file -> tuple(tuple(meta.name, panel.name), tuple(meta, panel, file)) }
  .groupBy()

// records
channel.of( record(meta: [:], panel: 'panel1', file: 'file1'), /* ... */ )
  .map { r -> tuple(tuple(r.meta.name, r.panel.name), r) }
  .groupBy()

tuple() and [] are effectively interchangeable, but only tuple() will pass the type checker. it is also more verbose. this is why I recommend migrating to records if you adopt static typing. Much easier to package data and pass it around

The groupBy + records will give you 2-tuples of the form (K, List<V>), where V is your record. So you will get a single list of records for each group rather than multiple slices

[bentsherman]

See also: https://docs.seqera.io/nextflow/tutorials/static-types-operators#groupby

I can add a code example like the one above to these docs

[bentsherman]

How should deeply nested config objects be handled?

At a minimum you should be able to model this as a generic Map or Record and continue to use it as before, it just won't be type-checked.

If you're willing to define a record type for it, that's great. It will be quite long, but you should be able to do that once and re-use it everywhere else.

For example, if you define a record type like so:

record Autosomes {
    minEntryZscore: Float
    maxEntryZscore: Float
    maxGlobalFreq: Float
    maxInternalFreq: Float
    maxRatio: Float
    minRatio: Float
    maxCopies: Integer
    minCopies: Integer
    maxSegMean: Float
    minSegMean: Float
    maxPredLargeSeg: Float
    minPredLargeSeg: Float
    maxZscore: Float
    minZscore: Float
}

You can include it and re-use it across others process:

include { Autosomes } from './types.nf'

process MY_PROCESS {
    input:
    record(
        autosomes: Autosomes,
        // ...
    )
}

And this extends naturally to nested record types like you have. I'm speculating a bit based on your description, but I'm happy to make it more concrete based on your particular needs

See also:

• https://docs.seqera.io/nextflow/process-typed#record-inputs
• https://docs.seqera.io/nextflow/reference/syntax#include

[xsvato01]

Hi Ben,

Thank you for your detailed responses, as always!

I suspect part of my hesitation is just the natural instinct to resist new changes, combined with the sheer amount of code I'd need to rewrite for the new syntax. That's probably what's left me unconvinced so far.

On the migration itself: are there any tutorials covering the newly introduced data types? I've seen <record> used a lot throughout the docs, and a dedicated chapter on the practical usage of these data types would be really helpful.

This is slightly off-topic, but while we're on the subject of updating code, I've also been looking at the new Workflow outputs. Is there a way to specify the output copying method? With the older process.publishDir, there were options like copy, symlink, hardlink, and so on. The older docs seemed more thorough here, but I can't find any equivalent information in the new ones.

I'm asking because I've had (and still have) problems publishing large files, which occasionally end up corrupted in the output. As a workaround, I've ended up using a dedicated process that publishes via rsync with checksum verification, which has been reliable for me so far:

process PublishFiles {
    tag "${meta.id} (${task.cpus} CPUs, ${task.memory})"
    containerOptions "-v ${params.publishMount}:/output/"
    label "rsync"
    label "s_cpu"
    label "m_mem"
    maxForks 5

    input:
    tuple val(meta), path(files)

    output:
    tuple val(meta), path(files)

    script:
    """
    rsync -avL --checksum ${files} /output/${meta.group}/align/
    """
}

It would be great to know whether there's a cleaner, built-in way to achieve the same thing in the new output syntax.

Thanks again for your help.

Jan

[bentsherman]

I suspect part of my hesitation is just the natural instinct to resist new changes, combined with the sheer amount of code I'd need to rewrite for the new syntax. That's probably what's left me unconvinced so far.

There's no pressure to migrate if your code works and you're happy with it. Static typing is an opt-in mode and will likely remain that way for a long time. I'm currently working on an agent skill to migrate code to static typing as I believe it is largely mechanical. I do think records and type checking will help many people reason more clearly about their code

On the migration itself: are there any tutorials covering the newly introduced data types? I've seen <record> used a lot throughout the docs, and a dedicated chapter on the practical usage of these data types would be really helpful.

I would check out these pages:

• standard types reference
• Scripts page
• Nextflow training (especially the Side Quests)

Most of these types have always existed in Nextflow, we just didn't document them explicitly, so people had to infer what methods were available from examples / Groovy / Java docs

Is there a way to specify the output copying method? With the older process.publishDir, there were options like copy, symlink, hardlink, and so on. The older docs seemed more thorough here, but I can't find any equivalent information in the new ones.

These options are now under the workflow config scope: https://docs.seqera.io/nextflow/reference/config#workflow

Specifically workflow.output.mode and workflow.output.overwrite. You can set overwrite to deep to re-publish files when the checksum is different