Upgrade to Groovy 4

[pditommaso]

This PR upgrade the nextflow runtime to Groovy 4

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	d73a5f5
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/65c15c2c7f2b070008751f7a

[pditommaso]

Some tests fails due this issue https://issues.apache.org/jira/browse/GROOVY-11211

[marcodelapierre]

Blocks these PRs:

seqeralabs/libseqera#7

seqeralabs/wave#318 (comment)

[bentsherman]

Just ran nf-core/rnaseq small test with groovy 3 and groovy 4 from the CI instance. Both runs took about 30 min. Commencing full test

[bentsherman]

nf-core/rnaseq full test completed successfully with comparable runtime. Is there anything else you want to test?

[pditommaso]

Let's try also nf-core/fetchngs and Sarek

[bentsherman]

sarek full test was successful ✅

Note that for all three pipelines I had to use older versions that didn't use nf-validation, This plugin (and likely others) will need to be updated, with the current nf-validation I am getting this error:

groovy.lang.MissingMethodException: No signature of method: nextflow.processor.TaskHandler.getTraceRecord() is applicable for argument types: () values: []
Possible solutions: getTraceRecord(), safeTraceRecord()
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:72)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:148)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:166)
        at nextflow.cloud.aws.batch.AwsBatchTaskHandler.getTraceRecord(AwsBatchTaskHandler.groovy:836)
        at nextflow.Session.notifyTaskPending(Session.groovy:980)
        at nextflow.processor.TaskPollingMonitor.schedule(TaskPollingMonitor.groovy:229)
        at nextflow.executor.Executor.submit(Executor.groovy:98)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.processor.TaskProcessor.submitTask(TaskProcessor.groovy:2250)
        at nextflow.processor.TaskProcessor.checkCachedOrLaunchTask(TaskProcessor.groovy:823)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.processor.TaskProcessor.invokeTask(TaskProcessor.groovy:639)
        at nextflow.processor.InvokeTaskAdapter.call(InvokeTaskAdapter.groovy:52)
        at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120)
        at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.access$001(ForkingDataflowOperatorActor.java:35)
        at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(ForkingDataflowOperatorActor.java:58)
        at java.base/java.util.concurrent.ThreadPerTaskExecutor$TaskRunner.run(ThreadPerTaskExecutor.java:314)
        at java.base/java.lang.Thread.run(Thread.java:1583)

fetchngs could not be tested because the test profiles for older versions no longer seem to work

[bentsherman]

Investigating further, it looks like this exception happens whenever I install the groovy4 build using make install, regardless of the pipeline. When I run with launch.sh this error doesn't happen. So it must be an error with the PR itself. The stack trace is coming from the awsbatch executor, but I'm not sure why this error would happen, since the nf-amazon plugin is built alongside nextflow, so it should be using the Nextflow API built with groovy4.

[bentsherman]

Actually I think the above error is happening because when I build Nextflow with make install it is using an older version of nf-amazon, whereas the ./launch.sh approach uses the groovy4 build. So I think we are good to go with successful runs of rnaseq and sarek full tests.

Because external plugins can't be used with dev mode or the -all build, we won't be able to test those until we release a groovy4 version of Nextflow, and then those plugins will also need to update. Fortunately, the nf-core pipelines aren't pinning plugin versions yet so they won't need to be updated as long as nf-validation is updated.

[pditommaso]

We are ready for this!

[bentsherman]

Here we go 😅