conda environment from file not working using nextflow

[KochTobi]

Bug report

Expected behavior and actual behavior

Expected behavior

Running the nf-core/rnaseq pipeline and the SciLifeLab/Sarek pipeline with conda creates a conda environment from the respective environment.yml file.

Actual behavior

Nextflow fails to create the environment from file (see output in section below)

Conda is specified in the configs
process.conda = "$baseDir/environment.yml" in nextflow.config for nf-core/rnaseq
and

process {
  process.conda = 'environment.yml'
}

in the conf/conda.config file for Sarek

Steps to reproduce the problem

Fresh install of miniconda3 v4.6.7
execute:

nextflow -log tracedir/.nextflow.log run /anonymous-home/rnaseq --reads /anonymous-home/data/*.fastq -profile cfc,conda --genome GRCh37 --singleEnd -ansi

The conda profile was modified locally to disable docker and singularity as well.

Program output

Exception in thread "Thread-1" groovy.lang.GroovyRuntimeException: exception while reading process stream
        at org.codehaus.groovy.runtime.ProcessGroovyMethods$TextDumper.run(ProcessGroovyMethods.java:496)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Stream closed
        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
        at java.io.InputStreamReader.read(InputStreamReader.java:184)
        at java.io.BufferedReader.fill(BufferedReader.java:161)
        at java.io.BufferedReader.readLine(BufferedReader.java:324)
        at java.io.BufferedReader.readLine(BufferedReader.java:389)
        at org.codehaus.groovy.runtime.ProcessGroovyMethods$TextDumper.run(ProcessGroovyMethods.java:489)
        ... 1 more
ERROR ~ Error executing process > 'get_software_versions'

Caused by: java.lang.IllegalStateException: Failed to create Conda environment
  command: conda env create --prefix /anonym/work/conda/nf-core-rnaseq-1.3dev-b9db674885ce77e423da485b6eae2f18 --file /anonym-home/rnaseq/environment.yml
  status : 120
  message:


        at nextflow.conda.CondaCache.runCommand(CondaCache.groovy:286)
        at nextflow.conda.CondaCache.createLocalCondaEnv0(CondaCache.groovy:259)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:104)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:326)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1235)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1011)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:994)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:97)
        at nextflow.conda.CondaCache$_createLocalCondaEnv_closure1.doCall(CondaCache.groovy:223)
        at nextflow.conda.CondaCache$_createLocalCondaEnv_closure1.call(CondaCache.groovy)
        at nextflow.file.FileMutex.lock(FileMutex.groovy:105)
        at nextflow.conda.CondaCache.createLocalCondaEnv(CondaCache.groovy:223)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:104)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:326)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1235)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1011)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:994)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:97)
        at nextflow.conda.CondaCache$_getLazyImagePath_closure2.doCall(CondaCache.groovy:313)
        at nextflow.conda.CondaCache$_getLazyImagePath_closure2.call(CondaCache.groovy)
        at groovyx.gpars.dataflow.LazyDataflowVariable$1.run(LazyDataflowVariable.java:70)
        ... 3 common frames omitted

Environment

• Nextflow version: [18.10.1 build 5003]
• Java version: [Groovy 2.5.3 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13]
• Operating system: [Linux 3.10.0-514.26.2.el7.x86_64]

Additional context

Running the failed command conda env create --prefix /anonym/work/conda/nf-core-rnaseq-1.3dev-b9db674885ce77e423da485b6eae2f18 --file /anonym-home/rnaseq/environment.yml manually does work

After the environment was created and exists on the system the workflow works fine. Therefore I suspect there to be a problem with nextflow trying to create the environment from a file.

File permissions for the user in the directory are also not the problem as he can create it by hand within the same session.

[pditommaso]

The 120 exit status signals that it was reached the creation timeout. Try increasing it, eg.

conda.createTimeout = '1 h'

[pditommaso]

Is this solved?

[KochTobi]

Yes, extending the createTimeout worked great. Thanks a lot!