Spurious "File system ftp://... already exists" errors when staging

[jpfeuffer]

Bug report

Expected behavior and actual behavior

Expected: no errors during the staging of files
Actual: in roughly 30% of the cases I get the error with the output below

Steps to reproduce the problem

nextflow run nf-core/proteomicslfq -r dev -profile docker --input https://raw.githubusercontent.com/bigbio/proteomics-metadata-standard/master/annotated-projects/PXD001819/sdrf.tsv --database https://raw.githubusercontent.com/nf-core/test-datasets/proteomicslfq/testdata-aws/uniprot_yeast_reviewed_isoforms_ups1_crap.fasta_td.fasta

Program output

Error executing process > 'raw_file_conversion (11)'

Caused by:
  File system `ftp://ftp.pride.ebi.ac.uk` already exists

Environment

• Nextflow version: any (e.g. 20.07.01)
• Java version: 8
• Operating system: macOS, Linux
• Bash version: zsh, bash

Additional context

I am reading ftp URIs from a csv file and passing them to a subsequent step via
input: path(...)

This is most likely a race condition when creating the file system. I can see the error here:

nextflow/modules/nf-httpfs/src/main/nextflow/file/http/XFileSystemProvider.groovy

Line 78 in 553c311

throw new IllegalStateException("File system `$base` already exists")

[pditommaso]

Could you please provide the .nextflow.log file? tx!

[jpfeuffer]

Here you go. nextflow.log

[pditommaso]

Oh !

java.lang.IllegalStateException: File system `ftp://ftp.pride.ebi.ac.uk` already exists
	at nextflow.file.http.XFileSystemProvider.newFileSystem(XFileSystemProvider.groovy:78)
	at nextflow.file.http.XFileSystemProvider.getFileSystem(XFileSystemProvider.groovy:128)
	at nextflow.file.http.XFileSystemProvider.getPath(XFileSystemProvider.groovy:138)
	at java.nio.file.Paths.get(Paths.java:143)
	at nextflow.file.FileHelper.asPath(FileHelper.groovy:278)
	at nextflow.file.FileHelper.asPath(FileHelper.groovy:261)
	at nextflow.file.FileHelper$asPath$0.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
	at nextflow.processor.TaskProcessor.normalizeToPath(TaskProcessor.groovy:1655)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:168)
	at nextflow.processor.TaskProcessor.normalizeInputToFiles(TaskProcessor.groovy:1670)

[jpfeuffer]

Right! After some retrying it usually assembles all the tasks correctly. Sometimes it does not, though.

[pditommaso]

I guess it's race condition while loading that package.

[pditommaso]

Fixed f65296b

[jpfeuffer]

That was quick. Thanks a lot! Looking forward to v20.10 then.