Improve foreign files download

[pditommaso]

Currently foreign input files (eg. s3://foo/bar) are downloaded by the driver application even when using the distributed Ignite executor. Moreover an extra copy is needed when using a scratch directory.

This needs to be improved so that foreign files are download by the remote execution nodes and files are created in the target destination folder without any intermediate copy.

[ewels]

And to confirm, as discussed on Gitter - would be great if such files are only downloaded a single time if used in multiple places by a workflow (eg. the same foreign input file used as the input to a process that is run for many other files).

[ihaque-freenome]

would be great if such files are only downloaded a single time if used in multiple places by a workflow

One caveat: this may be impractical in the case of distributed execution. I'm running Nextflow in a shared SLURM environment with limited networked storage but dedicated fast scratch space per node. I definitely would not want large remote files to be staged into the shared mount; rather, they should go into scratch. It would certainly be ideal to schedule jobs dependent on the same file onto the same node (into the same scratch dir?) so they don't need to transfer the file twice, but I'd rather have two transfers from object store to scratch than have the shared filer become a bottleneck.

[pditommaso]

Is that cloud SLURM deployment? Yes, ideally it should be able to cover this use case as well.

[ihaque-freenome]

@pditommaso yes - it is a SLURM cluster implemented on Google Compute Engine. Each node has local SSD mounted at /scratch and a shared mount at /home.

[emi80]

Hello all,
I am awakening this thread since I am now facing the same problem.

Looking at related issues I found #686 (comment) and #686 (comment) by @ewels. I totally agree and I think we would cover most of the execution cases this way.