com.upplication.s3fs.S3Path error when running with ignite

[PhilPalmer]

Hey,
I've tried to run the test profile without changing any of the parameters. The only thing I did change was the working directory to an aws S3 path. The command I ran was:

nextflow run nf-core/rnaseq -w s3://workflow-workdir/rnaseq2 -profile test -with-docker -process.executor ignite

This gave the following error com.upplication.s3fs.S3Path as shown below:

Nextflow.log file can be found here: nextflow.log

However I have noticed that i do not get the error with an older version of the repo: 2cf2f0f779d39463b09d874449cbb9fdb9f58d2f

So I think some change between that version of the repo and present has caused an error when running rnaseq from an S3 working directory

Any help to fix this error would be much appreciated, thanks in advance

[ewels]

Hi @PhilPalmer,

Thanks for the detailed bug report. I guess this is due to this change - we no longer specify executor: 'local' for the process in question. This was removed because such a statement for other processes was causing problems on Kubernetes setups. However, this process is entirely in groovy and uses script, so it should be fine to revert the change I think. I'll put in a PR to revert the change now.

Also note that you're a teeny bit out of date with nextflow, but I doubt that this will have any effect.

Phil

[ewels]

Also, another thing to try is to use three slashes with the s3 URL - no idea why, but I came across that suggestion from @pditommaso when delving into the nextflow gitter history with google: https://gitter.im/nextflow-io/nextflow?at=580fca4bba810c1d5fd00d6d

[pditommaso]

This sounds like a NF bug, however my suggestion is to force the execution of workflow_summary_mqc process locally adding the executor 'local' directory in the main script, because there's no advantage to have it running remotely.

@ewels The problem with K8s you are referring is related to process like get_software_versions.

[ewels]

Yes, you're exactly right @pditommaso - I was a little over zealous in removing the local statements, but figured that it didn't really hurt (until now!). I've added it back in PR #78.

[pditommaso]

For the records nextflow-io/nextflow#843

[ewels]

@PhilPalmer - the pull request is now merged. If you could give it a try that would be great! It should work fine if you use -r dev to specify the development branch when running nextflow.

[PhilPalmer]

@ewels it worked with -r dev. Thank you both for responding so quickly and all your help

[ewels]

Great stuff, thanks for reporting and testing the fix! This will go out as part of the next stable pipeline release when it's ready.