-
Notifications
You must be signed in to change notification settings - Fork 621
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using storeDir
in conjunction with s3 path no longer works
#2790
Comments
I found that version 20.10.0 works: > NXF_VER=20.10.0 nextflow run main.nf --storeDir s3://my-bucket/foo
N E X T F L O W ~ version 20.10.0
Launching `main.nf` [pedantic_feynman] - revision: 27817df6cb
executor > local (1)
[9d/900580] process > foo [100%] 1 of 1 ✔ |
Hello @brandoncazander |
@bentsherman Changing` '$name' to "$name" i.e. adding double quotes to the parameter in .command.run fixes the issue. Need some one from Nextflow to add an urgent patch to fix this. This is blocking our active development since a month now.
|
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Bug report
Expected behavior and actual behavior
We use the
storeDir
directive together with an AWS s3 path to store the results of a time-consuming step. This has historically worked very well for us. At some point, this stopped working for us, and we don't quite understand why.When we use an s3 path for our storeDir directive now (pointing to a new bucket that doesn't have the cached output from previous iterations), the process succeeds but nexflow fails with 'Missing output file(s)'
Steps to reproduce the problem
Here's a simple workflow that reproduces the issue for me:
If you run this with
nextflow run main.nf --storeDir /tmp/foo
it will succeed and the cached file will be present in that directory and be picked up on subsequent runs.Running it with
nextflow run main.nf --storeDir s3://my-bucket/foo
fails with:Program output
The relevant part of the .nextflow.log file shows the problem:
Environment
Additional context
If we switch to use
publishDir
instead ofstoreDir
with the same bucket, I have no issues, which shows that it's not a local permissions issue. However,publishDir
doesn't let us use the cached results.The text was updated successfully, but these errors were encountered: