Using storeDir in conjunction with s3 path no longer works

[brandoncazander]

Bug report

Expected behavior and actual behavior

We use the storeDir directive together with an AWS s3 path to store the results of a time-consuming step. This has historically worked very well for us. At some point, this stopped working for us, and we don't quite understand why.

When we use an s3 path for our storeDir directive now (pointing to a new bucket that doesn't have the cached output from previous iterations), the process succeeds but nexflow fails with 'Missing output file(s)'

Steps to reproduce the problem

Here's a simple workflow that reproduces the issue for me:

#!/usb/bin/env nextflow
nextflow.enable.dsl=2

filename = "foo.txt"

process foo {
    output:
        path "${filename}"
    storeDir params.storeDir
    """
    echo "Hello World" > "${filename}"
    ls -alh "${filename}"
    """
}

workflow {
    foo()
}

If you run this with nextflow run main.nf --storeDir /tmp/foo it will succeed and the cached file will be present in that directory and be picked up on subsequent runs.

Running it with nextflow run main.nf --storeDir s3://my-bucket/foo fails with:

N E X T F L O W  ~  version 21.10.6
Launching `main.nf` [amazing_archimedes] - revision: 27817df6cb
executor >  local (1)
[e8/2e589d] process > foo [100%] 1 of 1, failed: 1 ✘
Error executing process > 'foo'

Caused by:
  Missing output file(s) `foo.txt` expected by process `foo`

Command executed:

  echo "Hello World" > "foo.txt"
  ls -alh "foo.txt"

Command exit status:
  0

Command output:
  -rw-r--r-- 1 brandon brandon 12 Apr 12 09:18 foo.txt

Work dir:
  /home/brandon/ch/nextflow-file-caching-example/work/e8/2e589d8cfb15e177f3ec3afc3b8d35

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

Program output

The relevant part of the .nextflow.log file shows the problem:

Apr-12 09:18:33.415 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Apr-12 09:18:33.419 [Task submitter] INFO  nextflow.Session - [e8/2e589d] Submitted process > foo
Apr-12 09:18:33.768 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: foo; status: COMPLETED; exit: 0; error: -; workDir: /home/brandon/ch/nextflow-file-caching-example/work/e8/2e589d8cfb15e177f3ec3afc3b8d35]
Apr-12 09:18:33.865 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Process `foo` is unable to find [S3Path]: `/my-bucket/foo/foo.txt` (pattern: `foo.txt`)
Apr-12 09:18:33.881 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'foo'

Environment

• Nextflow version: 21.10.6.5660
• Java version: openjdk 11.0.14.1 2022-02-08
• Operating system: Fedora 35

Additional context

If we switch to use publishDir instead of storeDir with the same bucket, I have no issues, which shows that it's not a local permissions issue. However, publishDir doesn't let us use the cached results.

[brandoncazander]

I found that version 20.10.0 works:

> NXF_VER=20.10.0 nextflow run main.nf --storeDir s3://my-bucket/foo
N E X T F L O W  ~  version 20.10.0
Launching `main.nf` [pedantic_feynman] - revision: 27817df6cb
executor >  local (1)
[9d/900580] process > foo [100%] 1 of 1 ✔

[syousry6]

Hello @brandoncazander
We are getting the same error in our pipeline

We are using 22.04.0 workflow version.
In .workflow.log we are getting exactly the same error you gotProcessalignment is unable to find [S3Path]:
Any suggestions?

[smplyayush]

@bentsherman Changing` '$name' to "$name" i.e. adding double quotes to the parameter in .command.run fixes the issue. Need some one from Nextflow to add an urgent patch to fix this. This is blocking our active development since a month now.

nxf_unstage() { true cp .command.out /mnt/efs/ngs_scratch/workdir/210812_NB501331_0892_AH57FVBGXK/updated-path4/14/96e35fb77d096c4ca2d3291e49b398/.command.out || true cp .command.err /mnt/efs/ngs_scratch/workdir/210812_NB501331_0892_AH57FVBGXK/updated-path4/14/96e35fb77d096c4ca2d3291e49b398/.command.err || true cp .command.trace /mnt/efs/ngs_scratch/workdir/210812_NB501331_0892_AH57FVBGXK/updated-path4/14/96e35fb77d096c4ca2d3291e49b398/.command.trace || true [[ ${nxf_main_ret:=0} != 0 ]] && return IFS=$'\n' for name in $(eval "ls -1d L1.Aligned.sortedByCoord.out.bam L1.Chimeric.out.sam L1.Chimeric.out.junction L1.Log.final.out" | sort | uniq); do nxf_s3_upload '**$name**' s3://k8s-temporal-efs/results/210812_NB501331_0892_AH57FVBGXK/A01-00159459-VL45582-A-d/align/L1 || true done unset IFS }