Pipeline fails for large studies

[alexblaessle]

Description of the bug

Trying to download a SRP project with 25000 crashes the pipeline at SRA_MERGE_SAMPLESHEET

The reason for this is that the list of files handed over is too long for bash commands. Probably chunking or handling it by reading in a list of files in a python script is probably a good way to circumvent this.

Command used and terminal output

nextflow run /path/to/fetchngs/ -profile singularity --input ids.csv --dbgap_key prj_34697.ngc --outdir out/ --skip_fastq_download -resume

ERROR ~ Error executing process > 'NFCORE_FETCHNGS:SRA:SRA_MERGE_SAMPLESHEET'

Caused by:
  Process `NFCORE_FETCHNGS:SRA:SRA_MERGE_SAMPLESHEET` terminated with an error exit status (139)

Command executed:

  cp ./samplesheets/* ~/tmp/
  cp ./mappings/* ~/tmp/


  head -n 1 `ls ./samplesheets/* | head -n 1` > samplesheet.csv
  for fileid in `ls ./samplesheets/*`; do
      awk 'NR>1' $fileid >> samplesheet.csv
  done

  head -n 1 `ls ./mappings/* | head -n 1` > id_mappings.csv
  for fileid in `ls ./mappings/*`; do
      awk 'NR>1' $fileid >> id_mappings.csv
  done

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_FETCHNGS:SRA:SRA_MERGE_SAMPLESHEET":
      sed: $(echo $(sed --version 2>&1) | sed 's/^.*GNU sed) //; s/ .*$//')
  END_VERSIONS

Command exit status:
  139

Command output:
  (empty)

Work dir:

Relevant files

No response

System information

NF version: 23.04.2
Hardware: HPC
Executor: local
Container: Singularity
OS: CentOS
Version: 1.11

[drpatelh]

Fixed in #238 #243