error executing process

[Kiliankleemann]

Description of the bug

Hi,
I go the following error:

ERROR ~ Error executing process > 'NFCORE_FETCHNGS:SRA:SRA_FASTQ_FTP (SRX14737000_SRR18609980)'

Caused by:
Process NFCORE_FETCHNGS:SRA:SRA_FASTQ_FTP (SRX14737000_SRR18609980) terminated with an error exit status (56)

Command executed:

curl
--retry 5 --continue-at - --max-time 1200
-L ftp.sra.ebi.ac.uk/vol1/fastq/SRR186/080/SRR18609980/SRR18609980_1.fastq.gz
-o SRX14737000_SRR18609980_1.fastq.gz

echo "51357cee238d9e90d306fe6c580bc058 SRX14737000_SRR18609980_1.fastq.gz" > SRX14737000_SRR18609980_1.fastq.gz.md5
md5sum -c SRX14737000_SRR18609980_1.fastq.gz.md5

curl
--retry 5 --continue-at - --max-time 1200
-L ftp.sra.ebi.ac.uk/vol1/fastq/SRR186/080/SRR18609980/SRR18609980_2.fastq.gz
-o SRX14737000_SRR18609980_2.fastq.gz

echo "ab93cc6edc517b8601f0450e202d2614 SRX14737000_SRR18609980_2.fastq.gz" > SRX14737000_SRR18609980_2.fastq.gz.md5
md5sum -c SRX14737000_SRR18609980_2.fastq.gz.md5

cat <<-END_VERSIONS > versions.yml
"NFCORE_FETCHNGS:SRA:SRA_FASTQ_FTP":
curl: $(echo $(curl --version | head -n 1 | sed 's/^curl //; s/ .*$//'))
END_VERSIONS

Command exit status:
56

Command output:
(empty)

Command error:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

curl: (56) Recv failure: Connection reset by peer

The flow initiated successfully and this is the first echo that I saw:
executor > local (459)
[a8/3c6cca] process > NFCORE_FETCHNGS:SRA:SRA_IDS_TO_RUNINFO (SRR18610076) [100%] 114 of 114 ✔
[0b/f7f9ce] process > NFCORE_FETCHNGS:SRA:SRA_RUNINFO_TO_FTP (114) [100%] 114 of 114 ✔
[fd/dc60e8] process > NFCORE_FETCHNGS:SRA:SRA_FASTQ_FTP (SRX14737000_SRR18609979) [ 66%] 228 of 342, failed: 228, retries: 228
[de/ddb1a0] process > NFCORE_FETCHNGS:SRA:FASTQ_DOWNLOAD_PREFETCH_FASTERQDUMP_SRATOOLS:CUSTOM_SRATOOLSNCBISETTINGS (ncbi-settings) [100%] 1 of 1 ✔
[- ] process > NFCORE_FETCHNGS:SRA:FASTQ_DOWNLOAD_PREFETCH_FASTERQDUMP_SRATOOLS:SRATOOLS_PREFETCH -

Would greatly appreciate advice. Kind regards
Kilian

Command used and terminal output

nextflow run nf-core/fetchngs --input SRR_Acc_List.txt -profile docker --outdir fastq

Relevant files

SRR_Acc_List.txt

System information

Version: 23.04.3 build 5875
Created: 11-08-2023 18:37 UTC (20:37 CEST)
System: Linux 6.2.0-32-generic
Runtime: Groovy 3.0.16 on OpenJDK 64-Bit Server VM 17.0.8-internal+0-adhoc..src
Encoding: UTF-8 (UTF-8)

Container is Docker:
Client: Docker Engine - Community
Version: 24.0.6
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.21.0
Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 3
Server Version: 24.0.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
runc version: v1.1.8-0-g82f18fe
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.2.0-32-generic
Operating System: Ubuntu 22.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 62.5GiB
Name: kilian-Alienware-Aurora-R15
ID: 1a6c2e37-dec8-4ca9-9811-ad714e49340f
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

[ellisrichardj]

Hi - I am getting exactly the same problem. I think its linked to the curl set up as I can get the files using wget without a problem

[rjpbonnal]

We have the same issue, curl detects ftp as default protocol because the url starts with ftp.
I you prepend https:// in the curl commands it works as expected.
A possible and transparent solution is to force curl using:

--proto-default https

configuring the ext.args within a config file.

    process {

        withName: SRA_FASTQ_FTP {
            ext.args = '--retry 5 --continue-at - --max-time 1200 --proto-default https'
        }

[drpatelh]

This should have been fixed in #241. We will be using wget instead of curl in the next release. Please feel to re-open if you have any more issues.