ModuleNotFoundError: No module named 'executor'

[ConYel]

Hello and thank you for this useful tool!
I have made a docker with the installation of fastq-dl
and I tried to run one example but it seems is not working.
the dockerfile can be found here: https://github.com/ConYel/docker_sncRNA_workflow/blob/master/Dockerfile

Do you believe there are some issues with the installation or the way the container is created?

fastq-dl SRX477044 ENA
2020-12-16 14:46:36:root:INFO - Aspera Connect not available, using FTP for ENA downloads
2020-12-16 14:46:37:root:INFO - Query: SRX477044
2020-12-16 14:46:37:root:INFO - Archive: ENA
2020-12-16 14:46:37:root:INFO - Total Runs To Download: 1
2020-12-16 14:46:37:root:INFO - 	Working on run SRR1178104...
Traceback (most recent call last):
  File "/root/miniconda/bin/fastq-dl", line 462, in <module>
    fastqs = ena_download(run, outdir, aspera=aspera,
  File "/root/miniconda/bin/fastq-dl", line 216, in ena_download
    fastq = download_ena_fastq(
  File "/root/miniconda/bin/fastq-dl", line 250, in download_ena_fastq
    execute(f'mkdir -p {outdir}')
  File "/root/miniconda/bin/fastq-dl", line 94, in execute
    from executor import ExternalCommand, ExternalCommandFailed
ModuleNotFoundError: No module named 'executor'

[rpetit3]

Let's see if we can get this figured out! I think first thing to check is to see is if the conda install of Python3 is being used since that's where executor is installed.

Try:

which fastq-dl

which python3

Both of these should be in /root/miniconda/bin/ (based on error message above)

[ConYel]

Thank you for the prompt reply!

root@111:/home# which fastq-dl
root@111:/home# which python3
/usr/bin/python3

I actually didn't get any results on the fastq-dl

ls /root/miniconda/bin/py
pydoc             pydoc3            pydoc3.8          python            python3           python3.8         python3.8-config  python3-config 

[rpetit3]

I'm guessing its a PATH issue. What does $PATH look like?

echo $PATH

[ConYel]

root@111:/home# echo $PATH ~/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

[rpetit3]

Is fastq-dl in /root/miniconda/bin?

[ConYel]

There is:
root@111:/home# ls /root/miniconda/bin/f f2py fasterq-dump fasterq-dump-orig.2.10.8 fastq-dump.2 fax2ps fc-conflist fc-query freetype-config f2py3 fasterq-dump.2 fastqc fastq-dump.2.10.8 fax2tiff fc-list fc-scan futurize f2py3.8 fasterq-dump.2.10.8 fastq-dl fastq-dump-orig fc-cache fc-match fc-validate fastaFromBed fasterq-dump-orig fastq-dump fastq-dump-orig.2.10.8 fc-cat

[rpetit3]

Cool!

Let's give this a try:

/root/miniconda/bin/python3 /root/miniconda/bin/fastq-dl  SRX477044 ENA

I noticed the export to PATH in the dockerfile, but I'm wondering if might be worth adding this instead:

ENV PATH /root/miniconda/bin:$PATH

[ConYel]

So I tried it multiple times but I get always:

root@111:/home# /root/miniconda/bin/python3 /root/miniconda/bin/fastq-dl  SRX477044 ENA

2020-12-16 16:04:07:root:INFO - Aspera Connect not available, using FTP for ENA downloads
2020-12-16 16:04:09:root:INFO - Query: SRX477044
2020-12-16 16:04:09:root:INFO - Archive: ENA
2020-12-16 16:04:09:root:INFO - Total Runs To Download: 1
2020-12-16 16:04:09:root:INFO - 	Working on run SRR1178104...
2020-12-16 16:04:09:root:INFO - 		FTP download attempt 1
2020-12-16 16:04:09:root:ERROR - "wget --quiet -O /home/SRR1178104_1.fastq.gz ftp.sra.ebi.ac.uk/vol1/fastq/SRR117/004/SRR1178104/SRR1178104_1.fastq.gz" return exit code 8
2020-12-16 16:04:09:root:ERROR - Retry execution (1 of 10)
2020-12-16 16:04:19:root:ERROR - "wget --quiet -O /home/SRR1178104_1.fastq.gz ftp.sra.ebi.ac.uk/vol1/fastq/SRR117/004/SRR1178104/SRR1178104_1.fastq.gz" return exit code 8
2020-12-16 16:04:19:root:ERROR - Retry execution (2 of 10)
2020-12-16 16:04:30:root:ERROR - "wget --quiet -O /home/SRR1178104_1.fastq.gz ftp.sra.ebi.ac.uk/vol1/fastq/SRR117/004/SRR1178104/SRR1178104_1.fastq.gz" return exit code 8
2020-12-16 16:04:30:root:ERROR - Retry execution (3 of 10)
2020-12-16 16:04:40:root:ERROR - "wget --quiet -O /home/SRR1178104_1.fastq.gz ftp.sra.ebi.ac.uk/vol1/fastq/SRR117/004/SRR1178104/SRR1178104_1.fastq.gz" return exit code 8
2020-12-16 16:04:40:root:ERROR - Retry execution (4 of 10)
2020-12-16 16:04:50:root:ERROR - "wget --quiet -O /home/SRR1178104_1.fastq.gz ftp.sra.ebi.ac.uk/vol1/fastq/SRR117/004/SRR1178104/SRR1178104_1.fastq.gz" return exit code 8
2020-12-16 16:04:50:root:ERROR - Retry execution (5 of 10)
2020-12-16 16:05:00:root:ERROR - "wget --quiet -O /home/SRR1178104_1.fastq.gz ftp.sra.ebi.ac.uk/vol1/fastq/SRR117/004/SRR1178104/SRR1178104_1.fastq.gz" return exit code 8
2020-12-16 16:05:00:root:ERROR - Retry execution (6 of 10)
2020-12-16 16:05:10:root:ERROR - "wget --quiet -O /home/SRR1178104_1.fastq.gz ftp.sra.ebi.ac.uk/vol1/fastq/SRR117/004/SRR1178104/SRR1178104_1.fastq.gz" return exit code 8
2020-12-16 16:05:10:root:ERROR - Retry execution (7 of 10)
2020-12-16 16:05:20:root:ERROR - "wget --quiet -O /home/SRR1178104_1.fastq.gz ftp.sra.ebi.ac.uk/vol1/fastq/SRR117/004/SRR1178104/SRR1178104_1.fastq.gz" return exit code 8
2020-12-16 16:05:20:root:ERROR - Retry execution (8 of 10)
2020-12-16 16:05:30:root:ERROR - "wget --quiet -O /home/SRR1178104_1.fastq.gz ftp.sra.ebi.ac.uk/vol1/fastq/SRR117/004/SRR1178104/SRR1178104_1.fastq.gz" return exit code 8
2020-12-16 16:05:30:root:ERROR - Retry execution (9 of 10)
2020-12-16 16:05:40:root:ERROR - "wget --quiet -O /home/SRR1178104_1.fastq.gz ftp.sra.ebi.ac.uk/vol1/fastq/SRR117/004/SRR1178104/SRR1178104_1.fastq.gz" return exit code 8
Traceback (most recent call last):
  File "/root/miniconda/bin/fastq-dl", line 462, in <module>
    fastqs = ena_download(run, outdir, aspera=aspera,
  File "/root/miniconda/bin/fastq-dl", line 216, in ena_download
    fastq = download_ena_fastq(
  File "/root/miniconda/bin/fastq-dl", line 255, in download_ena_fastq
    execute(f'wget --quiet -O {fastq} {ftp}', max_attempts=max_attempts)
  File "/root/miniconda/bin/fastq-dl", line 126, in execute
    raise error
  File "/root/miniconda/bin/fastq-dl", line 104, in execute
    command.start()
  File "/root/miniconda/lib/python3.8/site-packages/executor/__init__.py", line 1441, in start
    self.start_once(**kw)
  File "/root/miniconda/lib/python3.8/site-packages/executor/__init__.py", line 1508, in start_once
    self.wait(check=check)
  File "/root/miniconda/lib/python3.8/site-packages/executor/__init__.py", line 1551, in wait
    self.check_errors(check=check)
  File "/root/miniconda/lib/python3.8/site-packages/executor/__init__.py", line 1673, in check_errors
    raise self.error_type(self)
executor.ExternalCommandFailed: External command failed with exit code 8!

Command:
bash -c 'wget --quiet -O /home/SRR1178104_1.fastq.gz ftp.sra.ebi.ac.uk/vol1/fastq/SRR117/004/SRR1178104/SRR1178104_1.fastq.gz'

Regarding the path I have tried it like this in the past and didn't work properly for some cases I could try again and see if it will change something but it seems there's a different issue.

[rpetit3]

That's a different error though, so progress!

Looks like the service is unavailable at the moment:

http://ftp.sra.ebi.ac.uk/vol1/fastq/SRR117/004/SRR1178104/SRR1178104_1.fastq.gz

[rpetit3]

Disregard that, its working fine that!

[rpetit3]

Ok looks like somethign has changed on ENA side.

In the past, this worked:

wget -O ./SRR1178104_1.fastq.gz ftp.sra.ebi.ac.uk/vol1/fastq/SRR117/004/SRR1178104/SRR1178104_1.fastq.gz

Now, ftp:// is needed for it to work:

wget -O ./SRR1178104_1.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR117/004/SRR1178104/SRR1178104_1.fastq.gz

As a temporary solution you could try downloading from SRA

/root/miniconda/bin/python3 /root/miniconda/bin/fastq-dl  SRX477044 SRA

[rpetit3]

I submitted a new release (v1.0.6) with a fix for ENA downloads. It will be available from conda today or tomorrow.

I also created a work flow (https://github.com/rpetit3/fastq-dl/runs/1565163292) to make sure downloads are working.

Thank you very much for your help!

[ConYel]

Well, thank you very much! I will check tomorrow with a new build of the docker plus the ENV to see if it could help.

[ConYel]

I rebuilt it finally, and it is probably working,
I don't know if it is too much to ask but would it be possible to
print something like the percentage of the file while downloading it, to know if it is actually running?
I have used it as before: fastq-dl SRX477044 ENA (now the $PATH seems to be working properly with your suggestion!)
and it was left on the:
2020-12-17 10:16:43:root:INFO - FTP download attempt 1
for ~10 mins so I interrupted it.
I saw that one of the files: SRR1178104_1.fastq.gz, was partially, probably, downloaded.
I check the file but it is 0 bytes so probably it was never downloaded?
Anyway, I tried then the SRA:

root@111:/home# fastq-dl SRX477044 SRA
2020-12-17 10:35:17:root:INFO - Query: SRX477044
2020-12-17 10:35:17:root:INFO - Archive: SRA
2020-12-17 10:35:17:root:INFO - Total Runs To Download: 1
2020-12-17 10:35:17:root:INFO - 	Working on run SRR1178104...
2020-12-17 10:39:39:root:ERROR - "pigz -p 1 -n --fast *.fastq" return exit code 25
Traceback (most recent call last):
  File "/root/miniconda/bin/fastq-dl", line 466, in <module>
    fastqs = sra_download(run["run_accession"], outdir, cpus=args.cpus,
  File "/root/miniconda/bin/fastq-dl", line 175, in sra_download
    execute(f'pigz -p {cpus} -n --fast *.fastq', directory=outdir)
  File "/root/miniconda/bin/fastq-dl", line 126, in execute
    raise error
  File "/root/miniconda/bin/fastq-dl", line 104, in execute
    command.start()
  File "/root/miniconda/lib/python3.8/site-packages/executor/__init__.py", line 1441, in start
    self.start_once(**kw)
  File "/root/miniconda/lib/python3.8/site-packages/executor/__init__.py", line 1508, in start_once
    self.wait(check=check)
  File "/root/miniconda/lib/python3.8/site-packages/executor/__init__.py", line 1551, in wait
    self.check_errors(check=check)
  File "/root/miniconda/lib/python3.8/site-packages/executor/__init__.py", line 1673, in check_errors
    raise self.error_type(self)
executor.ExternalCommandFailed: External command failed with exit code 25!

Command:
bash -c 'pigz -p 1 -n --fast *.fastq'

Standard error:
pigz: abort: write error on SRR1178104_1.fastq.gz (Inappropriate ioctl for device)
root@111/home# ls
download_SRA.sh  my_data  spar_prepare  SRR1178104_1.fastq  SRR1178104_1.fastq.gz  **SRR1178104_2.fastq**  STAR_sam_script.txt

root@111:/home# ls -la
total 2510848
drwxr-xr-x.  1 root root        122 Dec 17 10:39 .
drwxr-xr-x.  1 root root         56 Dec 17 10:14 ..
-rwx------.  1 root root       1111 Dec 17 09:56 download_SRA.sh
drwxr-xr-x.  2 root root         10 Dec 17 10:00 .empty
drwxr-xr-x. 21 1000 1000       4096 Dec 14 12:18 my_data
drwxr-xr-x.  4 root root       4096 Dec 17 10:03 spar_prepare
-rw-r--r--.  1 root root 1285544424 Dec 17 10:39 SRR1178104_1.fastq
-rw-r--r--.  1 root root          0 Dec 17 10:16 SRR1178104_1.fastq.gz
-rw-r--r--.  1 root root 1285544424 Dec 17 10:39 SRR1178104_2.fastq
-rwx------.  1 root root       1307 Dec 17 09:56 STAR_sam_script.txt
root@111:/home# cat SRR1178104_1.fastq |head

The fastq seems ok now.

So the only issue is the pigz not compressing.
I'll try to see if again is a PATH issue as pigz is installed before bioconda or it is just that pigz will not overwrite the SRR1178104_1.fastq.gz file from the interrupted run before.

EDIT
so the above issue was of the previous 0 bytes file and pigz not forcing overwrite over the same name.
I just deleted the fastq.gz + the fastq from the previous run and everything run smoothly.
I don't know if it would be useful to add an option to force rewriting the *.fastq.gz on the pigz call.
Thanks a lot!!!

[rpetit3]

Glad you were able to get it working!

I'll look into adding progress for downloads, and I agree I think and option to overwrite should be added. Looking at the pigz error, it wasn't very helpful. I'll add a "I found this, please use --force to overwrite" type of message before any downloads start.