Quast error about internet connection

[shaman-narayanasamy]

Dear authors,

I am attempting to run quast on some metagenomic assemblies. It seems that quast is able to download some of the references, however it terminates after a certain point, even if I repeat the command.

Logging to /output/Analysis/results/quast/metaquast.log

Contigs:

No references are provided, starting to search for reference genomes in SILVA rRNA database and to download them from NCBI...

2015-09-01 13:28:36
  Using existing BLAST alignments for MGMT.assembly.merged... 

Trying to use previously downloaded references...

2015-09-01 13:28:36
Trying to download found references from NCBI. Totally 46 organisms to try.
  Candidatus_Microthrix_parvicella_RN1               | was downloaded previously (total 1, 45 more to go)
  Dechloromonas_sp._SIUL                             | not found in the NCBI database
  Trachelomonas_volvocinopsis_var._spiralis          | not found in the NCBI database

ERROR! Cannot established internet connection to download reference genomes! Check internet connection or run MetaQUAST with option "--max-ref-number 0".

I am unable to find any other logs files to further troubleshoot this problem... I would be happy to send over my contig fasta file so that you may be able to test it yourself.

Best,
Shaman

[shaman-narayanasamy]

Looks like this was actually a network problem.

[kubu4]

I realize this issue is marked closed, but I've just encountered this error, and thought it would better to post here, rather than open a new issue. Anyway, I definitely have internet access, so I don't have any idea why that error's being thrown. Any suggestions on where to look or how to proceed would be greatly appreciated.

Here're the contents of the metaquast.log:

/home/sam/programs/quast-5.0.2/metaquast.py /home/sam/data/metagenomics/P_generosa/final.contigs.fa --threads 22

Version: 5.0.2

System information:
  OS: Linux-4.4.0-139-generic-x86_64-with-debian-stretch-sid (linux_64)
  Python version: 3.7.2
  CPUs number: 24

Started: 2019-01-10 11:10:58

Logging to /home/sam/analyses/20190110_geo_metagnomics_metaquast/quast_results/results_2019_01_10_11_10_58/metaquast.log

Contigs:
  Pre-processing...
  /home/sam/data/metagenomics/P_generosa/final.contigs.fa ==> final.contigs

No references are provided, starting to search for reference genomes in SILVA 16S rRNA database and to download them from NCBI...

2019-01-10 11:12:29
Running BlastN..
  processing final.contigs
/home/sam/programs/quast-5.0.2/quast_libs/blast/blastn -query /home/sam/data/metagenomics/P_generosa/final.contigs.fa \
-db /home/sam/programs/quast-5.0.2/quast_libs/silva/silva.123.db -outfmt 7 -num_threads \
22 > quast_results/results_2019_01_10_11_10_58/quast_downloaded_references/blast.res_final-contigs \
2>> quast_results/results_2019_01_10_11_10_58/quast_downloaded_references/blast.err
  BLAST results for final.contigs are saved to /home/sam/analyses/20190110_geo_metagnomics_metaquast/quast_results/results_2019_01_10_11_10_58/quast_downloaded_references/blast.res_final-contigs...


2019-01-10 11:56:47
Trying to download found references from NCBI. Totally 50 organisms to try.
  unidentified_proteobacterium                                          | not found in the NCBI database
  unidentified_proteobacterium was not found in NCBI database, trying to download the next best match

ERROR! Cannot established internet connection to download reference genomes! Check internet connection or run MetaQUAST with option "--max-ref-number 0".

In case you have troubles running QUAST, you can write to quast.support@cab.spbu.ru
or report an issue on our GitHub repository https://github.com/ablab/quast/issues
Please provide us with quast.log file from the output directory.

[kdiverson]

I'm having the same issue. I'm running metaquast on an AWS EC2 instance which has internet access. I have been running it successfully until about last week. Could there be a change to the NCBI API?

[kubu4]

Was also wondering if government shutdown was possibly impacting access?

[ChunfangZheng]

I fixed the problem by adding "time.sleep(1)" to the source code quast_lib/search_references_meta.py method "def try_send_request(url)".
NCBI recommends that users post no more than three URL requests per second; the time.sleep(1) in the method try_send_request(url) slows the speed of sending requests.

[kubu4]

I fixed the problem by adding "time.sleep(1)" to the source code quast_lib/search_references_meta.py method "def try_send_request(url)".
NCBI recommends that users post no more than three URL requests per second; the time.sleep(1) in the method try_send_request(url) slows the speed of sending requests.

Thanks for this! This alleviated the issue for me.

[kubu4]

Thanks for this! This alleviated the issue for me.

Never mind. Got too excited, too quickly! Although I didn't get an NCBI connection error, I did get a NameError: name 'time' is not defined. Is there another line I need to add to define time()?

[ChunfangZheng]

import time at the beginning of the file

from __future__ import with_statement
import os
import shlex
import shutil
import re
from collections import defaultdict
import time
from os.path import isdir, isfile, join

[kubu4]

In order to alleviate the NameError: name 'time' is not defined error, I think you also need to import the time module (e.g.):

import time

Insert that line somewhere at the top of the script with the other import commands should fix it.

[holtjma]

I randomly happened to run into this issue at the same time as you guys; the time import and insertion of time.sleep(1) worked for me as well.

[alexeigurevich]

Thanks @ChunfangZheng, @kubu4 and @holtjma -- we added import time and time.sleep(1) solution to the master branch (will be available in the next release).

By the way, if you find a problem related to a closed issue feel free to reopen it!

[microbiomeparty]

Hi, sorry if I'm missing the resolution to this elsewhere but I'm running quest 5.0.2 in metaquast mode and still getting this issue downloading references from NCBI. My network isn't having any issues that I can find, is there another workaround that you might suggest? Thanks!

---

Version: 5.0.2

System information:
OS: Linux-3.10.0-693.21.1.el7.x86_64-x86_64-with-centos-7.4.1708-Core (linux_64)
Python version: 3.6.7
CPUs number: 32

Started: 2019-11-20 08:35:15

Logging to /home/mwoodwo/prj/PREMIX/mg/03.quast_output/metaquast.log
NOTICE: Output directory already exists and looks like a QUAST output dir. Existing results can be reused (e.g. previously generated alignments)!

Contigs:
Pre-processing...
1 ./PM1-S1/contigs.fasta ==> PM1-S1_contigs
2 ./PM1-S2/contigs.fasta ==> PM1-S2_contigs
3 ./PM1-S3/contigs.fasta ==> PM1-S3_contigs
4 ./PM1-C1D01/contigs.fasta ==> PM1-C1D01_contigs
5 ./PM1-C1D02/contigs.fasta ==> PM1-C1D02_contigs
6 ./PM1-C1D15/contigs.fasta ==> PM1-C1D15_contigs
7 ./PM1-C1D36/contigs.fasta ==> PM1-C1D36_contigs
8 ./PM3-S1/contigs.fasta ==> PM3-S1_contigs
9 ./PM3-C0D15/contigs.fasta ==> PM3-C0D15_contigs
10 ./PM3-C0D1/contigs.fasta ==> PM3-C0D1_contigs
11 ./PM3-C0D2/contigs.fasta ==> PM3-C0D2_contigs
12 ./PM3-C0D36/contigs.fasta ==> PM3-C0D36_contigs
13 ./SD-01/contigs.fasta ==> SD-01_contigs

No references are provided, starting to search for reference genomes in SILVA 16S rRNA database and to download them from NCBI...

2019-11-20 08:35:28
Using existing BLAST alignments for PM1-S1_contigs...
Using existing BLAST alignments for PM1-S2_contigs...
Using existing BLAST alignments for PM1-S3_contigs...
Using existing BLAST alignments for PM1-C1D01_contigs...
Using existing BLAST alignments for PM1-C1D02_contigs...
Using existing BLAST alignments for PM1-C1D15_contigs...
Using existing BLAST alignments for PM1-C1D36_contigs...
Using existing BLAST alignments for PM3-S1_contigs...
Using existing BLAST alignments for PM3-C0D15_contigs...
Using existing BLAST alignments for PM3-C0D1_contigs...
Using existing BLAST alignments for PM3-C0D2_contigs...
Using existing BLAST alignments for PM3-C0D36_contigs...
Using existing BLAST alignments for SD-01_contigs...

2019-11-20 08:35:36
Trying to download found references from NCBI. Totally 164 organisms to try.

ERROR! Cannot established internet connection to download reference genomes! Check internet connection or run MetaQUAST with option "--max-ref-number 0".

[alexeigurevich]

Hi,
Unfortunately, we didn't release a new version of Quast for a while, so the current latest release (version 5.0.2) is not yet affected by the fix mentioned here (I mean using time.sleep(1)).
So, could you please try to apply the fix locally to your copy of 5.0.2 and check whether it works for you?
To do so, please download the fixed copy of search_references_meta.py (use Right-Click and "Save as" menu), and put it in <your_quast_installation_dir>/quast_libs/ dir to replace the old version. After that, try to restart Quast.

[microbiomeparty]

thanks so much, this is working! m

…

On Nov 20, 2019, at 10:03 AM, alexeigurevich ***@***.***> wrote: Hi, Unfortunately, we didn't release a new version of Quast for a while, so the current latest release (version 5.0.2) is not yet affected by the fix mentioned here (I mean using time.sleep(1)). So, could you please try to apply the fix locally to your copy of 5.0.2 and check whether it works for you? To do so, please download the fixed copy of search_references_meta.py <https://raw.githubusercontent.com/ablab/quast/32a76ecd7d0b6419840f0d9d4351109dc669f070/quast_libs/search_references_meta.py> (use Right-Click and "Save as" menu), and put it in <your_quast_installation_dir>/quast_libs/ dir to replace the old version. After that, try to restart Quast. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8?email_source=notifications&email_token=AI7M44RI3VFW72BDYL6ZWFTQUVGVVA5CNFSM4BOXBBBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEESIQFA#issuecomment-556042260>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7M44W65DZSJHE5GD5Z4YLQUVGVVANCNFSM4BOXBBBA>.

[Gian77]

I am having the same issue with a newer version of quast when I try to run it over a series of assemblies using a slurm array.

quast 5.2.0 py39pl5321h2add14b_2 bioconda

This is how I try to run it:

conda activate quast
echo "version of `quast --version`" 

metaquast.py \
	--threads $cores \
	--no-icarus \
	$project_dir/outputs/b02_assembly_spades/${sampleID}/scaffolds_hq.fasta \
	-o $project_dir/outputs/b05_evaluateAssembly_metaquast/${sampleID}/	

conda deactivate

And this is the error I got

No references are provided, starting to search for reference genomes in SILVA 16S rRNA database and to download them from NCBI...

2023-04-05 16:48:16
Running BlastN..
  processing scaffolds_hq
  BLAST results for scaffolds_hq are saved to /mnt/research/EvansLab/Benucci/metaGAAP/outputs/b05_evaluateAssembly_metaquast/09_REX_JJ_S5_L003/quast_downloaded_references/blast.res_scaffolds_hq...


2023-04-05 16:49:16
Trying to download found references from NCBI. Totally 50 organisms to try.
WARNING: Empty/No response from NCBI. Could be an Internet connection issue!
  delta_proteobacterium_WY32                      | not found in the NCBI database
  delta_proteobacterium_WY32 was not found in NCBI database, trying to download the next best match
WARNING: Empty/No response from NCBI. Could be an Internet connection issue!
  delta_proteobacterium_LX33                      | not found in the NCBI database
  delta_proteobacterium_LX33 was not found in NCBI database, trying to download the next best match

ERROR! Cannot established internet connection to download reference genomes! Check internet connection or run MetaQUAST with option "--max-ref-number 0" to disable reference search in the NCBI database.

In case you have troubles running QUAST, you can write to quast.support@cab.spbu.ru
or report an issue on our GitHub repository https://github.com/ablab/quast/issues
Please provide us with quast.log file from the output directory.

Thank you,
Gian

[MaryoHg]

Dear @Gian77

How is it going?
Were you able to fix the problem?
I'm facing the same error while using metaquast.py in a server with full access to the internet.

The option time.sleep(1) is already added to quast_libs/search_references_meta.py script, so I'm not sure how to proceed. I'll run the code in a personal computer to double check.

Dear @alexeigurevich, any advice will be appreciated.

Saludos.

[Gian77]

Hello @MaryoHg,

no, I still have that issue. @alexeigurevich any solve?

Thanks,

G.

[Nitin123-4]

I am also facing the same issue with the latest version (5.2.0)

Trying to download found references from NCBI. Totally 35 organisms to try.
human gut metagenome has too fragmented reference genome! It will not be downloaded.
human_gut_metagenome | not found in the NCBI database
human_gut_metagenome was not found in NCBI database, trying to download the next best match
Trichuris_trichiura_(human_whipworm) | not found in the NCBI database
Trichuris_trichiura_(human_whipworm) was not found in NCBI database, trying to download the next best match
Prevotella_copri_DSM_18205 | successfully downloaded (total 1, 34 more to go)
Veillonella_parvula | successfully downloaded (total 2, 33 more to go)
Tyzzerella_sp._Marseille-P3062 | successfully downloaded (total 3, 32 more to go)
Sutterellaceae_bacterium_Marseille-P2968 | successfully downloaded (total 4, 31 more to go)
Streptococcus_thermophilus_ND03 | successfully downloaded (total 5, 30 more to go)
Streptococcus_salivarius_subsp._thermophilus | successfully downloaded (total 6, 29 more to go)
Streptococcus_pneumoniae | successfully downloaded (total 7, 28 more to go)
Streptococcus_phage_2096.2 | not found in the NCBI database
Streptococcus_phage_2096.2 was not found in NCBI database, trying to download the next best match
Streptococcus_pyogenes_phage_6180.4 | not found in the NCBI database
Streptococcus_pyogenes_phage_6180.4 was not found in NCBI database, trying to download the next best match
Streptococcus_phage_10750.4 | not found in the NCBI database
Streptococcus_infantarius | successfully downloaded (total 8, 26 more to go)
Streptococcus_agalactiae_SS1219 | successfully downloaded (total 9, 25 more to go)
Serratia_marcescens | successfully downloaded (total 10, 24 more to go)
Neisseria_zalophi | successfully downloaded (total 11, 23 more to go)
Myroides_odoratimimus_CCUG_12700 | successfully downloaded (total 12, 22 more to go)
WARNING: Empty/No response from NCBI. Could be an Internet connection issue!
Lactobacillus_sp._C33LV5 | not found in the NCBI database
Lactobacillus_sp._C33LV5 was not found in NCBI database, trying to download the next best match
WARNING: Empty/No response from NCBI. Could be an Internet connection issue!
Lactobacillus_ruminis | not found in the NCBI database
Lactobacillus_ruminis was not found in NCBI database, trying to download the next best match

ERROR! Cannot established internet connection to download reference genomes! Check internet connection or run MetaQUAST with option "--max-ref-number 0" to disable reference search in the NCBI database.

[MCeciC]

Hello @MaryoHg,

no, I still have that issue. @alexeigurevich any solve?

Thanks,

G.

HI @Gian77 @MaryoHg could you solve the problem? 'm gettin the same error ERROR! Cannot established internet connection to download reference genomes! Check internet connection

Thanks!

[MaryoHg]

Dear @MCeciC

I ran the metaquast.py command last night and locally. It was a relatively successful run. I get some not found in the NCBI database but also get other such as successfully downloaded (total 3, 6 more to go). The command ended the final report though.

I'll try the same run on the server. I'm thinking about the changes from ftp to https at the NCBI but not sure.

I'll update you.

Cheers,