Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IndexError: list index out of range #104

Open
nick-youngblut opened this issue Jun 24, 2019 · 7 comments
Open

IndexError: list index out of range #104

nick-youngblut opened this issue Jun 24, 2019 · 7 comments
Assignees
@alexeigurevich

Comments

@nick-youngblut
Copy link

The error 

list index out of range
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 567, in __call__
    return self.func(*args, **kwargs)
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/joblib/parallel.py", line 225, in __call__
    for func, args, kwargs in self.items]
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/joblib/parallel.py", line 225, in <listcomp>
    for func, args, kwargs in self.items]
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/quast-5.0.2-py3.6.egg-info/scripts/metaquast.py", line 100, in _run_quast_per_ref
    is_parallel_run=is_parallel_run)
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/quast-5.0.2-py3.6.egg-info/scripts/metaquast.py", line 72, in _start_quast_main
    return_code = quast.main(args)
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/quast-5.0.2-py3.6.egg-info/scripts/quast.py", line 120, in main
    external_logger=logger)
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/quast_libs/reads_analyzer.py", line 941, in do
    temp_output_dir, output_dir, log_path, err_fpath)
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/quast_libs/reads_analyzer.py", line 376, in run_processing_reads
    correct_chr_names, sam_fpaths, bam_fpaths = run_parallel(align_single_file, parallel_align_args, n_jobs)
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/quast_libs/qutils.py", line 1044, in run_parallel
    results_tuples = [_fn(*args) for args in fn_args]
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/quast_libs/qutils.py", line 1044, in <listcomp>
    results_tuples = [_fn(*args) for args in fn_args]
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/quast_libs/reads_analyzer.py", line 493, in align_single_file
    correct_chr_names = get_correct_names_for_chroms(output_dirpath, fpath, sam_fpath, err_fpath, reads_fpaths, logger, is_reference)
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/quast_libs/ra_utils/misc.py", line 214, in get_correct_names_for_chroms
    chr_len = re.findall(chr_len_pattern, l)[0]
IndexError: list index out of range
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/quast-5.0.2-py3.6.egg-info/scripts/metaquast.py", line 406, in <module>
    return_code = main(sys.argv[1:])
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/quast-5.0.2-py3.6.egg-info/scripts/metaquast.py", line 323, in main
    run_parallel(_run_quast_per_ref, parallel_run_args, qconfig.max_threads, filter_results=True)
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/quast_libs/qutils.py", line 1065, in run_parallel
    results_tuples = Parallel(**parallel_args)(delayed(_fn)(*args) for args in fn_args)
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/joblib/parallel.py", line 934, in __call__
    self.retrieve()
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/site-packages/joblib/parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/ebio/abt3_projects/software/dev/DeepMAsED/.snakemake/conda/5b6511f9/lib/python3.6/multiprocessing/pool.py", line 670, in get
    raise self._value
IndexError: list index out of range
ERROR! exception caught!

In case you have troubles running QUAST, you can write to quast.support@cab.spbu.ru
or report an issue on our GitHub repository https://github.com/ablab/quast/issues
Please provide us with quast.log file from the output directory.

The entire log is attached: metaquast.log

My conda env 

# Name                    Version                   Build  Channel
blast                     2.9.0           pl526h979a64d_2    bioconda
bzip2                     1.0.6             h14c3975_1002    conda-forge
ca-certificates           2019.6.16            hecc5488_0    conda-forge
certifi                   2019.3.9                 py36_0    conda-forge
circos                    0.69.6                        5    bioconda
curl                      7.64.1               hf8cf82a_0    conda-forge
cycler                    0.10.0                     py_1    conda-forge
dbus                      1.13.6               he372182_0    conda-forge
entrez-direct             11.0                    pl526_0    bioconda
expat                     2.2.5             hf484d3e_1002    conda-forge
fontconfig                2.13.1            he4413a7_1000    conda-forge
freetype                  2.10.0               he983fc9_0    conda-forge
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
giflib                    5.1.9                h516909a_0    conda-forge
glib                      2.58.3            h6f030ca_1001    conda-forge
glimmerhmm                3.0.4                h2d50403_2    bioconda
gst-plugins-base          1.14.5               h0935bb2_0    conda-forge
gstreamer                 1.14.5               h36ae1b5_0    conda-forge
icu                       58.2              hf484d3e_1000    conda-forge
joblib                    0.13.2                     py_0    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
kiwisolver                1.1.0            py36hc9558a2_0    conda-forge
krb5                      1.16.3            h05b26f9_1001    conda-forge
libblas                   3.8.0                7_openblas    conda-forge
libcblas                  3.8.0                7_openblas    conda-forge
libcurl                   7.64.1               hda55be3_0    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0
libgd                     2.2.5             h0d07dcb_1005    conda-forge
libgfortran               3.0.0                         1    conda-forge
libiconv                  1.15              h516909a_1005    conda-forge
liblapack                 3.8.0                7_openblas    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0
libtiff                   4.0.10            h57b8799_1003    conda-forge
libuuid                   2.32.1            h14c3975_1000    conda-forge
libwebp                   1.0.2                h576950b_1    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxml2                   2.9.9                h13577e0_0    conda-forge
lz4-c                     1.8.3             he1b5a44_1001    conda-forge
matplotlib                3.1.0                    py36_1    conda-forge
matplotlib-base           3.1.0            py36hfd891ef_1    conda-forge
minimap2                  2.17                 h84994c4_0    bioconda
ncurses                   6.1               hf484d3e_1002    conda-forge
numpy                     1.16.4           py36h95a1406_0    conda-forge
openblas                  0.3.5                ha44fe06_0    conda-forge
openjdk                   11.0.1            h516909a_1016    conda-forge
openssl                   1.1.1b               h14c3975_1    conda-forge
pcre                      8.41              hf484d3e_1003    conda-forge
perl                      5.26.2            h516909a_1006    conda-forge
perl-app-cpanminus        1.7044                  pl526_1    bioconda
perl-archive-tar          2.32                    pl526_0    bioconda
perl-autoloader           5.74                    pl526_2    bioconda
perl-base                 2.23                    pl526_1    bioconda
perl-business-isbn        3.004                   pl526_0    bioconda
perl-business-isbn-data   20140910.003            pl526_0    bioconda
perl-carp                 1.38                    pl526_3    bioconda
perl-clone                0.41            pl526h14c3975_1    bioconda
perl-common-sense         3.74                    pl526_2    bioconda
perl-compress-raw-bzip2   2.086           pl526hf484d3e_0    bioconda
perl-compress-raw-zlib    2.086           pl526h6bb024c_1    bioconda
perl-config-general       2.63                    pl526_0    bioconda
perl-constant             1.33                    pl526_1    bioconda
perl-data-dumper          2.173                   pl526_0    bioconda
perl-digest-hmac          1.03                    pl526_3    bioconda
perl-digest-md5           2.55                    pl526_0    bioconda
perl-digest-perl-md5      1.9                     pl526_1    bioconda
perl-dynaloader           1.25                    pl526_1    bioconda
perl-encode               2.88                    pl526_1    bioconda
perl-encode-locale        1.05                    pl526_6    bioconda
perl-exporter             5.72                    pl526_1    bioconda
perl-exporter-tiny        1.002001                pl526_0    bioconda
perl-extutils-makemaker   7.36                    pl526_1    bioconda
perl-file-listing         6.04                    pl526_1    bioconda
perl-file-path            2.16                    pl526_0    bioconda
perl-file-temp            0.2304                  pl526_2    bioconda
perl-font-ttf             1.06                    pl526_0    bioconda
perl-gd                   2.71            pl526he860b03_0    bioconda
perl-html-parser          3.72            pl526h6bb024c_5    bioconda
perl-html-tagset          3.20                    pl526_3    bioconda
perl-html-tree            5.07                    pl526_1    bioconda
perl-http-cookies         6.04                    pl526_0    bioconda
perl-http-daemon          6.01                    pl526_1    bioconda
perl-http-date            6.02                    pl526_3    bioconda
perl-http-message         6.18                    pl526_0    bioconda
perl-http-negotiate       6.01                    pl526_3    bioconda
perl-io-compress          2.086           pl526hf484d3e_0    bioconda
perl-io-html              1.001                   pl526_2    bioconda
perl-io-socket-ssl        2.066                   pl526_0    bioconda
perl-io-string            1.08                    pl526_3    bioconda
perl-io-zlib              1.10                    pl526_2    bioconda
perl-json                 4.02                    pl526_0    bioconda
perl-json-xs              2.34            pl526h6bb024c_3    bioconda
perl-libwww-perl          6.36                    pl526_1    bioconda
perl-list-moreutils       0.428                   pl526_1    bioconda
perl-list-moreutils-xs    0.428                   pl526_0    bioconda
perl-lwp-mediatypes       6.04                    pl526_0    bioconda
perl-lwp-protocol-https   6.07                    pl526_4    bioconda
perl-math-bezier          0.01                    pl526_1    bioconda
perl-math-round           0.07                    pl526_1    bioconda
perl-math-vecstat         0.08                    pl526_1    bioconda
perl-mime-base64          3.15                    pl526_1    bioconda
perl-module-implementation 0.09                    pl526_2    bioconda
perl-module-runtime       0.016                   pl526_1    bioconda
perl-mozilla-ca           20180117                pl526_1    bioconda
perl-net-http             6.19                    pl526_0    bioconda
perl-net-ssleay           1.88            pl526h90d6eec_0    bioconda
perl-ntlm                 1.09                    pl526_4    bioconda
perl-number-format        1.75                    pl526_3    bioconda
perl-params-validate      1.29            pl526h14c3975_1    bioconda
perl-parent               0.236                   pl526_1    bioconda
perl-pathtools            3.75            pl526h14c3975_1    bioconda
perl-readonly             2.05                    pl526_0    bioconda
perl-regexp-common        2017060201              pl526_0    bioconda
perl-scalar-list-utils    1.50            pl526h14c3975_0    bioconda
perl-set-intspan          1.19                    pl526_1    bioconda
perl-socket               2.027                   pl526_1    bioconda
perl-statistics-basic     1.6611                  pl526_2    bioconda
perl-storable             3.15            pl526h14c3975_0    bioconda
perl-svg                  2.84                    pl526_0    bioconda
perl-test-requiresinternet 0.05                    pl526_0    bioconda
perl-text-format          0.59                    pl526_2    bioconda
perl-time-hires           1.9760          pl526h14c3975_1    bioconda
perl-time-local           1.28                    pl526_1    bioconda
perl-try-tiny             0.30                    pl526_1    bioconda
perl-types-serialiser     1.0                     pl526_2    bioconda
perl-uri                  1.76                    pl526_0    bioconda
perl-www-robotrules       6.02                    pl526_3    bioconda
perl-xml-namespacesupport 1.12                    pl526_0    bioconda
perl-xml-parser           2.44            pl526h4e0c4b3_7    bioconda
perl-xml-sax              1.00                    pl526_0    bioconda
perl-xml-sax-base         1.09                    pl526_0    bioconda
perl-xml-sax-expat        0.51                    pl526_3    bioconda
perl-xml-simple           2.25                    pl526_1    bioconda
perl-xsloader             0.24                    pl526_0    bioconda
pip                       19.1.1                   py36_0    conda-forge
pthread-stubs             0.4               h14c3975_1001    conda-forge
pyparsing                 2.4.0                      py_0    conda-forge
pyqt                      5.9.2            py36hcca6a23_0    conda-forge
python                    3.6.7             h381d211_1004    conda-forge
python-dateutil           2.8.0                      py_0    conda-forge
qt                        5.9.7                h52cfd70_2    conda-forge
quast                     5.0.2           py36pl526ha92aebf_0    bioconda
readline                  7.0               hf8c457e_1001    conda-forge
setuptools                41.0.1                   py36_0    conda-forge
simplejson                3.8.1                    py36_0    bioconda
sip                       4.19.8          py36hf484d3e_1000    conda-forge
six                       1.12.0                py36_1000    conda-forge
sqlite                    3.28.0               h8b20d00_0    conda-forge
tk                        8.6.9             hed695b0_1002    conda-forge
tornado                   6.0.2            py36h516909a_0    conda-forge
wheel                     0.33.4                   py36_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h14c3975_1004    conda-forge
zstd                      1.4.0                h3b9ef0a_0    conda-forge

@nick-youngblut
Copy link
Author

I'm still running into this error for QUAST v5.0.2. I did some troubleshooting, and the problem was the *.bed file that was being parsed by the find_all_sv() function. The .bed file was incomplete due to a disk quota error. The tail of the file:

Gluconacetobacter_sp_SXCC_1_AFCH01000040_1_Gluconacetobacter_sp_SXCC_1_Contig29211_whole_genome_shotgun_sequence	81122	81122	Gluconacetobacter_sp_SXCC_1_AFCH01000040_1_Gluconacetobacter_sp_SXCC_1_Contig29211_whole_genome_shotgun_sequence	80974	80974	QuastDEL
Gluconacetobacter_sp_SXCC_1_AFCH01000040_1_Gluconacetobacter_sp_SXCC_1_Contig29211_whole_genome_shotgun_sequence	81123	81123	Gluconacetobacter_sp_SXCC_1_AFCH01000040_1_Gluconacetobacter_sp_SXCC_1_Contig29211_whole_genome_shotgun_sequence	80981	80981	QuastDEL
Gluconacetobacter_sp_SXCC_1_AFCH01000040_1_Gluconacetobacter_sp_SXCC_1_Contig29211_whole_genome_shotgun_sequence	81130	81130	Gluconacetobacter_sp_SXCC_1_AFCH01000040_1_Gluconacetobacter_sp_SXCC_1_Contig29211_whole_genome_shotgun_sequence	80991	80991	QuastDEL
Gluconacetobacter_sp_SXCC_1_AFCH01000040_1_Gluconacetobacter_sp_SXCC_1_Contig29211_whole_genome_shotgun_sequence	81140	81140	Gluconacetobacter_sp_SXCC_1_AFCH01000040_1_Gluconacetobacter_sp_SXCC_1_Contig29211_whole_genome_shotgun_sequence	80995	80995	QuastDEL
Gluconacetobacter_sp_SXCC_1_AFCH01000040_1_Gluconacetobacter_sp_SXCC_1_Contig29211_whole_genome_shotgun_sequence	81144	81144	Gluconacetobacter_sp_SXCC_1_AFCH01000040_1_Gluconacetobacter_sp_SXCC_1_Contig29211_whole_genome_shotgun_sequence	81019	81019	QuastDEL
Gluconacetobacter_sp_SXCC_1_AFCH01000040_1_Gluconacetobacter_sp_SXCC_1_Contig29211_whole_genome_shotgun_sequence	81168	81

Upon re-running metaquast, the same .bed file was always re-used instead of re-created, so the IndexError kept happening. I don't see any option to force overwrite the .bed file.

It would be helpful to either include an option for force-overwrite of the .bed file, or to include an exception in the find_all_sv() function such as:

except IndexError:
    f.close()
    os.remove(bed_fpath)
    msg = 'This index error may be due to a corrupt .bed file. The bed file has been deleted. Try re-running metaquast'
    raise IndexError(msg)

@alexeigurevich
Copy link
Contributor

Hello Nick @nick-youngblut!
Thank you for figuring out the reason for the error and for your suggestion to solve it.

Just a few things to discuss before we implement the final fix:

  1. I think it doesn't make sense to have force-overwrite options for each separate type of "reusable" files in Quast output. We can add a single common force-overwrite option for overwriting everything in the output dir (which will be technically very similar to just removing all Quast-related content in the specified output directory and starting from scratch). Do you agree?
  2. I agree that it would be helpful to handle the exception in the find_all_sv() function as you suggested. I think we will go even further and try to check the correctness of the existing .bed file before performing any actions on it. If it is broken, we would just recreate it as if does not exist.

@alexeigurevich alexeigurevich self-assigned this Nov 4, 2019
@nick-youngblut
Copy link
Author

  1. I think it doesn't make sense to have force-overwrite options for each separate type of "reusable" files in Quast output. We can add a single common force-overwrite option for overwriting everything in the output dir (which will be technically very similar to just removing all Quast-related content in the specified output directory and starting from scratch). Do you agree?

The user could always manually delete existing output, if it exists. That is what I have now implemented in my pipeline. I think that it would just be helpful to have an error message that tells the user that pre-existing output might be the cause of the error.

  1. I agree that it would be helpful to handle the exception in the find_all_sv() function as you suggested. I think we will go even further and try to check the correctness of the existing .bed file before performing any actions on it. If it is broken, we would just recreate it as if does not exist.

Sounds great! I better error message could also work: something that just says how pre-existing output files might be the cause of the error.

@hlfreund
Copy link

Hi there! I got the "IndexError: list index out of range" error when running metaQUAST with my assembled contigs, scaffolds, and a sorted, indexed BAM file.

I tried running with the same parameters while excluding the BAM file, and I didn't have any errors.

Any idea why this is? I can attach the logs if necessary!

@alexeigurevich
Copy link
Contributor

Hi @hlfreund !
Yes, please provide the log file (metaquast.log) since your error is probably different from the one discussed in this issue initially (that one is related to the corrupted BED file, not input BAM).

@hlfreund
Copy link

hlfreund commented Dec 2, 2020

Hi, thanks for getting back to me! Here is the log files for one of the assembled metagenomes I checked with metaQUAST. Thank you for your help!
metaquast.log

@alexeigurevich
Copy link
Contributor

Hi @hlfreund,
Sorry for the late reply. I think your issue could be related to #121. Could you try our new 5.1.0 release candidate to check whether we fixed it already?

almiheenko added a commit that referenced this issue Aug 17, 2021
@almiheenko
check correctness of existing bed files (issue #104)
ef88149
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexeigurevich alexeigurevich

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@alexeigurevich @nick-youngblut @hlfreund