Sqlite unable open database

[fmarletaz]

Hi @lucventurini ,

Sorry to bother you again, I have encountered more problems related to sqlite3. I am tempting to try on another system as I am wondering whether the mounted volume could not be the cause.
Thanks!

Process AnnotationParser-1:
Traceback (most recent call last):
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/preparation/annotation_parser.py", line 101, in run
    strand_specific=strand_specific)
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/preparation/annotation_parser.py", line 429, in load_from_gff
    for row in gff_handle:
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/GFF.py", line 367, in __next__
    line = GffLine(line)
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/GFF.py", line 36, in __init__
    GFAnnotation.__init__(self, line, my_line, header=header)
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/gfannotation.py", line 94, in __init__
    self.score = self._fields[5]
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/gfannotation.py", line 295, in score
    score = fast_float(args[0])
SystemError: <built-in function fast_float> returned NULL without setting an error

and in the prepare.log

2020-02-24 16:52:00,826 - prepare.AnnotationParser-2 - annotation_parser.py:140 - DEBUG - run - AnnotationParser-2 - Sending 1 back, exiting.
2020-02-24 16:52:02,134 - prepare - prepare.py:518 - ERROR - prepare - MainProcess - unable to open database file
Traceback (most recent call last):
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/preparation/prepare.py", line 488, in prepare
    max_intron=args.json_conf["prepare"]["max_intron_length"],)
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/preparation/prepare.py", line 393, in load_exon_lines
    _load_exon_lines_multi(args, shelve_names, logger, min_length, strip_cds, threads, max_intron)
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/preparation/prepare.py", line 348, in _load_exon_lines_multi
    check_same_thread=False  # Necessary for SQLite3 to function in multiprocessing
sqlite3.OperationalError: unable to open database file
2020-02-24 16:52:02,138 - prepare - prepare.py:520 - ERROR - prepare - MainProcess - Mikado has encountered an error, exiting

[lucventurini]

Hi @fmarletaz, thank you for reporting.
I think that the problem is not sqlite3 but rather a third party library, fastnumbers. I will completely remove it in the next days, after tests for the last develop commit end.

In the meantime, the best solution is to change any instance of fast_float to float in the code, same for fast_int to int.

[fmarletaz]

Hi @lucventurini

Ok, just did that, it seems to go through the first file, but then:

Process AnnotationParser-1:
Process AnnotationParser-2:
Traceback (most recent call last):
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/gfannotation.py", line 325, in phase
    phase = int(value)
Traceback (most recent call last):
ValueError: invalid literal for int() with base 10: '.'
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/gfannotation.py", line 325, in phase
    phase = int(value)

During handling of the above exception, another exception occurred:

ValueError: invalid literal for int() with base 10: '.'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/preparation/annotation_parser.py", line 101, in run
    strand_specific=strand_specific)
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/preparation/annotation_parser.py", line 112, in run
    strand_specific=strand_specific)
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/preparation/annotation_parser.py", line 429, in load_from_gff
    for row in gff_handle:
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/preparation/annotation_parser.py", line 583, in load_from_gtf
    for row in gff_handle:
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/GFF.py", line 367, in __next__
    line = GffLine(line)
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/GTF.py", line 401, in __next__
    return GtfLine(line)
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/GFF.py", line 36, in __init__
    GFAnnotation.__init__(self, line, my_line, header=header)
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/GTF.py", line 46, in __init__
    super().__init__(line, my_line, header=header)
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/gfannotation.py", line 95, in __init__
    self.phase = self._fields[7]
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/gfannotation.py", line 95, in __init__
    self.phase = self._fields[7]
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/gfannotation.py", line 327, in phase
    raise ValueError("Invalid phase: {0} (type: {1})".format(value, type(value)))
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/gfannotation.py", line 327, in phase
    raise ValueError("Invalid phase: {0} (type: {1})".format(value, type(value)))
ValueError: Invalid phase: . (type: <class 'str'>)
ValueError: Invalid phase: . (type: <class 'str'>)
Mikado prepare has finished correctly. The output ./mikado_prepared.fasta FASTA file can now be used for BLASTX and/or ORF calling before the next step in the pipeline, `mikado serialise`.

[fmarletaz]

Actually, I don't understand as the case where phase is '.' seems explicated accounted for line 332...

[lucventurini]

HI @fmarletaz
Could you please try commit 7e41ca9 (UPDATED COMMIT HASH)? It is present in a new branch, offshoot of master, ie issue-274. This commit should remove the problem you are encountering.
Please let me know and thanks for your patience!

[fmarletaz]

Thanks. Still not happy...

Process AnnotationParser-2:
Traceback (most recent call last):
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/preparation/annotation_parser.py", line 101, in run
    strand_specific=strand_specific)
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/preparation/annotation_parser.py", line 429, in load_from_gff
    for row in gff_handle:
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/GFF.py", line 367, in __next__
    line = GffLine(line)
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/GFF.py", line 36, in __init__
    GFAnnotation.__init__(self, line, my_line, header=header)
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/gfannotation.py", line 96, in __init__
    self.score = self._fields[5]
  File "/home/f/fmarletaz/work/miniconda2/envs/Mikado/lib/python3.7/site-packages/Mikado/parsers/gfannotation.py", line 294, in score
    if score == "." or not isfloat(score):
SystemError: <built-in function isfloat> returned NULL without setting an error

[lucventurini]

Hi @fmarletaz
OK, I hoped I could still keep in some functions from that package :-(
But it seems that it is completed bugged. I should have it completely removed in ten minutes.

[lucventurini]

Hi @fmarletaz
If you could try e1596fb, this commit completely removes fastnumbers from Mikado2.

PS: @swarbred , after your tests are finished for develop, we will need to integrate this branch as well. No need to redo the tests, this branch will effectuate no change in behaviour for Mikado.

[lucventurini]

@fmarletaz

Another thing, before we remove fastnumbers from master: may I ask you to check

• whether the system you are running on is 32bit rather than 64bit
• which version of fastnumbers is present in your BioConda environment. It should be version 3.0.0. If it is not, and you could first update fastnumbers with pip and then retry with the master branch (which is still with fastnumbers) we (as in I and @swarbred ) would be grateful.

[lucventurini]

@fmarletaz

PS: if you could also do the following:

conda activate mikado2
git clone https://github.com/SethMMorton/fastnumbers
cd fastnumbers
pip install -r dev/requirements.txt
pytest

and tell us the results, we would be grateful.

[fmarletaz]

Hi @lucventurini
Thanks for your answers and I managed to get back to all that only today.
My system is definitely 64bits. Fastnumbers version is 3.0.0. I tried to reinstall as you suggested in your last reply. Below are the results of the pytest:

================== test session starts ===============
platform linux -- Python 3.7.6, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
rootdir: /work/RokhsarU/ferdi/installs/mikado, inifile: setup.cfg
plugins: hypothesis-5.1.0
collected 232 items

tests/test_builtins.py ..............................                                  [ 12%]
tests/test_fastnumbers.py ............................................................ [ 85%]
........................                                                               [ 96%]
tests/test_fastnumbers_examples.py .........

FASNUMBERS NUMERICAL LIMITS FOR THIS COMPILER BEFORE PYTHON FALLBACK:
MAXIMUM INTEGER LENTH: 18
MAX NUMBER FLOAT DIGITS: 11
MAXIMUM FLOAT EXPONENT: 99
MINIMUM FLOAT EXPONENT: -98
                                                                                       [100%]

========================= 232 passed in 28.12s =========================

If I try the commit 7e41ca9 after doing that, I still get the same error.

Then, the commit e1596fb seems to work for resolve this problem.

[lucventurini]

Hi @fmarletaz

Thank you for doing the tests and confirming this.
As I am trying to wrap my head around this: would you be able to do a

# This will get a unique list of scores in the input files
cat <MYGTFS/GFFS> | grep -v "#" | cut -f 6 | sort -u

then attach the file here? This should allow us to pinpoint the felon value(s).

[fmarletaz]

Ah when. I did that, I get only numbers but for one line:

ID=TRINITY_DN52047_c0_g1_i1.path1;Name=TRINITY_DN52047_c0_g1_i1;Target=TRINITY_DN52047_c0_g1_i1 1 76;Gap=M76

And I checked where it was, there is actually a weird line in one of my off files (the gmap one):

        14246916        98      -       .       ID=TRINITY_DN52047_c0_g1_i1.path1;Name=TRINITY_DN52047_c0_g1_i1;Target=TRINITY_DN52047_c0_g1_i1 1 76;Gap=M76

I am quite puzzled. But thanks, the problem was somehow my fault. It looks like one of the line has no chromosome name, that's strange

Thanks!

[lucventurini]

No problem! That somehow explains it. Although I do not understand why an error was raised - on my system, fastnumbers.isfloat on that string returns False, without any errors.

I will do the following:

• restore fastnumbers
• wrap the fastnumbers functions to catch these errors before.