Preserve on-disk order in SeqIO & SearchIO indexes

[peterjc]

Since the standard dict object provided in Python 3.6 onwards preserves order, the in-memory dictionary-like indexing in SeqIO and SearchIO will now preserve record order there.

This pull request generalises this to apply to on older versions of Python, and for the SQLite based on-disk dictionary-like indexes.

This means looping over the index will load the records in the on-disk order, which will be much faster (previously it would be effectively at random, based on the key hash sorting).

Essentially this works by explicitly using OrderedDict on Python 3.5 or older for the in-memory indexes, and adding explicit sorting to the SQL command for the on-disk SQLite indexes.

Potentially the SQL schema could add an index to the offset column to speed this up further?

--

I hereby agree to dual licence this and any previous contributions under both the Biopython License Agreement AND the BSD 3-Clause License.

[codecov]

Codecov Report

Merging #1619 into master will increase coverage by <.01%.
The diff coverage is 93.75%.

@@            Coverage Diff             @@
##           master    #1619      +/-   ##
==========================================
+ Coverage   85.47%   85.47%   +<.01%     
==========================================
  Files         301      301              
  Lines       49301    49313      +12     
==========================================
+ Hits        42141    42152      +11     
- Misses       7160     7161       +1

Impacted Files	Coverage Δ
Bio/SeqIO/__init__.py	84.44% <100%> (+0.35%)	⬆️
Bio/SearchIO/__init__.py	89.74% <100%> (+0.55%)	⬆️
Bio/File.py	87.01% <83.33%> (-0.13%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0364518...386a2d8. Read the comment docs.

[peterjc]

@bow do you have a favourite multi-record SearchIO file I should use for a unit test on the order?

[bow]

This looks good @peterjc. My only comment is about the mention of Python 3.6. As far as I know, it is CPython that has insertion-ordered dicts in 3.6, and it won't be until Python 3.7 that this becomes a language feature (as opposed to an implementation feature).

Is the use of CPython something we should also check against here?

As for adding an offset column index, I think it would be useful indeed. Have you played around to see how much speedup would be gained?

[bow]

@peterjc, the xml_2226_tblastn_001.xml is one you can use, but IIRC anything annotated with 'multiple queries' in Tests/Blast/README.txt or Tests/Hmmer/README.txt should be usable here.

[peterjc]

Thanks.

I've not run any detailed timings yet with an extra index, but that seems prudent.

As to the Python 3.6 case, given we only care about CPython and PyPy, these should all be ordered. However, yes, we can't promise this on Jython or IronPython. Do you have a better wording suggestion and/or check in the if-statement for this?

Finally, with this work in place, it ought to be possible to expand the dictionary-like interface to also accept integer indexes (0, 1, 2, ..., n-1 for a file with n records). I can see this being useful, although have not thought about exactly what this mean for implementation on the SQLite side.

[bow]

For the wording, how about:

Since Python 3.7, the default dict class maintains key order, meaning
this dictionary will reflect the order of records given to it. For CPython,
this was already implemented in 3.6.

As of Biopython 1.72, we explicitly use OrderedDict for CPython older than
3.6 (and for other Python older than 3.7) so that you can always assume the
record order is preserved.

And the if block check can include the platform.get_implementation() check:

# Ordered dict is a language feature from Python 3.7 onwards.
# CPython 3.6 also already has ordered dicts.
if sys.version_info >= (3, 7) or (sys.version_info == (3, 6) and platform.python_implementation() == "CPython"):
    _dict = dict
else:
    from collections import OrderedDict as _dict

[bow]

I just realized, perhaps this:

Since Python 3.7, the default dict class maintains key order, meaning
this dictionary will reflect the order of records given to it. For CPython,
this was already implemented in 3.6.

is better phrased as this (since maintains insertion order sounds less ambiguous than maintains key order):

Since Python 3.7, the default dict class maintains insertion order, meaning
this dictionary will reflect the order of records given to it. For CPython,
this was already implemented in 3.6.

[peterjc]

I forgot about this - I'll rebase it now to deal with the minor merge conflicts, and say Biopython 1.73 not 1.72 as we missed the last release.