Improve performance of annotation db creation, querying #1869
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[NEW] implemented using slots, so it takes less memory
than the dict it replaces. It supports dictionary
style indexing, so old code will work. It also
aliases certain old keys.
[CHANGED] updated return type hints to reflect the new class
[CHANGED] updated tests to reflect these changes
[CHANGED] this frees parser from having to always start at
the top of a file to figure out the format version.
[CHANGED] previously, this was reversed if a feature was on the minus
strand.
[CHANGED] this was a private method on GffAnnotationDb but has been
made a function to facilitate chunked reading of Gff files.
[CHANGED] iter_line_blocks() now supports num_lines=None, which results
in all lines being returned.
[CHANGED] just calls bound sqlitedb's close method
[CHANGED] incomplete records in a GFF database can be updated
…tations
[CHANGED] we achieve a ~75% reduction in RAM for creating a GffAnnotationDb
for the human genome by combining iter_line_blocks(), which uses
iter_splitlines(), merged_gff_records() and
GffAnnotationDb.update_record_spans(). The
load_annotations(lines_per_block=500_000) argument controls how many lines
are read before the insert is done. We track all record name's that have
been inserted and update their existing spans.
[NEW] builds indexes for standard columns, biotype, seqid, start, etc..
Pull Request Test Coverage Report for Build 9106127053Details
💛 - Coveralls |
khiron
reviewed
May 16, 2024
[NEW] thanks to comment in code review by khiron, added
# codacy:ignore[sql-injection] - limited SQL injection exposure
to silence this codacy warning. As this is purely in a test,
it doesn't seem to have much risk.
khiron
reviewed
May 16, 2024
src/cogent3/parse/gff.py
Outdated
| @functools.singledispatch | ||
| def is_gff3(f) -> bool: | ||
| """True if gff-version is 3""" | ||
| raise TypeError(f"unsopported type type {type(f)}") |
There was a problem hiding this comment.
and "type" x 2! Jeez
[CHANGED] seems comment ws incorrect
khiron
approved these changes
May 16, 2024
[CHANGED] this is from the bandit tool, which indicates B608
as the error for hardcoded_sql_expressions
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.