Improve variant loading performance, expand gnomAD annotations, and fix variant filtering and parsing

[0.2]

Added

• Loading MT variants from a separate file (#180)
• Parse additional gnomAD fields from the VCF INFO field (#200)

Changed

• Keep the highest-scoring variant when duplicate IDs occur within loading batch (#188)
• Do not parse conservation data for STRs (#191)
• Store null for spliceai_prediction when no prediction data is available (#198)

Fixed

• Fix incorrect rank_score conversion that caused all variants to be loaded (#177)
• Introduce an early variant inclusion check to avoid unnecessary processing (#177)
• Refactor update_compounds to improve variant lookup and reduce MongoDB queries (#177)
• Fix pathogenic variant filtering by matching VEP ClinVar significance fields case-insensitively (#182)
• Fix MEI variant filtering to load all mobile element insertions regardless of rank score (#184)
• Parsing of STRs genotypes (#190)
• Format of clnsig accession Int64 -> Int32 (#194)
• Parsing of spliceai_score and spliceai_position in gene documents (#195)
• Fix copy number parsing by reading the CN FORMAT field as an integer instead of a float (#199)

Performance improvements and release workflow

[0.1.4]

Added

• Created a RELEASE.md document, describing the procedures to creare a new software release (#)

Changed

• Optimize compound variant lookup to avoid cloning full documents (#168)
• Improve variant loading performance by using unordered bulk inserts and skipping duplicate variants without individual retries (#170)
• Speed up VCF record counting for progress reporting (#171)
• Optimize parsing of VEP transcripts by avoiding unnecessary string allocations when parsing CSQ fields (#172)
• Replace per-transcript HashMap allocations with VepFields to improve VEP annotation parsing performance (#173)

Fixed

• Remove leftover debug messages and fix printed sample names and panel IDs (#174)
• Order of actions to create a new release in RELEASE.md file (#175)

Create a binary file for Centos 7

[0.1.3]

Fixed

• Add a Linux release binary compatible with glibc 2.17 (#161, #162, #163, #164, #165)

Fetch cytobands from DB, parse new gnomAD MAX fields

[0.1.2]

Fixed

• Parse GNOMADAF_grpmax and MAX_AF from VCF info field (#158)

Changed

• Fetch cytobands directly from the scout database (#157)

Patch assets v0.1.1

[0.1.1]

Fixed

• Fix release workflow and versioned binaries (#148)

Initial release 0.1.0

[0.1.0]

Added

• Use the rust-htslib for reading VCF files (#1)
• First Variant model and parsing modules (#2)
• Parse alleles and missing docstrings (#3)
• Variant type enum (clinical or research) key/values passed to main function (#5)
• Custom issues and pull requests templates (#9)
• Pass case _id from CLI (#10)
• Parse variant.FILTER field (#11)
• Parse variant.QUAL field (#12)
• Parse compounds field (#14)
• Parse the 4 variant IDs: simple_id, variant_id, display_name, document_id (#15)
• Parse rank_score and norm_rank_scorefrom VCF (#16)
• Infer complete coordinate data (#17)
• Parse cytobands and collect cytoband start and end for each variant (#18)
• Parse genmod genetic models (#19)
• Parsing specific variant fields: str_swegen_mean and str_swegen_std (STRs) and somatic_score (cancer and cancer_sv variants) (#20)
• Parse a number of STR-related fields (#21)
• Parse SCOUT_CUSTOM INFO field (#22)
• Parse strs source fields (SourceDisplay, Source, SourceId) (#23)
• Parse mei info: name and polarity (#24)
• Parse genotype info (#25)
• Parse eventual dbSNP IDs (#27)
• Parse INFO.MATEID as mate_id, when available (#28)
• Parse AZLENGTH and AZQUAL values, when available (#31)
• Parse fusion variants INFO field (#32)
• Set fusion genes from fusion variant INFO (#33)
• Parse MitoMap-associated diseases (#34)
• Parse HmtVar annotations (#35)
• Parsing Genes and transcript annotations from VEP annotations - basic parsing (#36)
• Create list of genes from parsed VEP transcripts (#40)
• Create list of variant HGNC ids from variant genes (#41)
• Parse clnsig from VEP transcripts (#42)
• Parse oncogenicity clnsig from ONC, ONCREVSTAT, ONCDN, CLNVID INFO fields (#43)
• Parse external databases' frequencies from INFO field and parsed VEP transcripts (#44)
• Parse LoqusDB frequencies and metadata (#45)
• Added GitHub Action for linting, tests and changelog enforcement (#47)
• Parse severity predictions keys/values: CADD, Spidex and Revel scores (#53)
• Parse conservation: GERP, PhastCons and PhyloP (#54)
• Parse callers (#55)
• Parse RankResult INFO field (#56)
• Parse SV-specific frequencies key/values (#57)
• Parse MEI-specific frequencies key/values (#58)
• Parse cancer-specific MSK_MVL as mvl_tag (#59)
• Parse Mivmir and Gicam scores for SNVs (#60)
• Introduce a loading orchestrator which is responsible to load all types of variants for a case (#62)
• Collect gene_to_panels from database, based on case panels found on case config file (#65)
• Build an HGNC ID → gene document mapping reflecting genes in genome build provided on case loading config file (#67)
• Tests for loader.hgncid_to_gene and loader.gene_to_panels (#69)
• Add gene panel info to parsed variants (#70)
• Add gene symbols info to parsed variants (#71)
• Complete gene and transcript variant info using gene DB collection (#77)
• Retain panther_domain when parsing variant transcripts (#78)
• Check that provided institute exists in database prior to the parsing and loading of variants (#81)
• Added an optional --config argument to the CLI (#84)
• Load variants in database in batches of 10000 (#82, #93, #95)
• Save institute key on loaded variants documents (#87)
• Update variants' rank after loading them (#94)
• Show progress when loading variant categories (#97)
• Variant loading filter based on rank score and category (#98)
• Parse custom images for STR variants (#102)
• Support loading VCFs by category and research status (#111)
• Tests for clinical and research VCF category selection (#113)
• Improved documentation on README file (#115)
• Clean up loaded variants on failure (#122)
• A Codecov push step on the automatic tests GitHub action (#123)
• Codecov, license and test status badges on README page (#124, #132)
• A Release GitHub workflow, which build and publish standalone binaries for Linux x86_64, macOS x86_64, and macOS ARM64 (#144)
• A CLI version flag (#145)

Changed

• Structure of parsed entities, to reflect documents saved in the database (#20)
• Improved and simplified code (#22)
• Refactored VEP annotations parsing module (#38)
• Refactor and simplify main function to accept and parse case config file (#64)
• Switch to async Loader (#68)
• Set the same value for rank_score and fusion_score for fusion variants (#74)
• Print exact count of inserted variants by category (#93)
• Preserve genomic regions when batching VCF variants (#95)
• Update compound data for batched variants (#96)
• Always load variants which are pathogenic in ClinVar (#99)
• Always load managed variants (#100)
• Always load any variant found to be causative in other cases (#101)
• Split linter and tests GitHub actions (#123)
• Assign rank scores regardless of case ID, with test (#127)
• Decouple genetic model parsing from case ID (#130)
• Decouple compound parsing from case ID (#131)

Fixed

• Normalize chromosome names by stripping the chr prefix (#4)
• Parsing of end chromosome (#29)
• Parsing of variant sub-category (non SNVs) and end chromosome (#29)
• Parsing of multi-transcript genes from VEP CSQ field (#37)
• Fixed bug when parsing fusion variants (#74)
• Renamed score compound key to combined_score (#80)
• Clippy warning about a function with too many arguments (#86)
• Gene parsing for STRs (#105)
• Loading of fusion variants (#108)
• Parsing of fusion variants: FOUND_DB, ORIENTATION, genes, hgnc_ids, hgnc_symbols, genotypes, callers (#110)
• Reintroduced STRs and MEIs catetegories of parsed VCF, accidentally removed in a previous commit (#125)
• Variant position, end and length are now parsed and stored as i32 (#137)
• HGNC IDs in transcripts are stored as i32 and not i64 (#138)
• Variant batch sizing issue causing problem in updating compounds (#139)
• Gene exons should be set to empty string when None (#140)
• Remove compounds key from variant document if it's an empty array (#141)
• Fix parsing of 1000G frequencies (left and right) (#142)