Feature configparse redux (#2869)

* WiP: Pydantic

* Remove TODOs and Debug Logs

* Fix code style issues with Black

* Remove Deprecated Test

* Fix code style issues with Black

* Specify Except Error

* Add Pydantic to Requirements

* Add Gene-Fusion Params to Pydantic Base Class

* Update Changelog

* sorted imports in 2 files

* updated changelog

* Fix Config Init() -Solves Bugs

* Refactor Parse Test

* Rewrite tests with parametrize

* Refactor test_parse_case

* Parse Tests Refactoring

* WiP: Pydantic Replacing

* Delete Bad Test

* Remove Debug Logs

* Fix code style issues with Black

* Update and Refactor Tests

* Cast Commandline sv_rank_model_version to Str

* Bugfix: Parse Synopsis

* Fix Test_parse_case

* Fix Testfile -Remove Bad Test + Formatting

* Fix code style issues with Black

* Remove Fun `parse.parse_individual`

* Add Pedigree Validation in Pydantic Parser

* Lint parse/case.py

* Adapt Tests to Pydantic

* More Generic Tests -Pydantic Required

* Bugfix Pydantic Mandatory Checks

* Fix code style issues with Black

* Update Parse Tests for Pydantic

* Update Imports

* Fix code style issues with Black

* Add `rna_coverage_bigwig` and `splice_junctions_bed`

* Remove Resolved TODOs

* Fix Lint Issues

* Remove Dead Function Import

* Fix Lint Error

* Improve Validator Funs

* Refactor Funs/Add Comments

* Rename `config_base` to `models`

* Bugfix: Madeline and Peddy Files, Paus Capture Kit

* Fix code style issues with Black

* Refactor: Remove Superfluous Parse Config Call

* Bugfix: Missing Confirmed_parent

* Add Capture Kits to Models

* Fix code style issues with Black

* Bugfix: Recursion Dict

* Remove Debug Prints

* Fix TYpo

* Update Docstrings

Update Docstrings

* Fix Broken Test

Fix Broken Test

* Model: Remove Genome Build Duplicate and Fix Display Name Alias

* Fix Broken Tests -Change Fixture

* Bugfix: Parse/case Dict Element Access

* Fix Broken Tests

* Fix code style issues with Black

* Bugfix: Genome Build Was Not Set Correct

* Resolve Conflicts Relating to `custom_images`

* Fix code style issues with Black

* Please Linter + Remove Unused Function

* Remove Test of `parse_custom_images`

* Update CHANGELOG.md

Co-authored-by: Chiara Rasi <rasi.chiara@gmail.com>

* Update scout/parse/case.py

Co-authored-by: Chiara Rasi <rasi.chiara@gmail.com>

* Update scout/parse/case.py

Co-authored-by: Chiara Rasi <rasi.chiara@gmail.com>

* Remove Debug Logs and Unused Functions

* Fix code style issues with Black

* Replace Occurences of `parse_case()` With `parse_case_config()`

* Replace Occurences of  With  2

* Move Datetime Check to Parse/models

* Fix code style issues with Black

* updated changelog

* fixed changelog

* Bugfix: Erronous Mandatory Fields

Pydantic uses '...' to mark a field as mandatory...

* Clean Code and Notes

* Fix code style issues with Black

* Bugfix: Set `display_name` Correct

* Fix code style issues with Black

* Fix Test: display_name vs. family_name

* More Explicit if Param is Optional or Mandatory

* Update Config Docs

* Fix Docs: analysis_type

* Fix Mandatory/Optional Params

* Individual_id Replaces Sample_id Internally

* Bugfix: Remove Sample_id

* WiP: Getting Image Parsing to Work

* Fix code style issues with Black

* Make Compatible With New Image Wildcard Functionality

* Fix code style issues with Black

* Remove Comment

* Fix code style issues with Black

* Fix Tests

* Fix code style issues with Black

* WiP: Getting Tests to Work

* Fix code style issues with Black

* WiP Getting Tests to Work More

* Fix code style issues with Black

* trigger new tests

* isort a couple of files

* Passes PyTests

* Fix code style issues with Black

* Update changelog

Co-authored-by: mikaell <mikael.laaksonen@mobilearts.com>
Co-authored-by: Lint Action <lint-action@samuelmeuli.com>
Co-authored-by: mikaell <mikael.laaksonen@scilifelab.se>
Co-authored-by: Daniel Nilsson <daniel.k.nilsson@gmail.com>

CHANGELOG.md

@@ -14,6 +14,7 @@ About changelog [here](https://keepachangelog.com/en/1.0.0/)
 - Reassign MatchMaker Exchange submission to another user when a Scout user is deleted
 ### Changed
 - Updated the python config file documentation in admin guide
+- Case configuration parsing now uses Pydantic for improved typechecking and config handling
 ### Fixed
 
 

docs/admin-guide/load-config.md

@@ -23,10 +23,11 @@ Below are available configuration parameters for a Scout case. Names marked with
 - **default_gene_panels** _List of strings_ List of default gene panels. Variants from the genes in the gene panels specified will be shown when opening the case in scout.
 - **delivery_report** _String_: Path to HTML delivery report.
 - **family(*)**  _String_ Unique ID of the case.
+- **family_name**  _String_ Optional name of the case.
 - **gene_fusion_report** _String_ Path to a static file containing a gene fusion report produced by [Arriba][arriba]. Generated from default clinical data.
 - **gene_fusion_report_research** _String_ Path to a static file containing a gene fusion report produced by [Arriba][arriba]. Generated from research data.
 - **gene_panels** _List of strings_ List of gene panels. Specifies what panels the case has been run with.
-- **human_genome_build** _String_ Version of genome version used.
+- **human_genome_build** _String_ Version of genome version used, 37 or 38. Defaults to 37.
 - **lims_id** _String_ Case ID in Lims
 - **madeline** _String_ Path to a madeline pedigree file in XML format.
 - **multiqc** _String_ Path to a [multiqc][multiqc] report with arbitrary information.
@@ -39,7 +40,7 @@ Below are available configuration parameters for a Scout case. Names marked with
 - **rank_score_threshold** _Float_ Only include variants with a rank score above this threshold.
 - **samples** _List_ List of samples included in the case:
 	- **alignment_path** _String_ Path to BAM/CRAM file to view alignments.
-	- **analysis_type** _String_ Specifies the analysis type for the sample. Options: {wgs, wes, panel}.
+	- **analysis_type** _String_ Specifies the analysis type for the sample. Options: {wgs, wes, panel, unknown, external}.
 	- **bam_file** _String_ Path to BAM/CRAM file to view alignments **WARNING:** Soon to be deprecated, use *alignment_path*.
 	- **bam_path** _String_ Path to BAM/CRAM file to view alignments **WARNING:** Soon to be deprecated, use *alignment_path*.
 	- **capture_kit** _String_ Exome specifies the capture kit.

requirements.txt

@@ -30,8 +30,9 @@ flask_login
 
 # Parsing
 cyvcf2<0.10.0
-PyYaml>=5.1
 ped_parser
+pydantic
+PyYaml>=5.1
 
 # extras
 intervaltree==3.0.2

scout/adapter/mongo/case.py

@@ -10,7 +10,6 @@
 from scout.build.case import build_case
 from scout.constants import ACMG_MAP
 from scout.exceptions import ConfigError, IntegrityError
-from scout.parse.case import parse_case
 from scout.parse.variant.ids import parse_document_id
 from scout.utils.algorithms import ui_score
 
@@ -632,18 +631,16 @@ def load_case(self, config_data, update=False, keep_actions=True):
         institute_obj = self.institute(config_data["owner"])
         if not institute_obj:
             raise IntegrityError("Institute '%s' does not exist in database" % config_data["owner"])
-        # Parse the case information
-        parsed_case = parse_case(config=config_data)
         # Build the case object
-        case_obj = build_case(parsed_case, self)
+        case_obj = build_case(config_data, self)
         # Check if case exists with old case id
         old_caseid = "-".join([case_obj["owner"], case_obj["display_name"]])
         old_case = self.case(old_caseid)
 
         # This is to keep sanger order and validation status
         old_sanger_variants = self.case_sanger_variants(case_obj["_id"])
 
-        genome_build = str(parsed_case.get("genome_build", 37))
+        genome_build = str(config_data.get("genome_build", 37))
 
         if old_case:
             LOG.info(

scout/commands/load/case.py

@@ -109,8 +109,8 @@ def case(
         )
         raise click.Abort()
 
-    if config_data.get("human_genome_build") not in [37, 38, "37", "38"]:
-        config_data["human_genome_build"] = int(
+    if config_data.get("genome_build") not in [37, 38, "37", "38"]:
+        config_data["genome_build"] = int(
             click.prompt(
                 f"Please enter a valid genome build for this case",
                 type=click.Choice(["37", "38"]),

scout/commands/update/case.py

@@ -152,7 +152,7 @@ def case(
         LOG.info("Set needs_check to True for case %s", case_id)
         updates = {"needs_check": True}
         if sv_rankmodel_version:
-            updates["sv_rank_model_version"] = sv_rankmodel_version
+            updates["sv_rank_model_version"] = str(sv_rankmodel_version)
         if vcf_sv:
             updates["vcf_files.vcf_sv"] = vcf_sv
         if vcf_sv:

scout/demo/643595.config.yaml

@@ -17,6 +17,8 @@ samples:
     mt_bam: scout/demo/reduced_mt.bam
     vcf2cytosure: scout/demo/ADM1059A2.test.cgh
     tissue_type: blood
+    rna_coverage_bigwig: scout/demo/ACC5963A1_lanes_1234_star_sorted_sj_filtered.bigWig
+    splice_junctions_bed: scout/demo/ACC5963A1_lanes_1234_star_sorted_sj_filtered_sorted.bed.gz
 
   - analysis_type: wes
     sample_id: ADM1059A1