Merge fd5bc7c into 1f2f46c

HISTORY.rst

@@ -29,9 +29,16 @@ End-User Summary
 - Improved developer setup documentation and added Windows installation instructions (#533).
 - Fixed broken VariantValidator query (#523).
 - Fixed smallvariant flags filter query (#502).
-- Added flags `segregates`, `doesnt_segregate` and `no_disease_association` to file export (#502).
+- Added flags ``segregates``, ``doesnt_segregate`` and ``no_disease_association`` to file export (#502).
 - Adjusting path to new varfish-annotator db download (#546).
 - Adding feature to enable and configure link-out to HGMD (#576).
+- The StructuralVariant model is augmented (#566).
+  This adds capability to properly represent break-end (BND) records.
+  Also, per--genotype type counts are added.
+  A full re-import after re-annotation with varfish-annotator v0.23 or above is recommended.
+  Alternatively, you can use ``python manage.py svs_sv_fill_nulls`` to update the records on the fly.
+- Implement new in-house background database for structural variants (#32).
+- Allow to exclude cases from in-house database through project settings (#579).
 
 Full Change List
 ================
@@ -59,10 +66,12 @@ Full Change List
 - Fixed broken VariantValidator query (#523).
 - Converted not cooperative tooltip to standard title on Filter & Display button (#508).
 - Fixed smallvariant flags filter query (#502).
-- Added flags `segregates`, `doesnt_segregate` and `no_disease_association` to file export (#502).
+- Added flags ``segregates``, ``doesnt_segregate`` and ``no_disease_association`` to file export (#502).
 - Adjusting path to new varfish-annotator db download (#546).
 - Fixing issue with sync-from-remote when no remote is defined (#570).
 - Adding feature to enable and configure link-out to HGMD (#576).
+- Implement new in-house background database for structural variants (#32).
+- Allow to exclude cases from in-house database through project settings (#579).
 
 ------
 v1.2.0

config/settings/base.py

@@ -508,6 +508,9 @@ def set_logging(level=None):
 # Number of cases to perform in one query for joint queries.
 QUERY_MAX_UNION = env.int("VARFISH_QUERY_MAX_UNION", 20)
 
+# Timeout (in hours) for VarFish cleaning up background SV sets in "building" state.
+SV_CLEANUP_BUILDING_SV_SETS = env.int("VARFISH_SV_CLEANUP_BUILDING_SV_SETS", 48)
+
 # Varfish: Exomiser
 # ------------------------------------------------------------------------------
 

docs_manual/admin_upgrade.rst

@@ -26,6 +26,50 @@ You can then apply the patch to your database with the following command.
 
 You can find out more details, give feedback, and ask for help `in this Github discussion <https://github.com/bihealth/varfish-server/discussions/451>`__.
 
+----------------
+v1.2.* to v2.*.*
+----------------
+
+**In-House Background Database.**
+A number of changes were made to the implementation of the background database.
+The upgrade will re-create the in-house database as empty.
+
+You will have to re-build the database manually with the command ``python manage.py rebuild_variant_summary``.
+Assuming that you are running within ``varfish-docker-compose``, you can use the following command directly.
+
+::
+
+    $ docker exec -it varfish-docker-compose_varfish-web_1 python /usr/src/app/manage.py \
+        rebuild_variant_summary
+
+**Structural Variants.**
+In case that the support for structural variants has been used, it is **strongly recommended** to re-annotate the structural variants with an updated version of ``varfish-annotator`` (v0.24 or above).
+You will need to use ``varfish-cli`` in a recent version (v0.3.4 or above) for being able to import the data into VarFish.
+Otherwise, see below for background and explanation on how to fill empty values in your database after upgrading the software an database to bollonaster.
+
+In VarFish anthenea, the support for structural variants has been experimental only.
+To prepare for improved support of structural variants, the database structure has been changed.
+Prior, the number of carriers in the background had to be annotated outside of VarFish (in your pipeline).
+This has changed now and VarFish can build a background database (and will do so weekly on sundays).
+
+If you do not want to wait for next sunday to get your background database, you can force building a new background structural variant data set with:
+
+::
+
+    $ docker exec -it varfish-docker-compose_varfish-web_1 python /usr/src/app/manage.py \
+        svs_bg_sv_set_build
+
+New fields have been added for the proper representation of break-ends in VarFish.
+They will only be properly filled after re-import of re-annotated data as described above.
+You can fill the fields with reasonable values (that will work well for all cases except for breakends/BNDs where the connectivity of 3' and 5' ends cannot be properly detected without re-annotation) by using the following.
+
+::
+
+    $ docker exec -it varfish-docker-compose_varfish-web_1 python /usr/src/app/manage.py \
+        svs_sv_fill_nulls
+
+This is not strictly necessary and it is recommended to re-annotate and re-import.
+
 ------------------
 v0.23.0 to v1.2.1
 ------------------

importer/models.py

@@ -4,7 +4,6 @@
 import tempfile
 import uuid as uuid_object
 
-import aldjemy
 from bgjobs.models import BackgroundJob, LOG_LEVEL_ERROR, JobModelMessageMixin
 from django.contrib import auth
 from django.db import models, transaction

requirements/base.txt

@@ -154,3 +154,15 @@ django-saml2-auth-ai>=2.1.6, <2.2
 # Code formatting.
 black ==19.10b0
 click >=8.0,<8.1
+
+# Editable interval trees
+intervaltree >=3.1.0,<4.0.0
+
+# Pretty tables
+prettytable >=3.3.0,<4.0
+
+# process introspection
+psutil >=5.9.0,<6.0
+
+# progress display
+tqdm