Implementing new in-house background for svs (#32)

Closes: #32
Related-Issue: #32
Projected-Results-Impact: require-revalidation

HISTORY.rst

@@ -80,6 +80,7 @@ Full Change List
 - Fixed bug where Exac and thousand genomes settings were not shown in frequency tab for GRCh37 (#597).
 - Form template reports error if genomebuild variable is not set (#607).
 - Making ``keyvalue`` more robust to failure (#613).
+- Implement new in-house background database for structural variants (#32).
 
 ------
 v1.2.0

config/settings/base.py

@@ -508,6 +508,9 @@ def set_logging(level=None):
 # Number of cases to perform in one query for joint queries.
 QUERY_MAX_UNION = env.int("VARFISH_QUERY_MAX_UNION", 20)
 
+# Timeout (in hours) for VarFish cleaning up background SV sets in "building" state.
+SV_CLEANUP_BUILDING_SV_SETS = env.int("VARFISH_SV_CLEANUP_BUILDING_SV_SETS", 48)
+
 # Varfish: Exomiser
 # ------------------------------------------------------------------------------
 

docs_manual/admin_upgrade.rst

@@ -26,6 +26,38 @@ You can then apply the patch to your database with the following command.
 
 You can find out more details, give feedback, and ask for help `in this Github discussion <https://github.com/bihealth/varfish-server/discussions/451>`__.
 
+----------------
+v1.2.* to v2.*.*
+----------------
+
+**Structural Variants.**
+In case that the support for structural variants has been used, it is **strongly recommended** to re-annotate the structural variants with an updated version of ``varfish-annotator`` (v0.24 or above).
+You will need to use ``varfish-cli`` in a recent version (v0.3.4 or above) for being able to import the data into VarFish.
+Otherwise, see below for background and explanation on how to fill empty values in your database after upgrading the software an database to bollonaster.
+
+In VarFish anthenea, the support for structural variants has been experimental only.
+To prepare for improved support of structural variants, the database structure has been changed.
+Prior, the number of carriers in the background had to be annotated outside of VarFish (in your pipeline).
+This has changed now and VarFish can build a background database (and will do so weekly on sundays).
+
+If you do not want to wait for next sunday to get your background database, you can force building a new background structural variant data set with:
+
+::
+
+    $ docker exec -it varfish-docker-compose_varfish-web_1 python /usr/src/app/manage.py \
+        svs_bg_sv_set_build
+
+New fields have been added for the proper representation of break-ends in VarFish.
+They will only be properly filled after re-import of re-annotated data as described above.
+You can fill the fields with reasonable values (that will work well for all cases except for breakends/BNDs where the connectivity of 3' and 5' ends cannot be properly detected without re-annotation) by using the following.
+
+::
+
+    $ docker exec -it varfish-docker-compose_varfish-web_1 python /usr/src/app/manage.py \
+        svs_sv_fill_nulls
+
+This is not strictly necessary and it is recommended to re-annotate and re-import.
+
 ------------------
 v0.23.0 to v1.2.1
 ------------------

importer/models.py

@@ -4,7 +4,6 @@
 import tempfile
 import uuid as uuid_object
 
-import aldjemy
 from bgjobs.models import BackgroundJob, LOG_LEVEL_ERROR, JobModelMessageMixin
 from django.contrib import auth
 from django.db import models, transaction