Fixing issue with UNLOGGED tables (#161) (#164)

* Adding migration to fix UNLOGGED tables problem.
* Fixing CI for upstream changed files.

.github/workflows/ci.yml

@@ -45,6 +45,7 @@ jobs:
     steps:
       - name: install server dependencies
         run: |
+          sudo apt-get update
           sudo apt-get install -qq build-essential zlib1g-dev libtiff5-dev libjpeg8-dev libfreetype6-dev
           sudo apt-get install -qq liblcms2-dev libwebp-dev libpq-dev graphviz-dev
       - name: install Python

HISTORY.rst

@@ -6,16 +6,30 @@ History / Changelog
 v0.23.6
 -------
 
+**IMPORTANT**
+
+This release contains a critical update.
+Prior to this release, all small and structural variant tables were marked as ``UNLOGGED``.
+This was originally introduce to improve insert performance.
+However, it turned out that stability is greatly decreased.
+In the case of a PostgreSQL crash, these tables are emptied.
+This change should have been rolled back much earlier but that rollback was buggy.
+**This release now includes a working and verified fix.**
+
 End-User Summary
 ================
 
 - Fixing problem with remote permission synchronization.
+- Fixing stability issue with database schema.
 
 Full Change List
 ================
 
 - Bump sodar-core to hotfix version.
   Fixes problem with remote permission synchronization.
+- Adding migration to mark all ``UNLOGGED`` tables back to ``LOGGED``.
+  This should have been reverted earlier but because of a bug it did not.
+- Fixing CI by calling ``sudo apt-get update`` once more.
 
 -------
 v0.23.5

svs/migrations/0015_set_logged_table.py

@@ -0,0 +1,32 @@
+# -*- coding: utf-8 -*-
+"""SET svs tables as LOGGED against data loss."""
+from __future__ import unicode_literals
+
+from django.db import migrations
+from django.conf import settings
+
+if not settings.IS_TESTING:
+    operations = (
+        [
+            migrations.RunSQL("ALTER TABLE svs_structuralvariant SET LOGGED;"),
+            migrations.RunSQL("ALTER TABLE svs_structuralvariantgeneannotation SET LOGGED;"),
+        ]
+        + [
+            migrations.RunSQL("ALTER TABLE svs_structuralvariant%d SET LOGGED;" % i)
+            for i in range(settings.VARFISH_PARTITION_MODULUS_SMALLVARIANT)
+        ]
+        + [
+            migrations.RunSQL("ALTER TABLE svs_structuralvariantgeneannotation%d SET LOGGED;" % i)
+            for i in range(settings.VARFISH_PARTITION_MODULUS_SMALLVARIANT)
+        ]
+    )
+else:
+    operations = []
+
+
+class Migration(migrations.Migration):
+    atomic = False
+
+    dependencies = [("svs", "0014_svs_stucturalvariants_alter_unique_constraint")]
+
+    operations = operations

variants/migrations/0069_set_logged_table.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-"""SET small variant table as UNLOGGED to improve insertion performance."""
+"""SET small variant table as LOGGED to improve stability."""
 from __future__ import unicode_literals
 
 from django.db import migrations

variants/migrations/0081_set_logged_table_again.py

@@ -0,0 +1,22 @@
+# -*- coding: utf-8 -*-
+"""SET small variant table as LOGGED to improve stability (#2)."""
+from __future__ import unicode_literals
+
+from django.db import migrations
+from django.conf import settings
+
+if not settings.IS_TESTING:
+    operations = [migrations.RunSQL("ALTER TABLE variants_smallvariant SET LOGGED;")] + [
+        migrations.RunSQL("ALTER TABLE variants_smallvariant_%d SET LOGGED" % i)
+        for i in range(settings.VARFISH_PARTITION_MODULUS_SMALLVARIANT)
+    ]
+else:
+    operations = []
+
+
+class Migration(migrations.Migration):
+    atomic = False
+
+    dependencies = [("variants", "0080_spanrsubmissionbgjob")]
+
+    operations = operations