merge v0.14.0 release

.github/ISSUE_TEMPLATE/release_cleanup.md

@@ -0,0 +1,31 @@
+---
+name: Release Cleanup
+about: Minor tasks and checklist for maintainer to cleanup and prepare a release
+title: 'Cleanup and prepare RELEASE_VERSION'
+labels: documentation, internal
+assignees: 'mikkonie'
+
+---
+
+## Minor Tasks
+
+TBA
+
+## Issues to Add in CHANGELOG
+
+TBA
+
+## Release Checklist
+
+- [ ] Review code style and cleanup
+- [ ] Review and update docs entries
+- [ ] Update `SODAR_API_DEFAULT_VERSION` and `SODAR_API_ALLOWED_VERSIONS`
+- [ ] Update Vue app version with `npm version`
+- [ ] Update version in CHANGELOG and SODAR Release Notes doc
+- [ ] Update version in docs conf.py
+- [ ] Ensure both SODAR and SODAR Core API versioning is correct in API docs
+- [ ] Ensure docs can be built without errors
+
+## Notes
+
+N/A

.github/workflows/build.yml

@@ -67,9 +67,9 @@ jobs:
           pip install -r requirements/local.txt
           pip install -r requirements/test.txt
       - name: Setup Node.js
-        uses: actions/setup-node@v1
+        uses: actions/setup-node@v3
         with:
-          node-version: 16.x
+          node-version: 18.x
       - name: Build and run Vue app
         run: |
           npm ci --prefix samplesheets/vueapp

AUTHORS.rst

@@ -6,5 +6,6 @@ Credits
 * Oliver Stolpe <oliver.stolpe@bih-charite.de>
 * Dzmitry Hramyka <dzmitry.hramyka@bih-charite.de>
 * Mathias Kuhring <mathias.kuhring@bih-charite.de>
+* Thomas Sell <thomas.sell@charite.de>
 * Franziska Schumann <franziska.schumann@fu-berlin.de>
 * Tim Garrels <tim.garrels@student.hpi.uni-potsdam.de>

CHANGELOG.rst

@@ -5,6 +5,124 @@ Changelog for the SODAR project. Loosely follows the
 `Keep a Changelog <http://keepachangelog.com/en/1.0.0/>`_ guidelines.
 
 
+v0.14.0 (2023-09-27)
+====================
+
+Added
+-----
+
+- **General**
+    - Release cleanup issue template (#1797)
+    - LDAP settings for TLS and user filter (#1803)
+- **Irodsbackend**
+    - ``get_trash_path()`` helper (#1658)
+    - iRODS trash statistics for siteinfo (#1658)
+- **Irodsinfo**
+    - ``IrodsEnvRetrieveAPIView`` for retrieving iRODS environment (#1685)
+- **Landingzones**
+    - Landing zone updating (#1267)
+    - "Nothing to do" check for landing zone validation and moving (#339)
+    - iRODS path clipboard copying button in iRODS collection list modal (#1282)
+    - ``constants`` module for zone constants (#1398)
+    - Assay link from zone assay icon (#1747)
+    - Missing permission tests (#1739)
+- **Samplesheets**
+    - User setting for study and assay table height (#1283)
+    - Study table cache disabling (#1639)
+    - ``SHEETS_ENABLE_STUDY_TABLE_CACHE`` setting (#1639)
+    - ``cytof`` assay plugin (#1642)
+    - New ISA-Tab templates from ``cubi-isa-templates`` (#1697, #1757)
+    - General iRODS access ticket management for assay collections (#804, #1717)
+    - Disabled row delete button tooltips (#1731)
+    - ``IrodsDataRequest`` REST API views (#1588, #1706, #1734, #1735, #1736)
+    - Davrods links in iRODS delete request list (#1339)
+    - Batch accepting and rejecting for iRODS delete requests (#1340, #1751)
+    - Cookiecutter prompt support in sheet templates (#1726)
+    - "Create" tag for sheet versions (#1296)
+    - Template tag tests (#1723)
+    - iRODS file count in sheet overview tab (#1295)
+    - ``get_url()`` helpers for ``Investigation``, ``Study`` and ``Assay`` models (#1748)
+    - ``normalizesheets`` management command for sheet cleanup (#1661)
+    - Boolean field support in sheet templates (#1757)
+    - iRODS access ticket REST API views (#1707, #1800, #1801)
+- **Taskflowbackend**
+    - ``BatchCalculateChecksumTask`` iRODS task (#1634)
+    - Automated generation of missing checksums in ``landing_zone_move`` (#1634, #1767)
+    - Cleanup of trash collections in testing (#1658)
+    - ``TaskflowPermissionTestBase`` base test class (#1718)
+    - Taskflow session timeout management (#1768)
+    - ``TASKFLOW_IRODS_CONN_TIMEOUT`` Django setting (#1768)
+
+Changed
+-------
+
+- **General**
+    - Upgrade to django-sodar-core v0.13.2 (#1617, #1720, #1775, #1792)
+    - Upgrade to cubi-isa-templates v0.1.0 (#1757)
+    - Upgrade to python-irodsclient v1.1.8 (#1538)
+    - Upgrade Python dependencies (#1620)
+    - Upgrade Vue app dependencies (#1620)
+    - Upgrade to nodejs v18 (#1765, #1766)
+    - Update deprecated Nodejs install method in Docker and dev (#1769)
+    - Timeline event names and descriptions if called from syncmodifyapi (#1761)
+    - Update tour help (#1583)
+    - Enable setting ``ADMINS`` via environment variable (#1796)
+    - Update ``ADMINS`` default value (#1796)
+- **Irodsadmin**
+    - Output ``irodsorphans`` results during execution (#1319)
+    - Order ``irodsorphans`` results by project (#1741)
+- **Landingzones**
+    - Move iRODS object helpers to ``TaskflowTestMixin`` (#1699)
+    - Enable superuser landing zone controls for locked zones (#1607)
+    - Add ``DELETING`` to locked states in UI (#1657)
+    - Query for landing zone status in batch (#1684, #1752)
+    - Create expected collections if zone sync is called from syncmodifyapi (#1761)
+    - Define and use zone status constants (#1398)
+- **Samplesheets**
+    - Sample sheet table viewport background color (#1692)
+    - Contract sheet table height to fit content (#1693)
+    - Hide internal fields from ISA-Tab templates (#1698, #1733)
+    - Refactor ``IrodsDataRequest`` model and tests (#1706)
+    - Update ``get_sheets_url()`` helper to only handle ``Project`` objects (#1771)
+    - Display full path under assay for iRODS data requests in UI (#1749)
+    - Return full path under assay from ``IrodsDataRequest.get_short_path()`` (#1749)
+    - Make ``request`` optional in ``SheetVersionMixin.save_version()``
+- **Taskflowbackend**
+    - Move iRODS object helpers from ``LandingZoneTaskflowMixin`` (#1699)
+    - Move iRODS test cleanup to ``TaskflowTestMixin.clear_irods_test_data()`` (#1722)
+    - Refactor base test classes (#1722)
+
+Fixed
+-----
+
+- **General**
+    - Local Chromedriver install failure (#1753, bihealth/sodar-core#1255)
+- **Ontologyaccess**
+    - Batch import tests failing from forbidden obolibrary access (#1694)
+- **Samplesheets**
+    - ``perform_project_sync()`` crash with no iRODS collections created (#1687)
+    - iRODS delete request modification UI view permission checks failing for non-creator contributors (#1737)
+    - Investigation object ref broken in timeline ``sheet_replace`` events (#1774)
+    - External links column width estimation crash in table rendering (#1787)
+    - Comment field editing with semicolon in data (#1790)
+    - Ontology URLs not encoded if passed as query string in wrapper template (#1762)
+
+Removed
+-------
+
+- **Landingzones**
+    - Unused ``data_tables`` references from templates (#1710)
+    - ``get_zone_samples_url()`` template tag (#1748)
+- **Samplesheets**
+    - ``SHEETS_TABLE_HEIGHT`` Django setting (#1283)
+    - Duplicate ``IrodsAccessTicketMixin`` from ``test_views_ajax`` (#1703)
+    - ``IRODS_DATA_REQUEST_STATUS_CHOICES`` constant (#1706)
+    - ``HIDDEN_SHEET_TEMPLATE_FIELDS`` constant (#1733)
+    - ``sheet_export*`` timeline events (#1773)
+    - ``SHEETS_ENABLED_TEMPLATES`` Django setting (#1756)
+    - ``tumor_normal_triplets`` ISA-Tab template (#1757)
+
+
 v0.13.4 (2023-05-15)
 ====================
 

config/settings/base.py

@@ -53,6 +53,7 @@
 ]
 THIRD_PARTY_APPS = [
     'crispy_forms',  # Form layouts
+    'crispy_bootstrap4',  # Bootstrap4 theme for Crispy
     'rules.apps.AutodiscoverRulesConfig',  # Django rules engine
     'djangoplugins',  # Django plugins
     'pagedown',  # For markdown
@@ -100,6 +101,7 @@
     'samplesheets.assayapps.meta_ms.apps.MetaMsConfig',
     'samplesheets.assayapps.microarray.apps.MicroarrayConfig',
     'samplesheets.assayapps.pep_ms.apps.PepMsConfig',
+    'samplesheets.assayapps.cytof.apps.CytofConfig',
     # Landingzones config sub-apps
     'landingzones.configapps.bih_proteomics_smb.apps.BihProteomicsSmbConfig',
     # Admin apps
@@ -144,8 +146,8 @@
 
 # MANAGER CONFIGURATION
 # ------------------------------------------------------------------------------
-ADMINS = [("""Mikko Nieminen""", 'mikko.nieminen@bih-charite.de')]
-
+# Provide ADMINS as: Name:email,Name:email
+ADMINS = [x.split(':') for x in env.list('ADMINS', default=[])]
 # See: https://docs.djangoproject.com/en/3.2/ref/settings/#managers
 MANAGERS = ADMINS
 
@@ -212,6 +214,7 @@
                 'projectroles.context_processors.urls_processor',
                 'projectroles.context_processors.site_app_processor',
                 'projectroles.context_processors.app_alerts_processor',
+                'projectroles.context_processors.sidebar_processor',
             ],
         },
     }
@@ -355,7 +358,6 @@
 
     # Default values
     LDAP_DEFAULT_CONN_OPTIONS = {ldap.OPT_REFERRALS: 0}
-    LDAP_DEFAULT_FILTERSTR = '(sAMAccountName=%(user)s)'
     LDAP_DEFAULT_ATTR_MAP = {
         'first_name': 'givenName',
         'last_name': 'sn',
@@ -366,12 +368,22 @@
     AUTH_LDAP_SERVER_URI = env.str('AUTH_LDAP_SERVER_URI', None)
     AUTH_LDAP_BIND_DN = env.str('AUTH_LDAP_BIND_DN', None)
     AUTH_LDAP_BIND_PASSWORD = env.str('AUTH_LDAP_BIND_PASSWORD', None)
-    AUTH_LDAP_CONNECTION_OPTIONS = LDAP_DEFAULT_CONN_OPTIONS
+    AUTH_LDAP_START_TLS = env.str('AUTH_LDAP_START_TLS', False)
+    AUTH_LDAP_CA_CERT_FILE = env.str('AUTH_LDAP_CA_CERT_FILE', None)
+    AUTH_LDAP_CONNECTION_OPTIONS = {**LDAP_DEFAULT_CONN_OPTIONS}
+    if AUTH_LDAP_CA_CERT_FILE is not None:
+        AUTH_LDAP_CONNECTION_OPTIONS[
+            ldap.OPT_X_TLS_CACERTFILE
+        ] = AUTH_LDAP_CA_CERT_FILE
+        AUTH_LDAP_CONNECTION_OPTIONS[ldap.OPT_X_TLS_NEWCTX] = 0
+    AUTH_LDAP_USER_FILTER = env.str(
+        'AUTH_LDAP_USER_FILTER', '(sAMAccountName=%(user)s)'
+    )
 
     AUTH_LDAP_USER_SEARCH = LDAPSearch(
         env.str('AUTH_LDAP_USER_SEARCH_BASE', None),
         ldap.SCOPE_SUBTREE,
-        LDAP_DEFAULT_FILTERSTR,
+        AUTH_LDAP_USER_FILTER,
     )
     AUTH_LDAP_USER_ATTR_MAP = LDAP_DEFAULT_ATTR_MAP
     AUTH_LDAP_USERNAME_DOMAIN = env.str('AUTH_LDAP_USERNAME_DOMAIN', None)
@@ -391,12 +403,22 @@
         AUTH_LDAP2_SERVER_URI = env.str('AUTH_LDAP2_SERVER_URI', None)
         AUTH_LDAP2_BIND_DN = env.str('AUTH_LDAP2_BIND_DN', None)
         AUTH_LDAP2_BIND_PASSWORD = env.str('AUTH_LDAP2_BIND_PASSWORD', None)
-        AUTH_LDAP2_CONNECTION_OPTIONS = LDAP_DEFAULT_CONN_OPTIONS
+        AUTH_LDAP2_START_TLS = env.str('AUTH_LDAP2_START_TLS', False)
+        AUTH_LDAP2_CA_CERT_FILE = env.str('AUTH_LDAP2_CA_CERT_FILE', None)
+        AUTH_LDAP2_CONNECTION_OPTIONS = {**LDAP_DEFAULT_CONN_OPTIONS}
+        if AUTH_LDAP2_CA_CERT_FILE is not None:
+            AUTH_LDAP2_CONNECTION_OPTIONS[
+                ldap.OPT_X_TLS_CACERTFILE
+            ] = AUTH_LDAP2_CA_CERT_FILE
+            AUTH_LDAP2_CONNECTION_OPTIONS[ldap.OPT_X_TLS_NEWCTX] = 0
+        AUTH_LDAP2_USER_FILTER = env.str(
+            'AUTH_LDAP2_USER_FILTER', '(sAMAccountName=%(user)s)'
+        )
 
         AUTH_LDAP2_USER_SEARCH = LDAPSearch(
             env.str('AUTH_LDAP2_USER_SEARCH_BASE', None),
             ldap.SCOPE_SUBTREE,
-            LDAP_DEFAULT_FILTERSTR,
+            AUTH_LDAP2_USER_FILTER,
         )
         AUTH_LDAP2_USER_ATTR_MAP = LDAP_DEFAULT_ATTR_MAP
         AUTH_LDAP2_USERNAME_DOMAIN = env.str('AUTH_LDAP2_USERNAME_DOMAIN')
@@ -586,7 +608,7 @@ def set_logging(level=None):
 
 
 # General API settings
-SODAR_API_DEFAULT_VERSION = '0.13.4'
+SODAR_API_DEFAULT_VERSION = '0.14.0'
 SODAR_API_ALLOWED_VERSIONS = [
     '0.7.0',
     '0.7.1',
@@ -605,6 +627,7 @@ def set_logging(level=None):
     '0.13.2',
     '0.13.3',
     '0.13.4',
+    '0.14.0',
 ]
 SODAR_API_MEDIA_TYPE = 'application/vnd.bihealth.sodar+json'
 SODAR_API_DEFAULT_HOST = env.url(
@@ -614,6 +637,10 @@ def set_logging(level=None):
 
 # Projectroles app settings
 PROJECTROLES_SITE_MODE = env.str('PROJECTROLES_SITE_MODE', 'SOURCE')
+PROJECTROLES_TEMPLATE_INCLUDE_PATH = env.path(
+    'PROJECTROLES_TEMPLATE_INCLUDE_PATH',
+    os.path.join(APPS_DIR, 'templates', 'include'),
+)
 PROJECTROLES_SECRET_LENGTH = env.int('PROJECTROLES_SECRET_LENGTH', 32)
 PROJECTROLES_INVITE_EXPIRY_DAYS = env.int('PROJECTROLES_INVITE_EXPIRY_DAYS', 14)
 PROJECTROLES_SEND_EMAIL = env.bool('PROJECTROLES_SEND_EMAIL', False)
@@ -712,6 +739,8 @@ def set_logging(level=None):
 
 
 # Taskflow backend settings
+# Connection timeout for taskflowbackend flows (other sessions not affected)
+TASKFLOW_IRODS_CONN_TIMEOUT = env.int('TASKFLOW_IRODS_CONN_TIMEOUT', 480)
 TASKFLOW_LOCK_RETRY_COUNT = env.int('TASKFLOW_LOCK_RETRY_COUNT', 2)
 TASKFLOW_LOCK_RETRY_INTERVAL = env.int('TASKFLOW_LOCK_RETRY_INTERVAL', 3)
 TASKFLOW_LOCK_ENABLED = True
@@ -740,10 +769,12 @@ def set_logging(level=None):
 SHEETS_ALLOW_CRITICAL = env.bool('SHEETS_ALLOW_CRITICAL', False)
 # Temporary, see issue #556
 SHEETS_ENABLE_CACHE = True
+# Enable study table cache
+SHEETS_ENABLE_STUDY_TABLE_CACHE = env.bool(
+    'SHEETS_ENABLE_STUDY_TABLE_CACHE', True
+)
 # iRODS file query limit
 SHEETS_IRODS_LIMIT = env.int('SHEETS_IRODS_LIMIT', 50)
-# Study/assay table height
-SHEETS_TABLE_HEIGHT = env.int('SHEETS_TABLE_HEIGHT', 400)
 # Minimum edit config version
 SHEETS_CONFIG_VERSION = '0.8.0'
 # Min default column width
@@ -774,17 +805,6 @@ def set_logging(level=None):
     os.path.join(ROOT_DIR, 'samplesheets/config/ext_links.json'),
 )
 
-# HACK: Supported cubi-tk templates, excluding ones which altamISA cannot parse
-SHEETS_ENABLED_TEMPLATES = [
-    'bulk_rnaseq',
-    'generic',
-    'germline',
-    'microarray',
-    'ms_meta_biocrates',
-    'single_cell_rnaseq',
-    'tumor_normal_triplets',
-]
-
 # Remote sample sheet sync interval in minutes
 SHEETS_SYNC_INTERVAL = env.int('SHEETS_SYNC_INTERVAL', 5)
 

config/settings/production.py

@@ -144,6 +144,9 @@
 # ------------------------------------------------------------------------------
 
 
+# Samplesheets app settings
+SHEETS_ENABLE_STUDY_TABLE_CACHE = True
+
 # Plugin settings
 ENABLED_BACKEND_PLUGINS = env.list(
     'ENABLED_BACKEND_PLUGINS',

config/settings/test.py

@@ -18,6 +18,11 @@
 # Note: This key only used for development and testing.
 SECRET_KEY = env('DJANGO_SECRET_KEY', default='CHANGEME!!!')
 
+# MANAGER CONFIGURATION
+# ------------------------------------------------------------------------------
+ADMINS = [('Admin User', 'admin@example.com')]
+MANAGERS = ADMINS
+
 # Mail settings
 # ------------------------------------------------------------------------------
 EMAIL_HOST = 'localhost'
@@ -98,6 +103,7 @@
 
 # Samplesheets app settings
 SHEETS_ENABLE_CACHE = False  # Temporarily disabled to fix CI, see issue #556
+SHEETS_ENABLE_STUDY_TABLE_CACHE = True
 SHEETS_EXTERNAL_LINK_PATH = os.path.join(
     ROOT_DIR, 'samplesheets/tests/config/ext_links.json'
 )

config/urls.py

@@ -33,11 +33,7 @@ def handler500(request, *args, **argv):
         view=auth_views.LoginView.as_view(template_name='users/login.html'),
         name='login',
     ),
-    path(
-        route='logout/',
-        view=auth_views.logout_then_login,
-        name='logout',
-    ),
+    path(route='logout/', view=auth_views.logout_then_login, name='logout'),
     # User Profile URLs
     path('user/', include('userprofile.urls')),
     # Auth