Merge branch 'develop' into dependabot/pip/conf/cryptography-41.0.6

README.md

@@ -7,36 +7,56 @@
 
 The introduction of massive sequencing (MS) in genomics facilities has meant an exponential growth in data generation, requiring a precise tracking system, from library preparation to fastq file generation, analysis and delivery to the researcher. Software designed to handle those tasks are called Laboratory Information Management Systems (LIMS), and its software has to be adapted to their own genomics laboratory particular needs. iSkyLIMS is born with the aim of helping with the wet laboratory tasks, and implementing a workflow that guides genomics labs on their activities from library preparation to data production, reducing potential errors associated to high throughput technology, and facilitating the quality control of the sequencing. Also, iSkyLIMS connects the wet lab with dry lab facilitating data analysis by bioinformaticians.
 
-![Image](https://github.com/BU-ISCIII/iskylims/blob/main/img/iskylims_scheme.png)
+![Image](img/iskylims_scheme.png)
 
 According to existent infrastructure sequencing is performed on an Illumina NextSeq instrument. Data is stored in NetApp mass storage device and fastq files are generated (bcl2fastq) on a Sun Grid Engine High Performance Computing cluster (SGE-HPC).
 Application servers run web applications for bioinformatics analysis (GALAXY), the iSkyLIMS app, and host the MySQL information tier. iSkyLIMS WetLab workflow deals with sequencing run tracking and statistics. Run tracking passes through five states: "recorded” genomics user record the new sequencing run into the system, the process will wait till run is completed by the machine and data is transferred to the mass storage device; “Sample sheet sent” sample sheet file with the sequencing run information will be copied to the run folder for bcl2fastq process; “Processing data” run parameters files are processed and data is stored in the database; “Running stats” demultiplexing data generated in bcl2fastq process is processed and stored into the database, “Completed” all data is processed and stored successfully. Statistics per sample, per project, per run and per investigation are provided, as well as annual and monthly reports. iSkyLIMS DryLab workflow deals with bioinformatics services request and statistics. User request services that can be associated with a sequencing run. Stats and services tracking is provided.
 
 - [iSkyLIMS](#iskylims)
   - [Installation](#installation)
-  - [iSkyLIMS docker installation](#iskylims-docker-installation)
-  - [Install iSkyLIMS in your server running ubuntu/CentOS](#install-iskylims-in-your-server-running-ubuntucentos)
     - [Pre-requisites](#pre-requisites)
-    - [Clone github repository](#clone-github-repository)
-    - [Create iskylims database and grant permissions](#create-iskylims-database-and-grant-permissions)
-    - [Configuration settings](#configuration-settings)
-    - [Run installation script](#run-installation-script)
-    - [Configure Apache server](#configure-apache-server)
-    - [Installation verification](#installation-verification)
-  - [Upgrade to iSkyLIMS version 3.0.0](#upgrade-to-iskylims-version-300)
-    - [Pre-requisites](#pre-requisites-1)
-    - [Executing the upgrade](#executing-the-upgrade)
-    - [Clone github repository](#clone-github-repository-1)
-    - [Configuration settings](#configuration-settings-1)
-    - [Runing upgrade script](#runing-upgrade-script)
-    - [Verification installation](#verification-installation)
-  - [iSkyLIMS documentation](#iskylims-documentation)
+    - [iSkyLIMS docker installation](#iskylims-docker-installation)
+    - [Install iSkyLIMS in your server running ubuntu/CentOS](#install-iskylims-in-your-server-running-ubuntucentos)
+      - [Clone github repository](#clone-github-repository)
+      - [Create iskylims database and grant permissions](#create-iskylims-database-and-grant-permissions)
+      - [Configuration settings](#configuration-settings)
+      - [Run installation script](#run-installation-script)
+    - [Upgrade to iSkyLIMS version 3.0.0](#upgrade-to-iskylims-version-300)
+      - [Pre-requisites](#pre-requisites-1)
+      - [Clone github repository](#clone-github-repository-1)
+      - [Configuration settings](#configuration-settings-1)
+      - [Running upgrade script](#running-upgrade-script)
+        - [Steps requiring root](#steps-requiring-root)
+        - [Steps not requiring root](#steps-not-requiring-root)
+      - [What to do if something fails](#what-to-do-if-something-fails)
+    - [Final configuration steps](#final-configuration-steps)
+      - [SAMBA configurarion](#samba-configurarion)
+      - [Email verification](#email-verification)
+      - [Configure Apache server](#configure-apache-server)
+      - [Verification of the installation](#verification-of-the-installation)
+    - [iSkyLIMS documentation](#iskylims-documentation)
 
 ## Installation
 
 For any problems or bug reporting please post us an [issue](https://github.com/BU-ISCIII/iSkyLIMS/issues)
 
-## iSkyLIMS docker installation
+### Pre-requisites
+
+Before starting the installation make sure :
+
+- You have **sudo privileges** to install the additional software packets that iSkyLIMS needs.
+- Database MySQL > 8.0 or MariaDB > 10.4
+- Local server configured for sending emails
+- Apache server v2.4
+- git > 2.34
+- Python > 3.8
+- Connection to samba shared folder where run folders are stored (p.e galera/NGS_Data)
+- Dependencies:
+  - lsb_release:
+    - RedHat/CentOS: ```yum install redhat-lsb-core```
+    - Ubuntu: ```apt install lsb-core lsb-release```
+
+### iSkyLIMS docker installation
 
 You can test iSkyLIMS by creating a docker container on your local machine.
 
@@ -59,24 +79,9 @@ Follow the prompt message to create the super user account.
 
 When script ends open your navigator typing **localhost:8001** to access to iSkyLIMS
 
-## Install iSkyLIMS in your server running ubuntu/CentOS
-
-### Pre-requisites
-
-Before starting the installation make sure :
-
-- You have **sudo privileges** to install the additional software packets that iSkyLIMS needs.
-- Database (MySQL/MariaDB) > 10.4 is running.
-- Local server configured for sending emails
-- Apache server is running.
-- Python > 3.8 installed
-- Connection to samba dir where run folders are stored.
-*- Dependencies:
-  - lsb_release:
-    - RedHat/CentOS: ```yum install redhat-lsb-core```
-    - Ubuntu: ```apt install lsb-core lsb-release```
+### Install iSkyLIMS in your server running ubuntu/CentOS
 
-### Clone github repository
+#### Clone github repository
 
 Open a linux terminal and move to a directory where iSkyLIMS code will be
 downloaded
@@ -87,13 +92,13 @@ git clone https://github.com/BU-ISCIII/iskylims.git iskylims
 cd iskylims
 ```
 
-### Create iskylims database and grant permissions
+#### Create iskylims database and grant permissions
 
 1. Create a new database named "iskylims" (this is mandatory)
 2. Create a new user with permission to read and modify that database.
 3. Write down user, passwd and db server info.
 
-### Configuration settings
+#### Configuration settings
 
 Copy the initial setting template into a file named install_settings.txt
 
@@ -105,11 +110,10 @@ Open with your favourite editor the configuration file to set your own values fo
 database ,email settings and the local IP of the server where iSkyLIMS will run.
 
 ```bash
-
-sudo nano install_settings.txt
+nano install_settings.txt
 ```
 
-### Run installation script
+#### Run installation script
 
 iSkyLIMS should be installed on the "/opt" directory.
 
@@ -133,36 +137,25 @@ bash install.sh --install app
 sudo bash install.sh --install full
 ```
 
-### Configure Apache server
-
-Copy the apache configuration file according to your distribution inside the apache configutation directory and rename it to iskylims.conf
-
-### Installation verification
-
-After installation is completed and apache server is up and running open you navigator typing "localhost" or the "server local IP".
-
-## Upgrade to iSkyLIMS version 3.0.0
+### Upgrade to iSkyLIMS version 3.0.0
 
 If you have already iSkyLIMS on version 2.3.0 you can upgrade to the latest stable version 3.0.0.
 
 Version 3.0.0 is a major release with important upgrades in third parties dependencies like bootstrap. Also, we 've done a huge work on refactoring and variables/function renaming that affects the database. For more details about the changes see the release notes.
 
-### Pre-requisites
+#### Pre-requisites
 
 Because in this upgrade many tables in database are modified it is required that you backup:
 
 - iSkyLIMS database
-- iSkyLIMS document folder
+- iSkyLIMS folder (complete installation folder, p.e /opt/iSkyLIMS)
 
-It is highly recomended that you made these backups and keep them safely in case of
-upgrade failure, to recover your system.
+It is highly recomended that you made these backups and keep them safely in case of upgrade failure, to recover your system.
 
-### Executing the upgrade
+#### Clone github repository
 
 We've also change the way that iSkyLIMS is installed and upgraded. From now on iskylims is downloaded in a user folder and installed elsewhere (p.e /opt/).
 
-### Clone github repository
-
 Open a linux terminal and move to a directory where iSkyLIMS code will be
 downloaded
 
@@ -172,7 +165,7 @@ git clone https://github.com/BU-ISCIII/iSkyLIMS.git iskylims
 cd iskylims
 ```
 
-### Configuration settings
+#### Configuration settings
 
 Copy the initial setting template into a file named install_settings.txt
 
@@ -182,39 +175,96 @@ cp conf/template_install_settings.txt install_settings.txt
 
 Open with your favourite editor the configuration file to set your own values for
 database ,email settings and the local IP of the server where iSkyLIMS will run.
+> If you use a windows-based system for modifying the file, make sure the file is saved using a linux-friendly encoding like ASCII or UTF-8
 
 ```bash
-
 sudo nano install_settings.txt
 ```
 
-### Runing upgrade script
+#### Running upgrade script
 
 If your organization requires that dependencies / stuff that needs root are installed by a different person that install the application the you can use the install script in several steps as follows.
 
 First you need to rename the folder app name in the installation folder (`/opt/iSkyLIMS`):
 
+##### Steps requiring root
+
 ```bash
 # You need root for this operation
-mv /opt/iSkyLIMS /opt/iskylims
+sudo mv /opt/iSkyLIMS /opt/iskylims
 ```
 
 Make sure that the installation folder has the correct permissions so the person installing the app can write in that folder.
 
+```bash
+# In case you have a script for this task. You'll need to adjust this script according to the name changing: /opt/iSkyLIMS to /opt/iskylims
+/scripts/hardening.sh
+```
+
 In the linux terminal execute one of the following command that fit better to you:
 
 ```bash
 # to upgrade only software packages dependences. NEEDS ROOT.
 sudo bash install.sh --upgrade dep
 
+# to install both software. NEEDS ROOT.
+sudo bash install.sh --upgrade full  --ren_app --script drylab_service_state_migration --script rename_app_name --script rename_sample_sheet_folder --script migrate_sample_type --script  migrate_optional_values --tables
+```
+
+##### Steps not requiring root
+
+Next you need to upgrade iskylims app. Please use the command below:
+
+```bash
 # to upgrade only iSkyLIMS application including changes required in this release. DOES NOT NEED ROOT.
 bash install.sh --upgrade app --ren_app --script drylab_service_state_migration --script rename_app_name --script rename_sample_sheet_folder --script migrate_sample_type --script  migrate_optional_values --tables
+```
 
-# to install both software. NEEDS ROOT.
-sudo bash install.sh --upgrade full  --ren_app --script drylab_service_state_migration --script rename_app_name --script rename_sample_sheet_folder --script migrate_sample_type --script  migrate_optional_values --tables
+Make sure that the installation folder has the correct permissions.
+
+```bash
+# In case you have a script for this task. Some paths have changed in this version, so you may need to adjust your hardening script.
+/scripts/hardening.sh
+```
+
+#### What to do if something fails
+
+When we upgrade using the installation script we are performing several changes in the database. If something fails we need to restore the app situation before anything happened and start all over.
+
+We need to copy back the full `/opt/iSkyLIMS` folder back to `/opt` (or your installation path preference), and restore the database doing something like this:
+
+```bash
+sudo rm -rf /opt/iskylims
+sudo cp -r /home/dadmin/backup_prod/iSkyLIMS/ /opt/
+sudo /scripts/hardening.sh
+mysql -u iskylims -h dmysqlps.isciiides.es
+# drop database iskylims;
+# create database iskylims;
+mysql -u iskylims -h dmysqlps.isciiides.es iskylims < /home/dadmin/backup_prod/bk_iSkyLIMS_202310160737.sql
 ```
 
-### Verification installation
+### Final configuration steps
+
+#### SAMBA configurarion
+
+- Login with admin account.
+- Go to Massive sequencing
+![go_to_wetlab](img/got_to_wetlab.png){width:50px}
+- Go to Configuration -> Samba configuration
+- Fill the form with the appropiate params for the samba shared folder:
+![samba form](img/samba_form.png)
+
+#### Email verification
+
+- Go to Massive sequencing
+- Go to Configuration -> Email configuration
+- Fill the form with the needed params for your email configuration and try to send a test email.
+
+#### Configure Apache server
+
+Copy the apache configuration file according to your distribution inside the apache configutation directory and rename it to iskylims.conf
+
+#### Verification of the installation
 
 Open the navigator and type "localhost" or the "server local IP" and check that iSkyLIMs is running.
 
@@ -225,6 +275,6 @@ You can also check some of the functionality, while also checking samba and data
 - Check all tabs so every connectin is successful.
 - Run the 3 tests for each sequencing machine: MiSeq, NextSeq and NovaSeq.
 
-## iSkyLIMS documentation
+### iSkyLIMS documentation
 
 iSkyLIMS documentation is available at [https://iskylims.readthedocs.io/en/latest](https://iskylims.readthedocs.io/en/latest)

conf/first_install_tables.json

@@ -1856,6 +1856,14 @@
         "error_text": "Sample names differ between sample sheet and demultiplexing stats."
     }
 },
+{
+    "model": "wetlab.runerrors",
+    "pk": 34,
+    "fields": {
+        "error_code": "34",
+        "error_text": "Run has been cancelled or ended with errors."
+    }
+},
 {
     "model": "wetlab.runconfigurationtest",
     "pk": 1,

conf/requirements.txt

@@ -1,9 +1,9 @@
-wheel==0.37.1
+wheel==0.38.1
 asn1crypto==1.5.0
 bcrypt==4.0.1
 biopython==1.79
 cryptography==41.0.6
-Django==4.2
+Django==4.2.7
 django-crispy-forms==2.0
 crispy-bootstrap5==0.7
 django-crontab==0.7.1

conf/template_settings.txt

@@ -165,18 +165,19 @@ CRISPY_TEMPLATE_PACK = "bootstrap5"
 # Redirect to home URL after login (Default redirects to /accounts/profile/)
 LOGIN_REDIRECT_URL = "/"
 
-EMAIL_BACKEND = (
-    "django.core.mail.backends.console.EmailBackend"  # During development only
-)
-# EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
+#EMAIL_BACKEND = (
+#    "django.core.mail.backends.console.EmailBackend"  # During development only
+#)
+
+EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
 
 # EMAIL settings
 EMAIL_HOST = "emailhostserver"
 EMAIL_PORT = "emailport"
 EMAIL_HOST_USER = "emailhostuser"
 EMAIL_HOST_PASSWORD = "emailhostpassword"
 EMAIL_USE_TLS = emailhosttls
-ALLOWED_EMAIL_DOMAINS = []
+ALLOWED_EMAIL_DOMAINS = ["isciii.es", "externos.isciii.es"]
 
 LOG_CRONTAB_FILE = os.path.join(BASE_DIR, "logs", "crontab.log")
 LOG_CLEAN_FILE = os.path.join(BASE_DIR, "logs", "crontab_cleanup.log")
@@ -185,10 +186,12 @@ LOG_CLEAN_FILE = os.path.join(BASE_DIR, "logs", "crontab_cleanup.log")
 # Crontab settings
 CRONJOBS = [
     ("*/15 * * * *", "wetlab.cron.looking_for_new_runs", ">>" + LOG_CRONTAB_FILE),
-    ("0 0 1 * *", "wetlab.cron.delete_invalid_run", ">>" + LOG_CLEAN_FILE),
 ]
 
 CRONTAB_COMMAND_SUFFIX = "2>&1"
 
 DEFAULT_AUTO_FIELD = "django.db.models.AutoField"
 DATA_UPLOAD_MAX_MEMORY_SIZE = 7000000
+
+# Needed when using a proxy for https forwading
+SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')