Create a Dockerfile for serving the app via gunicorn - with GitHub ac…

…tion (#3053)

* Created a dedicated Dockerfile for running a prod server

* modify docker compose to run server from gunicorn Dockerfile

* add action to push to docker hub the server dockerfile (stage)

* updated changelog

* Push the prod server image to Docker Hub whenever a new release is created

* fix port on readme file

* Fixed documentation and updated changelog

* Set hide alamut link to True

* Fix a test accordimng to modified demo config file

* Update docs/admin-guide/server.md

Co-authored-by: Daniel Nilsson <daniel.k.nilsson@gmail.com>

Co-authored-by: Daniel Nilsson <daniel.k.nilsson@gmail.com>

.github/workflows/build_and_publish.yml

@@ -47,14 +47,23 @@ jobs:
     - name: Check out git repository
       uses: actions/checkout@v2
 
-    - name: Build and publish to Docker Hub
+    - name: Publish main image (Dockerfile) to Registry
       uses: elgohr/Publish-Docker-Github-Action@master
       with:
         name: clinicalgenomics/scout
         username: ${{ secrets.DOCKER_USERNAME }}
         password: ${{ secrets.DOCKER_PASSWORD }}
         tags: "latest,${{ github.event.release.tag_name }}"
 
+    - name: Publish server image (Dockerfile-server) to Registry
+      uses: elgohr/Publish-Docker-Github-Action@master
+      with:
+        name: clinicalgenomics/scout-server
+        username: ${{ secrets.DOCKER_USERNAME }}
+        password: ${{ secrets.DOCKER_PASSWORD }}
+        dockerfile: Dockerfile-server
+        tags: "latest,${{ github.event.release.tag_name }}"
+
  deploy-docs:
    name: Deploy Docs to GitHubIO
    runs-on: ubuntu-latest
@@ -69,4 +78,3 @@ jobs:
       run: pip install mkdocs mkdocs-material markdown-include
     - name: Deploy documentation
       run: mkdocs gh-deploy --force
-

.github/workflows/server_stage_docker_push.yml

@@ -0,0 +1,37 @@
+name: Publish to Docker stage
+
+on:
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  docker-stage-push:
+    name: Create staging docker image
+    runs-on: ubuntu-latest
+    steps:
+     - name: Check out git repository
+       uses: actions/checkout@v2
+
+     - name: Get branch name
+       id: branch-name
+       uses: tj-actions/branch-names@v5
+
+     - name: Login to Docker Hub
+       uses: docker/login-action@v1
+       with:
+         username: ${{ secrets.DOCKER_USERNAME }}
+         password: ${{ secrets.DOCKER_PASSWORD }}
+
+     - name: Set up Docker Buildx
+       id: buildx
+       uses: docker/setup-buildx-action@v1
+
+     - name: Build and push
+       if: steps.branch-name.outputs.is_default == 'false'
+       uses: docker/build-push-action@v2
+       with:
+         context: ./
+         file: ./Dockerfile-server
+         push: true
+         tags: "clinicalgenomics/scout-server-stage:${{steps.branch-name.outputs.current_branch}}, clinicalgenomics/scout-server-stage:latest"

CHANGELOG.md

@@ -7,7 +7,12 @@ About changelog [here](https://keepachangelog.com/en/1.0.0/)
 
 ## []
 ### Added
+- Created a Dockefile to be used to serve the dockerized app in production
+- Modified the code to collect database params specified as env vars
+- Created a GitHub action that pushes the Dockerfile-server image to Docker Hub (scout-server-stage) every time a PR is opened
+- Created a GitHub action that pushes the Dockerfile-server image to Docker Hub (scout-server) every time a new release is created
 ### Changed
+- Updated the python config file documentation in admin guide
 ### Fixed
 
 

Dockerfile

@@ -9,7 +9,7 @@ WORKDIR /app
 
 # Install Scout dependencies
 COPY requirements.txt .
-RUN pip install --no-cache-dir -r requirements.txt gunicorn
+RUN pip install --no-cache-dir -r requirements.txt
 
 
 #########
@@ -35,7 +35,7 @@ ENV PATH="/venv/bin:$PATH"
 RUN echo export PATH="/venv/bin:\$PATH" > /etc/profile.d/venv.sh
 
 # Create a non-root user to run commands
-RUN groupadd --gid 10001 worker && useradd -g worker --uid 10001 --shell /usr/sbin/nologin --create-home worker
+RUN groupadd --gid 1000 worker && useradd -g worker --uid 1000 --shell /usr/sbin/nologin --create-home worker
 
 # Copy virtual environment from builder
 COPY --chown=worker:worker --from=python-builder /venv /venv

Dockerfile-server

@@ -0,0 +1,67 @@
+###########
+# BUILDER #
+###########
+FROM clinicalgenomics/python3.8-cyvcf2-venv:1.0 AS python-builder
+
+ENV PATH="/venv/bin:$PATH"
+
+WORKDIR /app
+
+# Install Scout dependencies
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt gunicorn
+
+#########
+# FINAL #
+#########
+FROM python:3.8-slim
+
+LABEL about.home="https://github.com/Clinical-Genomics/scout"
+LABEL about.documentation="https://clinical-genomics.github.io/scout"
+LABEL about.tags="WGS,WES,Rare diseases,VCF,variants,SNP,Next generation sequencing"
+LABEL about.license="MIT License (MIT)"
+
+# Install base dependencies
+RUN apt-get update && \
+     apt-get -y upgrade && \
+     apt-get -y install -y --no-install-recommends libpango-1.0-0 libpangocairo-1.0-0 && \
+     apt-get clean && \
+     rm -rf /var/lib/apt/lists/*
+
+# Do not upgrade to the latest pip version to ensure more reproducible builds
+ENV PIP_DISABLE_PIP_VERSION_CHECK=1
+ENV PATH="/venv/bin:$PATH"
+RUN echo export PATH="/venv/bin:\$PATH" > /etc/profile.d/venv.sh
+
+# Create a non-root user to run commands
+RUN groupadd --gid 1000 worker && useradd -g worker --uid 1000 --shell /usr/sbin/nologin --create-home worker
+
+# Copy virtual environment from builder
+COPY --chown=worker:worker --from=python-builder /venv /venv
+
+WORKDIR /home/worker/app
+COPY . /home/worker/app
+
+# Install only Scout app
+RUN pip install --no-cache-dir --editable .[coverage]
+
+# Run the app as non-root user
+USER worker
+
+ENV GUNICORN_WORKERS=1
+ENV GUNICORN_THREADS=1
+ENV GUNICORN_BIND="0.0.0.0:8000"
+ENV GUNICORN_TIMEOUT=400
+
+CMD gunicorn \
+    --workers=$GUNICORN_WORKERS \
+    --bind=$GUNICORN_BIND  \
+    --threads=$GUNICORN_THREADS \
+    --timeout=$GUNICORN_TIMEOUT \
+    --proxy-protocol \
+    --forwarded-allow-ips="10.0.2.100,127.0.0.1" \
+    --log-syslog \
+    --access-logfile - \
+    --error-logfile - \
+    --log-level="debug" \
+    scout.server.auto:app

README.md

@@ -36,9 +36,9 @@ The repository includes a Makefile with common shortcuts to simplify setting up
 This demo is consisting of 3 containers:
 - a MongoDB instance, on the default port 27017 in the container, mapped to host port 27013
 - scout-cli --> the Scout command line, connected to the database. Populates the database with demo data
-- scout-web --> the Scout web app, that serves the app on localhost, port 5000.
+- scout-web --> the Scout web app, that serves the app on localhost, port 8000.
 
-Once the server has started you and open the app in the web browser at the following address: http://localhost:5000/
+Once the server has started you and open the app in the web browser at the following address: http://localhost:8000/
 
 The command to stop the demo are either `docker-compose down` or `make down`.
 

docker-compose.yml

@@ -28,13 +28,21 @@ services:
       - mongodb
 
   scout-web:
-    build: .
-    container_name: scout-web
+    build:
+      context: .
+      dockerfile: Dockerfile-server
     expose:
-      - '5000'
+      - '8000'
     ports:
-      - '5000:5000'
-    command: scout --host mongodb --demo serve --host 0.0.0.0
+      - '8000:8000'
+    environment:
+      SCOUT_CONFIG: /home/worker/app/scout/server/config.py
+      MONGO_HOST: mongodb
+      MONGO_DBNAME: scout-demo
+      GUNICORN_WORKERS: 1
+      GUNICORN_TREADS: 1
+      GUNICORN_BIND: 0.0.0.0:8000
+      GUNICORN_TIMEOUT: 400
     volumes:
       - ./scout:/home/worker/app/scout
     networks:

docs/admin-guide/server.md

@@ -4,78 +4,111 @@ This document will explain how we like to run Scout in both development and prod
 
 ## Config
 
-The server has it's own config file, separate from the command line tool. It's written in Python for full expressivity. These are the settings you can use:
+In order to run the web server, it is usually necessary to provide a number of parameters. All available parameters should be specified in a python-formatted configuration file. This config file is passed as a parameter when the application is launched. The following example of a config file is is used for running the development server (read further down):
 
 ```python
-# flask.conf.py
+SECRET_KEY = "this is not secret..."
+REMEMBER_COOKIE_NAME = "scout_remember_me"
 
-# to encrypt cookie data
-SECRET_KEY = 'makeThisSomethingNonGuessable'  # required
+# MONGO_URI = "mongodb://127.0.0.1:27011,127.0.0.1:27012,127.0.0.1:27013/?replicaSet=rs0&readPreference=primary"
+MONGO_DBNAME = "scout"
 
-# connection details for MongoDB
-MONGO_DBNAME = 'scout'                        # required
-MONGO_PORT = 27017
-MONGO_USERNAME = 'mongoUser'
-MONGO_PASSWORD = 'mongoUserPassword'
+BOOTSTRAP_SERVE_LOCAL = True
+TEMPLATES_AUTO_RELOAD = True
+
+DEBUG_TB_INTERCEPT_REDIRECTS = False
+
+# Flask-mail: http://pythonhosted.org/flask-mail/
+# see: https://bitbucket.org/danjac/flask-mail/issue/3
+MAIL_SERVER = "smtp.gmail.com"
+MAIL_PORT = 587
+MAIL_USE_TLS = True
+MAIL_USE_SSL = False
 
 # Filename of accrediation bagde image in server/bluprints/public/static
-# If not set no badge is displayed in scout
+# If null no badge is displayed in scout
 ACCREDITATION_BADGE = "swedac-1926-iso17025.png"
 
-# connection string for Chanjo coverage database
-SQLALCHEMY_DATABASE_URI = 'mysql://chanjo:CHANJO_PASSWORD@localhost:3306/chanjo'
+# Parameters required for Google Oauth 2.0 login
+# GOOGLE = dict(
+#    client_id="client.apps.googleusercontent.com",
+#    client_secret="secret",
+#    discovery_url="https://accounts.google.com/.well-known/openid-configuration",
+# )
 
-# connection details for LoqusDB MongoDB database
-LOQUSDB_SETTINGS = dict(
-    database='loqusdb',
-    uri=("mongodb://{}:{}@localhost:{}/loqusdb"
-         .format(MONGO_USERNAME, MONGO_PASSWORD, MONGO_PORT)),
-)
+# Chanjo database connection string - used by chanjo report to create coverage reports
+# SQLALCHEMY_DATABASE_URI = "mysql+pymysql://test_user:test_passwordw@127.0.0.1:3306/chanjo"
+
+# Configure gens service
+# GENS_HOST = "127.0.0.1"
+# GENS_PORT = 5000
 
-# default chanjo coverage report language: 'sv' or 'en'
-REPORT_LANGUAGE = 'sv'
+# Connection details for communicating with a rerunner service
+# RERUNNER_API_ENTRYPOINT = "http://rerunner:5001/v1.0/rerun"
+# RERUNNER_TIMEOUT = 10
+# RERUNNER_API_KEY = "I am the Keymaster of Gozer"
 
-# Google email account to user for sending emails
-MAIL_USERNAME = 'paul.anderson@gmail.com'
-MAIL_PASSWORD = 'mySecretPassw0rd'
+# MatchMaker connection parameters
+# - Tested with PatientMatcher (https://github.com/Clinical-Genomics/patientMatcher) -
+# MME_ACCEPTS = "application/vnd.ga4gh.matchmaker.v1.0+json"
+# MME_URL = "http://localhost:9020"
+# MME_TOKEN = "DEMO"
 
-TICKET_SYSTEM_EMAIL = 'support@sekvens.nu'
+# Beacon connection settings
+# - Tested with cgbeacon2 (https://github.com/Clinical-Genomics/cgbeacon2) -
+# BEACON_URL = "http://localhost:6000/apiv1.0"
+# BEACON_TOKEN = "DEMO"
 
-## FEATURE FLAGS
-# Allow users to browse variants marked causative from among all their institutes cases
+# connection details for LoqusDB MongoDB database
+# Example with 2 instances of LoqusDB, one using a binary file and one instance connected via REST API
+# When multiple instances are available, admin users can modify which one is in use for a given institute from the admin settings page
+# LOQUSDB_SETTINGS = {
+#    "default" : {"binary_path": "/miniconda3/envs/loqus2.5/bin/loqusdb", "config_path": "/home/user/settings/loqus_default.yaml"},
+#    "loqus_api" : {"api_url": "http://127.0.0.1:9000"},
+# }
+
+#
+# Cloud IGV tracks can be configured here to allow users to enable them on their IGV views.
+# CLOUD_IGV_TRACKS = [
+#    {
+#        "name": "custom_public_bucket",
+#        "access": "public",
+#        "tracks": [
+#            {
+#                "name": "dbVar Pathogenic or Likely Pathogenic",
+#                "type": "variant",
+#                "format": "vcf",
+#                "build": "37",
+#                "url": "https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/25777457/GRCh37.variant_call.clinical.pathogenic_or_likely_pathogenic.vcf.gz",
+#                "indexURL": "https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/25777460/GRCh37.variant_call.clinical.pathogenic_or_likely_pathogenic.vcf.gz.tbi",
+#            }
+#        ],
+#    },
+# ]
+
+# Chanjo-Report
+REPORT_LANGUAGE = "en"
+ACCEPT_LANGUAGES = ["en", "sv"]
+
+# TICKET_SYSTEM_EMAIL = "support@test_service.com"
+
+# FEATURE FLAGS
 SHOW_CAUSATIVES = True
-# Optionally show cards with legacy observed variant frequencies on variant pages
 SHOW_OBSERVED_VARIANT_ARCHIVE = True
-# Optionally hide API link to Alamut Visual. If False or non-existing Scout will show Alamut links.
-HIDE_ALAMUT_LINK = False
-
-
-# emails to send error log message to
-ADMINS = ['robin.andeer@scilifelab.se']
-
-# enable Google OAuth-based login
-GOOGLE = dict(
-    consumer_key='CLIENT_ID.apps.googleusercontent.com',
-    consumer_secret='CLIENT_SECRET',
-    # Prepend to all (non-absolute) request URLs
-    base_url='https://www.googleapis.com/oauth2/v1/',
-    authorize_url='https://accounts.google.com/o/oauth2/auth',
-    request_token_url=None,
-    request_token_params={
-        'scope': ("https://www.googleapis.com/auth/userinfo.profile "
-                  "https://www.googleapis.com/auth/userinfo.email"),
-    },
-    access_token_url='https://accounts.google.com/o/oauth2/token',
-    access_token_method='POST'
-)
-
-# enable the phenomizer service which links HPO phenotype terms to diseases/genes
-PHENOMIZER_USERNAME = 'phenoUser'
-PHENOMIZER_PASSWORD = 'phenoPassword'
-
-# Add a primer order link to complementary method verification emails - customise per genome build
-EXTERNAL_PRIMER_ORDER_LINK_37 = "https://www.primercompany.com/order?chr={chromosome}&pos={position}&build=37"
-EXTERNAL_PRIMER_ORDER_LINK_38 = "https://www.primercompany.com/order?chr={chromosome}&pos={position}"
+HIDE_ALAMUT_LINK = True
+
+# OMIM API KEY: Required for downloading definitions from OMIM (https://www.omim.org/api)
+# OMIM_API_KEY = 'valid_omim_api_key'
+
+# Parameters for enabling Phenomizer queries
+# PHENOMIZER_USERNAME = "test_user"
+# PHENOMIZER_PASSWORD = "test_password"
+
+# Rank model links
+RANK_MODEL_LINK_PREFIX = "https://github.com/Clinical-Genomics/reference-files/blob/master/rare-disease/rank_model/rank_model_-v"
+RANK_MODEL_LINK_POSTFIX = "-.ini"
+SV_RANK_MODEL_LINK_PREFIX = "https://github.com/Clinical-Genomics/reference-files/blob/master/rare-disease/rank_model/svrank_model_-v"
+SV_RANK_MODEL_LINK_POSTFIX = "-.ini"
 ```
 ### Minimal config
 
@@ -106,7 +139,7 @@ FLASK_DEBUG=1 \
 SCOUT_CONFIG="/full/path/to/flask.conf.py" \
 flask run
 ```
-The server can also be started from the scout cli with the command `scout serve`. There are multiple options here. To use a alternative flask config pass the path with option `-f/ --flask_config` like `scout -f path/to/flask.conf.py serve`.
+The server can also be started from the scout cli with the command `scout serve`. The default config file that will be used when no other configuration option is provided will be the one present under scout/server/config.py. To use an alternative flask config, you can use the `-f/ --flask_config` option. Example: `scout -f path/to/flask.conf.py serve`.
 
 
 ## Production
@@ -123,6 +156,8 @@ gunicorn \
     scout.server.auto:app
 ```
 
+Note that while it might still be necessary to provide a `SCOUT_CONFIG` environment variable pointing to python config file to run a full web server, some parameters like those used to connect to MongoDB (`MONGO_HOST`, `MONGO_DBNAME`, `MONGO_PORT`, `MONGO_USERNAME`, `MONGO_PASSWORD`, `MONGO_URI`) can be also be passed as environment variables. In that case they will have precedence over those provided in the config file.
+
 If you are running a larger environment, where this is one component, we encourage a reverse proxy configuration where Scout is served by Gunicorn, and reverse proxied by [NGINX](https://nginx.org/en/). Then NGINX will handle the secure communication.
 
 [gunicorn]: http://gunicorn.org/