Dockerize

.dockerignore

@@ -0,0 +1,9 @@
+.git
+.github
+.gitignore
+.travis.yml
+mkdocs.yml
+docs
+tests
+Dockerfile
+docker-compose.yml

.github/workflows/workflow-pr.yaml

@@ -5,8 +5,6 @@ jobs:
   qa:
     name: Quality check
     runs-on: ubuntu-18.04
-    env:
-      CGBEACON2_CONFIG: ../instance/config.py
     steps:
       - uses: actions/checkout@v2
       - name: Install missing linux libraries
@@ -24,6 +22,7 @@ jobs:
           pip install pytest
           pytest
         env:
+          CGBEACON2_CONFIG: ../instance/config.py
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       - name: Wemake Python Stylguide
         uses: wemake-services/wemake-python-styleguide@0.13.4

CHANGELOG.md

@@ -1,11 +1,12 @@
+
 ## [] -
 
 ### Fixed
-- Broken githut actions test
+- Broken github actions test
 ### Added
 - Codecov and CodeFactor github actions
 - Created deployment WSGI file
-- Cython requirement
+- Dockerfiles for backend and frontend
 
 
 ## [1.2] - 2020.10.19

README.md

@@ -8,32 +8,60 @@ An updated beacon supporting [ GA4GH API 1.0 ][ga4gh_api1]
 This README only gives a brief overview of the tool, for a more complete reference, please check out our docs: www.clinicalgenomics.se/cgbeacon2
 
 Table of Contents:
-1. [ Prerequisites ](#prerequisites)
-2. [ Installation ](#installation)
+1. [ Running the app using Docker ](#docker)
+2. [ Installation using pip in a virtual environment](#installation)
 3. [ Loading demo data into the database ](#Loading)
 4. [ Server endpoints ](#endpoints)
     - [ Obtaining beacon info ](#info)
     - [ Running queries ](#query)
 5. [ Web interface ](#webform)
 
-<a name="prerequisites"></a>
-## Prerequisites
-- Python 3.6+
-- A working instance of **MongoDB**. From the mongo shell you can create a database using this syntax:
+
+<a name="docker"></a>
+## Installing and running the app using Docker
+
+A demo instance of the app containing test data can be started from the docker-compose file using this command:
 ```
-use cgbeacon2-test
+docker-compose up -d
+```
+The command will create 3 containers:
+- mongodb: starting a mongodb server
+- beacon-cli: the a command-line app, which will connect to the server and populates it with demo data
+- beacon-web: a web server running on localhost and port 27017.
+
+The server will be running and accepting requests sent from outside the containers (another terminal or a web browser). Read further down to find out about request types and queries.
+
+To stop the containers (and the server), run:
+```
+docker-compose down
 ```
 
+To instantiate a server without demo data just remove the line:
+`command: bash -c 'cgbeacon2 add demo'`
+from the docker-compose.yml file.
+
+More info on how to set up a server containing app backend and frontend is available in the [docs](www.clinicalgenomics.se/cgbeacon2/)
+
 <a name="installation"></a>
 ## Installation
 
+### Prerequisites
+Python 3.6+
+- A working instance of **MongoDB**. From the mongo shell you can create a database using this syntax:
+```
+use cgbeacon2-test
+```
+
+It is recommended to install the app inside a virtual environment containing python >3.6
+
 Clone this repository from github using this command:
 ```
 git clone https://github.com/Clinical-Genomics/cgbeacon2.git
 ```
 
 Change directory to the cloned folder and from there, install the software using the following command:
 ```
+pip install -r requirements.txt
 pip install -e .
 ```
 

cgbeacon2/__version__.py

@@ -1 +1 @@
-__version__ = "1.2"
+__version__ = "1.3"

cgbeacon2/server/__init__.py

@@ -41,7 +41,12 @@ def create_app():
         LOG.warning("Please add database settings param in your config file.")
         quit()
 
-    client = MongoClient(app.config["DB_URI"])
+    # If app is runned from inside a container, override host port
+    db_uri = app.config["DB_URI"]
+    if os.getenv("MONGODB_HOST"):
+        db_uri=f"mongodb://{os.getenv('MONGODB_HOST')}:{'27017'}/{'cgbeacon2'}"
+
+    client = MongoClient(db_uri)
     app.db = client[app.config["DB_NAME"]]
     LOG.info("database connection info:{}".format(app.db))
 

docker-compose.yml

@@ -0,0 +1,34 @@
+version: '3'
+# usage:
+# sudo docker-compose build
+# sudo docker-compose up
+services:
+  mongodb:
+    image: mvertes/alpine-mongo
+    container_name: mongodb
+    ports:
+      - '27017:27017'
+    expose:
+      - '27017'
+
+  beacon-cli:
+    environment:
+      MONGODB_HOST: mongodb
+    image: northwestwitch/cgbeacon2
+    container_name: beacon-cli
+    links:
+      - mongodb
+    command: bash -c 'cgbeacon2 add demo'
+
+  beacon-web:
+    environment:
+      MONGODB_HOST: mongodb
+    image: northwestwitch/cgbeacon2
+    container_name: beacon-web
+    links:
+      - mongodb
+    expose:
+      - '5000'
+    ports:
+      - '5000:5000'
+    command: bash -c 'cgbeacon2 run --host 0.0.0.0'

docs/docker_run.md

@@ -0,0 +1,120 @@
+## Setting up a server from a Docker image
+
+The beacon app is consisting of a **backend** that is used for:
+- Creating and removing datasets
+- Adding or removing variants for one of more samples of a dataset
+- Loading demo data (not used in production, just in a test server)
+- Updating the genes in the database
+
+At the same time, the command `cgbeacon run` starts a **frontend** server with the following API endpoints:
+```
+Endpoint           Methods    Rule
+-----------------  ---------  ------------------------------
+api_v1.add         POST       /apiv1.0/add
+api_v1.delete      POST       /apiv1.0/delete
+api_v1.info        GET        /apiv1.0/
+api_v1.query       GET, POST  /apiv1.0/query
+api_v1.query_form  GET, POST  /apiv1.0/query_form
+```
+
+A Docker image for creating both backend and frontend containers is available on [Docker Hub](https://hub.docker.com/repository/docker/northwestwitch/cgbeacon2).
+Alternatively the Dockerfile used for creating the image is available in this repositiory, under containers/base/Dockerfile.
+
+A local image of the repository can be created by moving the Dockerfile in the root folder of the app and from the same location, in a terminal, running the following command:
+
+```
+docker build -t cgbeacon2 .
+```
+
+The container with the docker image contains only the beacon app and its required libraries. In order to work the container must be connected with at least one other container hosting a running mongodb instance.
+
+
+## Setting up the app backend:
+
+A simple running instance of the app backend connected to the database and ready to execute commands could be created in different ways. This is an example using docker-compose:
+
+Create a file docker-compose.yml containing the following code:
+
+```
+version: '3'
+# usage:
+# sudo docker-compose build
+# sudo docker-compose up
+services:
+  mongodb:
+    image: mvertes/alpine-mongo
+    container_name: mongodb
+    ports:
+      - '27017:27017'
+    expose:
+      - '27017'
+
+  beacon-cli:
+    environment:
+      MONGODB_HOST: mongodb
+    image: northwestwitch/cgbeacon2
+    container_name: beacon-cli
+    links:
+      - mongodb
+    stdin_open: true # docker run -i
+    tty: true        # docker run -t
+```
+
+Run the containers and open an interactive shell for the backend by typing:
+```
+docker-compose run beacon-cli /bin/bash
+```
+
+To populate the database with demo data (dataset + case variants) type:
+```
+cgbeacon2 add demo
+```
+
+Exit from the execution of the images by typing `exit`
+
+## Starting an app server connected to the database
+
+An app server instance connected to the server might be started in a similar way using Docker Compose. This is an example of a such server, listening for incoming requests on port 5000, from hosts outside the container.
+
+Example of docker-compose.yml file:
+
+```
+version: '3'
+
+services:
+  mongodb:
+    image: mvertes/alpine-mongo
+    container_name: mongodb
+    ports:
+      - '27017:27017'
+    expose:
+      - '27017'
+
+  beacon-web:
+    environment:
+      MONGODB_HOST: mongodb
+    image: northwestwitch/cgbeacon2
+    container_name: beacon-web
+    links:
+      - mongodb
+    expose:
+      - '5000'
+    ports:
+      - '5000:5000'
+    command: bash -c 'cgbeacon2 run --host 0.0.0.0'
+```
+
+Run the server as a service (detached mode) by typing
+```
+docker.compose up -d
+```
+
+The server should be now listing for requests. Test that it is working by sending a request to the beacon info endpoint from another terminal window:
+```
+curl -X GET 'http://127.0.0.1:5000/apiv1.0/'
+```
+
+Stop the server by typing:
+```
+docker.compose down
+```

docs/install.md → docs/env_install.md

@@ -2,27 +2,25 @@
 ## Beacon setup
 
 1. [ Prerequisites ](#prerequisites)
-1. [ Installation ](#installation)
+1. [ Installation - on a virtual environment using pip](#installation-pip)
 1. [ Server settings ](#settings)
 1. [ Running the server](#running)
 
 
-<a name="prerequisites"></a>
-## Prerequisites
-- Python 3.6+ installed
+<a name="installation-pip"></a>
+
+### Prerequisites
+- A virtual environment containing Python 3.6+
 - A working instance of **MongoDB**. From the mongo shell you can create a database using this syntax:
 ```
 use <name_of_database>
 ```
 
 Database name can be customized. If you don't have any preferences`cgbeacon2` will work just fine.
 
-
-
-<a name="installation"></a>
 ## Installation
 
-Clone this repository from github using this command:
+Once activated the virtual environment, clone this repository from github using this command:
 ```
 git clone https://github.com/Clinical-Genomics/cgbeacon2.git
 ```

mkdocs.yml

@@ -2,7 +2,8 @@ site_name: cgbeacon2 docs
 
 nav:
     - Home: index.md
-    - Installation: install.md
+    - Running the app in Docker: docker_run.md
+    - Installation on a virtual environment: env-install.md
     - Loading datasets and variants: loading.md
     - Removing datasets and variants: removing.md
     - Queries: queries.md

requirements.txt

@@ -18,6 +18,4 @@ PyJWT
 # utils
 cyvcf2
 pybedtools
-pysam<0.16 #Avoid ImportError: libchtslib.cpython-35m-x86_64-linux-gnu.so
 jsonschema
-cython

setup.py

@@ -22,29 +22,6 @@
 
 here = os.path.abspath(os.path.dirname(__file__))
 
-def parse_reqs(req_path='./requirements.txt'):
-    """Recursively parse requirements from nested pip files."""
-    install_requires = []
-    with io.open(os.path.join(here, 'requirements.txt'), encoding='utf-8') as handle:
-        # remove comments and empty lines
-        lines = (line.strip() for line in handle
-                 if line.strip() and not line.startswith('#'))
-
-        for line in lines:
-            # check for nested requirements files
-            if line.startswith('-r'):
-                # recursively call this function
-                install_requires += parse_reqs(req_path=line[3:])
-
-            else:
-                # add the line as a new requirement
-                install_requires.append(line)
-
-    return install_requires
-
-# What packages are required for this module to be executed?
-REQUIRED = parse_reqs()
-
 # The rest you shouldn't have to touch too much :)
 # ------------------------------------------------
 # Except, perhaps the License and Trove Classifiers!
@@ -106,7 +83,6 @@ def run(self):
     download_url = '/'.join([URL,'tarball',version]),
     keywords = KEYWORDS,
     packages=find_packages(),
-    install_requires=REQUIRED,
     include_package_data=True,
     license=LICENSE,
     classifiers=[

tests/server/test_init.py

@@ -6,5 +6,22 @@
 def test_create_app_from_envar(monkeypatch):
     """Test option to create app from a file specified by an environment variable"""
 
+    # GIVEN a a config file defined in the environment
     monkeypatch.setenv("CGBEACON2_CONFIG", config_file_path, prepend=False)
-    assert create_app()
+    # THEN the app should connect to a database on localhost, port 27017 as defined on config file
+    app = create_app()
+    assert app
+    db_attrs = str(vars(app.db))  # convert database attributes to string
+    assert "host=['127.0.0.1:27017']" in db_attrs
+
+
+def test_create_app_in_container(monkeypatch):
+    """Test creating app from inside a container, when an env varianble named 'MONGODB_HOST' ovverides the host provided in config file"""
+
+    # GIVEN an env var named MONGODB_HOST
+    monkeypatch.setenv("MONGODB_HOST", "mongodb")
+    # THEN the app should connect to a mongo host named mongodb on port 27017
+    app = create_app()
+    assert app
+    db_attrs = str(vars(app.db))  # convert database attributes to string
+    assert "host=['mongodb:27017']" in db_attrs