🐳📺🎨 Automappa web app with server upload and background pre-/post-processing services

.dockerignore

@@ -0,0 +1,6 @@
+data
+test
+Automappa.egg-info
+build
+dist
+.vscode

.env

@@ -0,0 +1,27 @@
+# Required for docker-compose.yml
+SERVER_ROOT_UPLOAD_FOLDER="${HOME}/.automappa/uploads"
+SERVER_HOST="0.0.0.0"
+SERVER_PORT=8050
+# By, default remove debugging tools --> For development switch this to `True`
+SERVER_DEBUG=False
+POSTGRES_USER="admin"
+POSTGRES_PASSWORD="mypass"
+POSTGRES_DB="automappa"
+POSTGRES_URL="postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}"
+POSTGRES_POOL_SIZE=1
+POSTGRES_POOL_PRE_PING=False
+RABBITMQ_DEFAULT_USER="user"
+RABBITMQ_DEFAULT_PASS="pass"
+RABBITMQ_URL="amqp://${RABBITMQ_DEFAULT_USER}:${RABBITMQ_DEFAULT_PASS}@rabbitmq:5672/"
+CELERY_BACKEND_URL='redis://redis:6379/0'
+CELERY_BROKER_URL="amqp://${RABBITMQ_DEFAULT_USER}:${RABBITMQ_DEFAULT_PASS}@rabbitmq:5672//"
+FLOWER_BROKER_API_URL="http://${RABBITMQ_DEFAULT_USER}:${RABBITMQ_DEFAULT_PASS}@rabbitmq:15672/api"
+# https://github.com/mher/flower/issues/1036
+
+# Grafana configuration
+# Allow anonymous authentication or not
+GF_AUTH_DISABLE_LOGIN_FORM="false"
+# Role of anonymous user
+GF_AUTH_ANONYMOUS_ENABLED="false"
+# Install plugins here our in your own config file
+GF_AUTH_ANONYMOUS_ORG_ROLE="Admin"

.gitignore

@@ -8,4 +8,7 @@ app/__pycache__
 .DS_Store
 dist
 build
-Automappa.egg-info
+Automappa.egg-info
+.env
+data
+db-data

Dockerfile

@@ -1,15 +1,20 @@
 FROM condaforge/miniforge3:latest
 
+RUN conda install --prune --name base mamba --yes
+
 COPY environment.yml ./environment.yml
 
-RUN conda env update -n base -f=environment.yml \
-    && conda clean --all --force-pkgs-dirs --yes
+RUN mamba env update --name base --file=./environment.yml \
+    && mamba clean --all --force-pkgs-dirs --yes
 
-# Test command is functional
-COPY . /Automappa/
-WORKDIR /Automappa/
+COPY . /usr/src/app
+WORKDIR /usr/src/app
 RUN python -m pip install . --ignore-installed --no-deps -vvv
+# Test command is functional
 RUN automappa -h
 
-CMD [ "-h" ]
-ENTRYPOINT [ "automappa" ]
+# Create an unprivileged user for running our Python code.
+RUN adduser --disabled-password --gecos '' automappa
+
+# CMD [ "-h" ]
+# ENTRYPOINT [ "automappa" ]

Makefile

@@ -49,6 +49,22 @@ image: Dockerfile
 install: 
 	$(PYTHON_INTERPRETER) -m pip install . --ignore-installed --no-deps -vvv
 
+## docker compose build from docker-compose.yml
+build: docker-compose.yml
+	docker-compose build
+
+## alias for docker-compose up --always-recreate-deps --remove-orphans --force-recreate
+up: docker-compose.yml
+	docker-compose up --always-recreate-deps --remove-orphans --force-recreate
+
+## alias for docker-compose down --remove-orphans
+down: docker-compose.yml
+	docker-compose down --remove-orphans
+
+## alias for docker-compose down --remove-orphans --volumes
+down-v: docker-compose.yml
+	docker-compose down --remove-orphans -v
+
 # Run Automappa on test data
 # test: test_data
 # 	$(PYTHON_INTERPRETER) index.py -i test/bins.tsv
@@ -64,13 +80,13 @@ test_environment: scripts/test_environment.py
 	$(PYTHON_INTERPRETER) $<
 
 ## Set up python interpreter environment
-create_environment: requirements.txt
+create_environment: environment.yml
 ifeq (True,$(HAS_CONDA))
 	@echo ">>> Detected conda, creating conda environment."
 ifeq (3,$(findstring 3,$(PYTHON_INTERPRETER)))
-	conda create -c conda-forge --name $(PROJECT_NAME) python=3.7 --file=$<
+	mamba env create --name $(PROJECT_NAME) --file=$<
 else
-	conda create -c conda-forge --name $(PROJECT_NAME) python=3.7 --file=$<
+	mamba env create --name $(PROJECT_NAME) --file=$<
 endif
 	@echo ">>> New conda env created. Activate with:\nsource activate $(PROJECT_NAME)"
 else

README.md

@@ -14,267 +14,34 @@
 
 ## Getting Started
 
-- [Install with conda](#install-with-conda)
-- [Run `automappa` using docker](#quickstart-using-docker-no-installation-required)
-- [Install from source](#install-from-source)
-- [Advanced Usage](#advanced-usage)
-  - [A breakdown of the docker run wrapper script](#full-docker-run-command-example)
-  - [Using a remote Automappa server](#using-a-remote-automappa-server)
-  - [Using a remote docker container Automappa server](#using-a-remote-docker-container-automappa-server)
+- [Clone the Automappa Repo](#clone-the-repository)
+- [Run `make build` using Makefile](#build-images-for-automappa-services)
+- [Run `make up` using Makefile](#build-and-run-automappa-services)
+- [Open the Automappa url](#navigate-to-automappa-page)
+## Clone the repository
 
-## Install with conda
+## Automappa testing setup/run commands
 
-If you are using `conda` (or `mamba`) as a package manager, you can simply install `automappa` using one of the following one-liners.
-
-### with `conda`
-
-```bash
-conda install -c bioconda automappa
-```
-
-### with `mamba`
-
-```bash
-mamba install -c bioconda automappa
-```
-
-After you have installed `automappa`, you can simply run `automappa -h` to see a list of available arguments.
-
-To start the `automappa` app, you must specify your main binning results and respective kingdom's single-copy marker annotations 
-generated from an [Autometa analysis](https://www.github.com/KwanLab/Autometa). If you do not yet have these annotations and are 
-not sure where to start, I would recommend checking out [Autometa's documentation](https://autometa.readthedocs.io/en/latest/)
-
-### Example `automappa` command
-
-```bash
-automappa --binning-main </path/to/bacteria.binning.main.tsv> --markers </path/to/bacteria.markers.tsv>
-```
-
-## Quickstart using Docker (No installation required)
-
- To quickly start exploring your data, run the app using a wrapper script that will run the docker image, `evanrees/automappa:latest`, ([available from Dockerhub](https://cloud.docker.com/repository/docker/evanrees/automappa/tags "Automappa Dockerhub Tags")). Now you can skip installation and start binning, examining and describing! Let the microbial exegesis begin!
-
-### Running with a docker container using `run_automappa.sh`
-
-A docker wrapper is available to run a docker container of `Automappa`.
-The only required input for this script is the autometa main binning output table and the respective markers table.
-
-```bash
-# First retrieve the script:
-curl -o run_automappa.sh https://raw.githubusercontent.com/WiscEvan/Automappa/main/docker/run_automappa.sh
-# (make it executable)
-chmod a+x run_automappa.sh
-```
-
-Now run automappa on autometa binning results using the downloaded script: `run_automappa.sh`.
-
-### Start automappa docker container
-
-***NOTE: This will pull the automappa docker image if it is not already available***
-
-```bash
-./run_automappa.sh --binning binning.main.tsv --markers binning.markers.tsv
-```
-
-----------------------------------------------------------------------------------------------------
-
-## Install from source
-
-### Installation from source (using `make`)
-
-You can install all of Automappa's dependencies using the Makefile found within the repository.
-
-#### Clone the Automappa repository
-
-```bash
-cd $HOME
-git clone https://github.com/WiscEvan/Automappa.git
-cd $HOME/Automappa
-```
-
-#### First create environment
-
-```bash
-make create_environment
-```
-
-#### Activate environment
-
-```bash
-source activate automappa
-```
-
-#### The following will install the automappa entrypoint
-
-```bash
-make install
-```
-
-Now that all of the dependencies are installed, you may run the app on your local machine or on a server.
-
-
-### Listing available `make` commands
-
-You may also list other available make commands by simply typing `make` with no other arguments.
-
-```bash
-make
-```
-
-A few examples:
-
-#### pull docker image
-
-```bash
-make docker
-```
-
-#### build docker image
-
-```bash
-make image
-```
-
-## Usage
-
-Simply provide the `automappa` entrypoint with the main binning file output by Autometa as well as the respective markers file.
-
-```bash
-automappa \
-    --binning-main <path to binning.main.tsv> \
-    --markers <path to binning.markers.tsv>
-```
-
-----------------------------------------------------------------------------------------------------
-
-## Advanced Usage
-
-### Full `docker run` command example
-
-```bash
-# Set automappa parameters (required)
-binning="$HOME/test/binning.main.tsv"
-markers="$HOME/test/binning.markers.tsv"
-
-# Set docker image/container parameters (optional)
-localport=8050
-containerport=8886
-imagetag="latest"
-
-#NOTE: Some necessary path handling here for binding docker volumes
-binning_dirname="$( cd -- "$(dirname "$binning")" >/dev/null 2>&1 ; pwd -P )"
-binning_filename=$(basename $binning)
-markers_dirname="$( cd -- "$(dirname "$markers")" >/dev/null 2>&1 ; pwd -P )"
-markers_filename=$(basename $markers)
-
-# Run with provided parameters
-docker run \
-    --publish $localport:$containerport \
-    --detach=false \
-    -v $binning_dirname:/binning:rw \
-    -v $markers_dirname:/markers:ro \
-    --rm \
-    evanrees/automappa:$imagetag \
-      --binning-main /binning/$binning_filename \
-      --markers /markers/$markers_filename \
-      --port $containerport \
-      --host 0.0.0.0
-```
-
-## Using a remote Automappa server
-
-If you'd like to run Automappa on a *remote* server but view the output on your *local* machine,
-
-### Example remote server login with ssh tunnel
-
-you first need to login to the remote server with a tunnel, e.g. `ssh -L localport:localhost:serverport user@hostaddress`.
-
-```bash
-#ssh -L localport:127.0.0.1:serverport user@kwan-bioinformatics.pharmacy.wisc.edu
-#example
-ssh -L 8888:127.0.0.1:8050 sam@kwan-bioinformatics.pharmacy.wisc.edu
-```
-
-Once you are on the server, simply start the Automappa server (with the appropriate port from the ssh tunnel).
+### clone the Automappa Repository
 
 ```bash
-automappa \
-    --binning-main <path to binning.main.tsv> \
-    --markers <path to binning.markers.tsv> \
-    --port 8050
+git clone -b home-tab https://github.com/WiscEvan/Automappa
 ```
 
-Navigate to the app view in your browser.
-
-This will correspond to the localport that was passed in upon login to the remote server.
-In the previous example above we would navigate to `localhost:8888`.
-
-I've numbered the ports here to help illustrate the network communication.
-
-| Bridge | Port Bridge | Communication Context |
-| :------------- | :------------- | :------------- |
-| `localport:remoteport` | `8888:8050` | `local:remote` |
-
-### Using a remote docker container Automappa server
-
-To access Automappa through a docker container that is on a remote machine, one additional bridge
-must be constructed.
-
-First we need to forward a port from the server back to our local machine.
+### build images for automappa services
 
 ```bash
-#ssh -L localport:localhost:serverport user@kwan-bioinformatics.pharmacy.wisc.edu
-ssh -L 8888:localhost:8887 sam@kwan-bioinformatics.pharmacy.wisc.edu
+make build
 ```
 
-Now run automappa using the docker wrapper script: `run_automappa.sh`
+### build and run automappa services
 
-> NOTE: A wrapper is available for download to run docker with port-forwarding.
+NOTE: you can skip `make build` if you’d like, as this command will build and pull any images not available.
 
 ```bash
-curl -o $HOME/run_automappa.sh https://raw.githubusercontent.com/WiscEvan/Automappa/main/docker/run_automappa.sh
-chmod a+x $HOME/run_automappa.sh
+make up
 ```
 
-Now start automappa while setting `--localport` to match the `serverport` (`8887` from above).
-
-```bash
-# NOTE: This will pull the automappa docker image if it is not already available.
-$HOME/run_automappa.sh \
-    --imagetag main \
-    # NOTE: The 'localport' here is referring to the port on the remote
-    --localport 8887 \
-    --containerport 8050 \
-    --binning binning.main.tsv \
-    --markers binning.markers.tsv
-```
-
-Now navigate to `http://localhost:8888` and you will see the loaded data.
-
-I've numbered the ports here to help illustrate the network communication.
-
-#### Example port forwarding breakdown
-
-| Server | Port |
-| :------------- | :------------- |
-| Docker container | 8050 |
-| Remote Server | 8887 |
-| Local Computer | 8888 |
-
-#### Note
-
-- You may change **any** of these values as long as you change the respective value.
-- This will be most useful if **multiple users** will need to use the app.
-
-| Bridge | Port Bridge | Communication Context |
-| :------------- | :------------- | :------------- |
-| `remoteport:containerport` | `8887:8050` | `remote:docker` |
-| `localport:remoteport` | `8888:8887` | `local:remote` |
-
-e.g.
-
-- `localhost:8888` <-> `8888:8887` <-> `8887:8050`
-
-or
+### Navigate to Automappa page
 
-- `localhost:localport` <-> `localport:serverport` <-> `serverport:containerport`
+Once you see `automappa_web_1` running from the terminal logs, you should be able to navigate to <localhost:8050> 🥳