Docker 2 Deploy

2️⃣ Docker: Deploy 🚀

⚒️ Build

Base script

To run the application in production, you can run the following script.

cd aikon/front/docker
bash docker.sh build

Other actions available through the docker.sh script:

bash docker.sh {start|stop|restart|update|build|log} ?[container_name]
# if a container_name is provided, only this container will be started/stopped/restarted/...

🔨 Pre-build process

Before each build, the docker.sh script will execute the init.sh script.

👆Click for details on `init.sh` script

The init.sh script will (if not already the case):

Define the environment variables of front/app/config/.env based on your input (default variable values should be fine)
- PROD_URL: domain name of your application (no https://)
- EMAIL_HOST / EMAIL_PORT / EMAIL_HOST_USER / EMAIL_HOST_PASSWORD: Email documentation
- GEONAMES_USER: username for geonames created earlier
- HTTP_PROXY / HTTPS_PROXY: proxy settings (see Proxies for more details)
Generate front/docker/.env
- USERID: user id from earlier
- DATA_FOLDER: folder on the host machine where the web container will store the media files – it will be created later on by the script
- SSL_KEY / SSL_CERTIFICATE: SSL key and certificate paths from earlier
Generate the cantaloupe.properties file based on your environment variables
Create the log files with correct permissions
Create the persistent storage folders with correct permissions: the directory will be mounted as volumes inside the containers
- $DATA_FOLDER/mediafiles => corresponding to /data/mediafiles inside the web container
Generate various nginx configuration files based on your environment variables

Building

Building the actual app is done through Docker and should run smoothy.

Automated post-build operations

manage.sh and the webapp's Dockerfile finishes the setup by running some automated Django commands:

collectstatic collects all static files in the web container
migrate runs those migrations
createsuperuser creates a default superuser based on your env variables:
- username is POSTGRES_USER
- password is POSTGRES_PASSWORD
- email adress is EMAIL_HOST_USER

🔧 Post-build

Here, we describe what you can do post-build:

Access containers

Once built, you can check that the containers (docker-web-1, docker-nginx-1, docker-db-1, docker-aiiinotate-1, docker-mongo-1, docker-cantaloupe-1, docker-redis-1) are correctly created and launched with the following commands:

# check logs of each container
docker logs $CONTAINER_NAME

# check that app is responded to HTTP requests
curl -i localhost:$NGINX_PORT/test

Useful commands

Here is an index of all the useful commands to complete setup of the application:

Django

Manually collect static files and run migrations (normally done automatically in Dockerfile with manage.sh):

# collect django static files inside web container
docker exec docker-web-1 /home/aikon/.venv/bin/python /home/aikon/app/manage.py collectstatic --noinput

# create django migrations
docker exec docker-web-1 /home/aikon/.venv/bin/python /home/aikon/app/manage.py makemigrations
# simulate what would happen if you run django migrations
docker exec docker-web-1 /home/aikon/.venv/bin/python /home/aikon/app/manage.py migrate --plan
# run django migrations
docker exec docker-web-1 /home/aikon/.venv/bin/python /home/aikon/app/manage.py migrate

# reindex images of specific witness / digitization / all witnesses
docker exec docker-web-1 /home/aikon/.venv/bin/python /home/aikon/app/manage.py reconvert_digitization -w <witness_id>
docker exec docker-web-1 /home/aikon/.venv/bin/python /home/aikon/app/manage.py reconvert_digitization -d <digitization_id>
docker exec docker-web-1 /home/aikon/.venv/bin/python /home/aikon/app/manage.py reconvert_digitization -a

Reindex the json property of searchable records for the database

# generate json indexation of witnesses
curl -i localhost:$NGINX_PORT/search/json-generation

Execute custom Django commands (defined in front/app/webapp/management/commands/):

# clear cache (notably for similar pairs retrieval)
docker exec docker-web-1 /home/aikon/venv/bin/python /home/aikon/app/manage.py clear_cache

# relaunch the process of digitization conversion (e.g. when a pdf wasn't correctly turn into jpg)
docker exec docker-web-1 /home/aikon/venv/bin/python /home/aikon/app/manage.py reconvert_digitization

# make sure regions pairs do not contain duplicates, incorrect image names or swapped regions ids
docker exec docker-web-1 /home/aikon/venv/bin/python /home/aikon/app/manage.py clean_regionpairs

Access the Django interactive shell:

# inside aikon/ directory
docker exec -it docker-web-1 bash

# open Django interactive shell
/home/aikon/venv/bin/python /home/aikon/app/manage.py shell

Aiiinotate & Mongo

Import data in an aiiinotate instance:

# run on the server
cd aikon/front/docker
# import annotations or manifests from a folder containing only IIIF AnnotationLists or IIIF manifests
bash docker_aiiinotate_import.sh <"annotations"|"manifests"> <path/to/directory>

Access aiiinotate's Mongo database through mongosh:

docker exec -it docker-mongo-1 mongosh $MONGODB_DB -p $MONGODB_PORT

PostgeSQL DB

Access AIKON's PostgreSQL database through psql:

psql -h 0.0.0.0 -p $DB_PORT -U $POSTGRES_USER -d $POSTGRES_DB

📮 Serve: nginx config

In production AIKON's nginx config is twofold:

server nginx: an nginx config in the server /etc/nginx/sites-enables directs incoming requests to the Docker-compose.
dockerized nginx: an NGINX Docker container routes incoming requests between all Docker containers within the Docker-compse.

Both are needed to run.

Dockerized nginx

Its config is auto-generated from nginx.conf.template. The nginx container should run once the build step has completed.

Server nginx

In order to serve the application to the World Wide Web, you will need a server-level nginx configuration file to:

proxy external HTTPS requests to the port used by the nginx container (NGINX_PORT in environment variables).
handle SSL termination

The init.sh script should already have created the server nginx's config file for you 🥳: cat nginx_external.conf. To enable it:

# copy configuration file to nginx sites-available
sudo cp nginx_external.conf /etc/nginx/sites-available/aikon
# create symbolic link to sites-enabled
sudo ln -s /etc/nginx/sites-available/aikon /etc/nginx/sites-enabled/
# test nginx configuration is correct
sudo nginx -t
# restart nginx (run this command each time you modify the external nginx config file)
sudo systemctl reload nginx

Other nginx configuration options

Other configurations are generated by default:

docker/nginx_reverse_proxy.conf: if your host is itself behind a reverse proxy
docker/nginx_ssl.conf: to handle external traffic directly with docker nginx container

🎣 Update a running AIKON Docker instance

Before updating your application code, follow these steps to ensure a smooth update process:

Note: The code changes won't take effect until you rebuild your Docker containers

Check code difference

TLDR;

# Preview incoming changes
gdiff

# Backup code / data / files
bash backup.sh

# Update code
git pull

Full process

Add the following alias to your ~/.bashrc to easily view changes between your local code and the remote repository before pulling:

cat << 'EOF' >> ~/.bashrc && source ~/.bashrc
function git_branch() {
    if [ -d .git ] || git rev-parse --git-dir > /dev/null 2>&1; then
        local branch_name=$(git branch 2>/dev/null | grep '^*' | colrm 1 2)

        if [ -n "$branch_name" ]; then
            echo "$branch_name"
        fi
    fi
}

alias gdiff='git fetch && git diff $(git_branch) origin/$(git_branch) -- ":(exclude)**/static/svelte/**"'
EOF

Run gdiff (defined above) in your project directory to preview incoming changes
```
gdiff  # review and make sure you're not pulling any dramatic mistake
```

Create a backup branch. Backup your your current production code using:

# EITHER run
bash backup/backup_code.sh

# OR, do it manually:
# Replace <your_prod_branch> with the name of your actual branch name
PROD_BRANCH=<your_prod_branch_name>
# delete the legacy branch if it already exists
git branch -d "${PROD_BRANCH}_legacy"
# create a new branch with the current code
git switch -c "${PROD_BRANCH}_legacy"
# push the backup branch to the remote repository
git push -u origin "${PROD_BRANCH}_legacy"
# Switch back to main branch
git switch $PROD_BRANCH

Docker 2 Deploy

2️⃣ Docker: Deploy 🚀

Table of contents

⚒️ Build

Base script

🔨 Pre-build process

👆Click for details on init.sh script

Building

Automated post-build operations

🔧 Post-build

Access containers

Useful commands

Django

Aiiinotate & Mongo

PostgeSQL DB

📮 Serve: nginx config

Dockerized nginx

Server nginx

Other nginx configuration options

🎣 Update a running AIKON Docker instance

Check code difference

TLDR;

Full process

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Project overview

Deployment

Web application

Utils

Clone this wiki locally

👆Click for details on `init.sh` script