CI & staging

How the pipeline works

1. Developers make changes to the main branch on Github.
2. The change in the main-branch triggers a GitHub Action, which results in a Docker image being built and getting pushed to a DockerHub repository.

(On the current staging server:)

3. A Watchtower container running on the staging server keeps track of new Docker images on DockerHub. It will pull the uploaded images when it detects changes and restarts containers, resulting in the updated images being run with minimal delay. The updating should thus work automagically.
4. The changes are now visible on the staging server (see that some changes like changes to the database might need more than just restarting. In other words, manual actions on the server. See Useful and needed docker commands.)

Setting up the pipeline

1. Create a Docker Hub account and create an access token. An access token can be created by logging in to Docker Hub and clicking on your username in the top right corner and selecting Account Settings > Security > New Access Token.
2. Provide DOCKERHUB_USERNAME and DOCKERHUB_TOKEN as Github Secrets. The secrets cannot be read after they have been set so they are safe from your fellow developers!
   • DOCKERHUB_USERNAME is your Docker Hub username
   • DOCKERHUB_TOKEN is the access token you created
3. Ask instructors for permissions to access the staging server.
4. Login to the staging server according to the instructions you should have been provided with by the instructors. At the moment of writing, this includes running a login script in the root folder.
5. Create a file docker-compose.yml to /qleader. Here is an example docker-compose.yml based on the current file in the staging.
6. Create a file qleader.subfolder.conf to /nginx/config/nginx/proxy-confs with the following contents:

location /qleader-static/ {
  proxy_pass	       http://qleader-static:80/;
}

location /qleader/ {
  proxy_pass         http://qleader-web:8000/qleader/;
  proxy_set_header   Host $host;
  proxy_set_header   X-Real-IP $remote_addr;
  proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header   X-Forwarded-Host $server_name;
  proxy_set_header   X-Forwarded-Proto $scheme;
  proxy_set_header   SCRIPT_NAME /qleader;
}

7. Create a .env file with all the OAuth2 secrets and other key information:

ROOT_DIR=
DEBUG=True

PRODUCTION=True
DJANGO_SETTINGS_MODULE=webmark2.settings
DJANGO_SECRET_KEY="NotDisclosed"

DATABASE_NAME=quantdb
DATABASE_USER=quantuser
DATABASE_PASSWORD=NotDisclosed
DATABASE_HOST=qleader-db
DATABASE_PORT=5432

GOOGLE_OAUTH2_KEY="NotDisclosed"
GOOGLE_OAUTH2_SECRET="NotDisclosed"

ORCID_KEY="NotDisclosed"
ORCID_SECRET="NotDisclosed"

FACEBOOK_KEY="NotDisclosed"
FACEBOOK_SECRET="NotDisclosed"

8. Create a folder ./static which holds the Django and REST Framework static files. Lighttpd in container qleader-static will serve these.

If the models change and the staging server returns an error message

If the data structure changes in the program, but the database itself is left unchanged, the mismatch will result in an error, and the service will not be able to work. At the moment of writing this, our solution for this has been to delete the old database and build a new one (if the program is in production, there would need to be a way to migrate the data from the old db to the new one -- we don't know how to do this yet).

The simplest way to delete the database on the staging server ("ohtup-staging") is to stop and remove the docker container. One way to do this is by navigating to the project folder (that has the docker-compose.yml -file) and giving the command

docker-compose down

After this check the ID of the docker container('s) that holds your database:

docker-container ls -a

And remove the container with the rm command (in this example the ID is 40d93a37 -- usually giving a few first characters is enough):

docker container rm 40d

After this you can start your project again with the command

docker-compose up --build

Useful and needed docker commands

If there is changes that require more than Watchtower's restarting actions, do the following in the given order:

1. Stopping the project (in the project folder with the docker-compose.yml -file):

   docker-compose down
   

2. Listing running containers (perhaps want to see what's the names of the containers):

   docker container ls
   

For seeing all the containers, even the stopped ones, add the flag -a.

3. Removing container (removing your project's old containers might be needed when creating new ones):

   docker container rm qleader-db  # Here is an example so replace the container name "qleader-db" with correct one!
   

   • See that the container is stopped before trying to remove it. For this there are more options like stop and kill (or docker-compose down, if you are in the project folder) depending on the situation.

4. Building project (in the beginning or if there are some bigger changes, for example, changes in the database)

   docker-compose up --build -d
   

5. Restarting (might be needed if some changes has been made)

   docker-compose restart
   

   • Note that restarting is useful in the project folder /qleader and in the /nginx

Known issues

• The staging server runs in a path /qleader so hard-coded links between pages will not work. For example, this is NOT going to work:

  return redirect('/newMolecule/')  # WILL NOT WORK ON STAGING!

  You should use a view name instead (defined in urls.py) so this is correct:

  return redirect('newMolecule')  # using view name defined in urls.py