## _Conforma_ Server build, versioning, and install using Docker

The following is a guide to building Conforma, pushing it to the servers, configuring the servers with Docker and docker-compose, and launching the app.

Most of this can be accomplished using custom scripts we've made which reduces the effort required to upgrade a server to just a few short lines. Please see [Server Upgrade Guide](Server-Upgrade-Guide) for an overview of that process, as well as how to configure a new server with the launch scripts.

## Prep

- Add token to githubtoken.txt (for downloading expression-evaluator)
  - requires token that has read access to packages on server repo
  - Troubleshooting error 403 Unauthorised when running `yarn install` [Source](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-npm-registry#authenticating-with-a-personal-access-token):
    - Run `npm login --scope=@msupply-foundation --registry=https://npm.pkg.github.com`
    - Then enter your github registration `username`, `passowrd` (token) and `email`
- Ensure your back-end `.env` has a variable called `FRONT_END_PATH` with the _full_ path to the front-end repo on your local system.  
  e.g. `FRONT_END_PATH='/Users/<username>/GitHub/conforma/conforma-web-app/'`
- Ensure your local Docker installation is logged in with the **msupplyfoundation** account (Password in Bitwarden)
- Ensure your docker Engine running locally to build a new image
- Yarn should be in the latest v1. i.e: `1.22.18` to correctly use `yarn release`

## Bump the version and generate a tag.

The `version` field in `package.json` is the source of truth for the current version number. Increases to version numbers should be done according to [semantic versioning](https://semver.org/), including "pre-release" versions (for testing purposes, for example).

- Bump the version and create a tag by running the script `yarn release <version-type>`, where `version-type` is on of "--major", "--minor", "--patch", "--prerelease", "--preminor", "--premajor", "--prepatch". ("--pre-release" is the default if not specified.) (See [https://classic.yarnpkg.com/en/docs/cli/version](https://classic.yarnpkg.com/en/docs/cli/version) for version type explanation)
  - Example:
    `yarn release` from develop to generate a Pre-release (release candidate)
    Or `yarn release --minor` from `main` to generate a Minor release.
- This will also apply the new version number to the front-end (on whatever branch is currently checked out, so it should normally be `develop` or `main`). The tag is auto-generated by the `yarn version` command (run as part of this release script), and is just `v<version-num>` (e.g. `v0.1.5-2`)
- Note: it is not necessary to create a "release" on Github for each tag unless we need to make a particular version publicly downloadable, but one can easily be created based on the current tag/version on Github if required.
- If anything goes wrong during the release process, you can revert with the following list of commands:
  - `git reset --hard HEAD~1` (to revert release bump)
  - `git tag -d <tag-name(s)>` (delete local tags)
  - `git push --delete origin <tag-name(s)>` (delete remote tags)
  - You will also need to do the same on the front-end repo.

## Build (and upload to Docker hub)

`yarn dockerise <tag-or-branch-name> [push]`

Example:  
`yarn dockerise v0.1.5 push`

(If you don't specify any parameters, it will build the current `develop` branch.)

**Note** that the previously-mentioned `yarn release` script will offer to start this build process automatically with the tag you've just created.

If building from a _branch_ (as opposed to a _tag_), be aware of the following:

- the branch name cannot have a `#` character in it, like our branches normally begin with.
- the front-end and back-end branch need to have the same name, so if you're not building from `develop` or `main`, you'll probably need to create a "dummy" branch on the other repo so the branch names match.

### Login to docker

It should take a while to run and will build a local Docker image. The optional parameter `push` specifies if the script should automatically push the created image to the **msupplyfoundation** [Docker Hub](https://hub.docker.com/) account. If you are logged to a different acount you need to `docker logout` and then `docker login` to your local Docker installation with the **msupplyfoundation** account (Password in Bitwarden).

### Docker image tag

The build process will create a local image with a tag of the form:  
`build-<releaseTag>_<date>_<uniqueId>`

(where `uniqueId` is a random 6-char string)

_To see what's going on under the hood when this command is run, please inspect the file `/docker/dockerise.sh`_

### Manually pushing to dockerhub

If you don't auto-push the image as part of the build process, you can do so manually by running: `docker push <full image name>`, where `<full-image-name>` includes the account name, repo name ("conforma-demo") and tag.

Example:  
`docker push msupplyfoundation/conforma:build-v0.2.0-7_2022-04-07_ee35c8`

## Test locally

When the build (above) completes, it should print a command for running the new image locally, which you can copy and paste. It will be something like:

`yarn docker_run msupplyfoundation/conforma:build-v0.2.0-7_2022-04-07_ee35c8`

You'll need to make sure you have the SMTP_PASSWORD in your local `.env` file.

_To see the actual Docker commands that are constructed, please inspect the file `/docker/run.sh`_

## Log in to the demo server with ssh

- Get key file from Bitwarden (Conforma demo (5 instances) server) and save locally (e.g. in `~/Documents/private/conformakey.pem`)
- SSH login to server:
  ```bash
  export KEY_LOC='/Users/<you>/Documents/private/conformakey.pem' (or your local location)
  sudo ssh -i $KEY_LOC ubuntu@conforma-demo.msupply.org
  ```
- Alternatively, log in to any specific country server
- View a list of currently running images:  
  `sudo docker container ls`
- Stop all instances:  
  `sudo docker stop <hashes… >`
- Pull image from docker hub:  
  `sudo docker pull <full-image-name>`
  Example: `sudo docker pull msupplyfoundation/conforma:build-v0.2.0-7_2022-04-07_ee35c8`

If the server is already configured, you can skip to [docker-compose](#docker-compose)

## Move files/folder to/from instance

Follow this process whenever the docker-compose or nginx configs are updated (`./docker/demo_server/docker-compose.yml`)

### Copy script folder demo server scripts

**Note**: You are sending local changes to the server

```bash
cd conforma-server/docker
scp -r -i $KEY_LOC ./demo_server ubuntu@conforma-demo.msupply.org:/home/ubuntu/
```

## Save backup of nginx config from demo server to local

```bash
cd conforma-server/docker
scp -i $KEY_LOC ubuntu@conforma-demo.msupply.org:/etc/nginx/sites-enabled/default ./demo_server/nginx_config
```

## Upload nginx config back to demo server

### Option 1: Multi-instances server

**Note** The configuration of nginx uses the certificate of conforma-demo.msupply.org domain (following other steps of this setup) if needed for a new server, replace with new domain.

```bash
# cannot directly replace default config, need to do it as sudo, so from within docker instance
sudo mv demo_server/nginx_config/default /etc/nginx/sites-enabled/
```

### Option 2: Single-instance server

**Note**: First you need to have a domain configured and have been through the steps to Install SSL Certificate with certbot as described in the [New server instance guide](New-Server-Instance)

The file is configured to forward incoming requests to port 50000 to internal port 80000 for **Conforma** App and requests to port 50001 to internal port 80001 for **Grafana** both running as docker containers.

You will need to open the file in `demo_server/nginx_config_single/default` and edit where is showing as `<host-domain>` replacing with the actual domain name (as configured in the SSL certificate).

```bash
sudo mv demo_server/nginx_config_single/default /etc/nginx/sites-enabled/
```

## NGINX

Everything should be configured via `default` config. Cert bot was installed and should auto update certs (https://certbot.eff.org/lets-encrypt/ubuntubionic-nginx).

After changing config as per above, run

`sudo service nginx restart`

Logs are in `/var/log/nginx`

## docker-compose

docker-compose allows us to map specific container folders to actual folders on the host system ([volumes](https://docs.docker.com/storage/volumes/)) so that data can be persisted between container restarts (such as when doing an app upgrade). See the `docker-compose.yml' file to see the volume mappings we are using.

Volumes are created automatically the first time each docker-compose container instance is launched.

Docker stores these volumes in its internal data folder (`/var/lib/docker/volumes` in the case of our demo server), so it is a good idea to make a symlink to this folder in a more accessible place (such as in the `demo_server` folder where the `docker-compose.yml` file is). To make a symlink:

`ln -s <source-folder> <target-folder>`

E.g. (as on our demo server):  
`ln -s /var/lib/docker/volumes ~/demo_server/docker_data_volumes`

### Launching instances with docker-compose

For each instance, docker-compose is expecting a set of environment variables, some of which are contained within an `.env` file, and some included on the launch command.

- In an `.env` file:
  - `SMTP_PASSWORD` -- password for the email server specified in the "[sendNotification](List-of-Action-plugins.md#send-notification)" action
  - `WEB_HOST` -- full host name as will be shown in the application URL (including port), e.g. `https://conforma-demo.msupply.org:50004`
  - `BACKUPS_FOLDER` (optional, default: `~/backups`) -- path *on the host system* where the internal "backups" folder should be mapped to (optional -- default is the default volumes location). An appropriate location would be a folder that is synced to a cloud backup service. (See [Backups](Backups) for more info.)
  - `BACKUPS_PASSWORD` (optional) -- password for encrypting the backup archives (AES-encrypted .zip files) (if no password provided, the backups will be unencrypted .zip files)
- As part of the launch command (not included in `.env` file as they will change every time):
  - `PORT_APP`: -- the Http port the Conforma server will listen on (recommend start with `8000` and increase by 2 for each additional instance)
  - `PORT_DASH`: -- the Http port the Grafana server will listen on (recommend start with `8001` and increase by 2 for each additional instance)
  - `TAG` -- the name of the tag you're about to launch
  - `JWT_SECRET` -- private key for generating and verifying JWT tokens. Should be a strong, randomly generated string.

Then, for each instance, run the following launch commands (you can either `export` the env vars or include them in the command):

Option 1:
```sh
export PORT_APP=8000
export PORT_DASH=8001
export TAG='build-v0.5.6-76_2023-08-09_802505'
export JWT_SECRET=$(openssl rand -hex 64) #one possible way to generate

sudo -E docker compose --project-name 'conforma-on-8000' up -d
# Change project name 'conforma-on-8000' for each instance
```

Option 1:
```sh
PORT_APP=8000 PORT_DASH=8001 TAG='build-v0.5.6-76_2023-08-09_802505' JWT_SECRET=<long-secret> sudo -E docker compose --project-name 'conforma-on-8000' up -d
```

`-d` is for detached, if you want to see all output then start without `-d`

Note the above commands must be run within directory where compose.yml is contained. Typically this is `/demo_server`

This will either launch or relaunch the server at the port specified in the PORT_APP

## Stop instances and reset volumes

Stop a particular docker-compose instance:

`PORT_APP=8000 PORT_DASH=8001 sudo -E docker-compose --project-name 'conforma-on-8000' down`

The data volumes will persist, so next time that instance is launched (with docker-compose), data will be unchanged.

To reset (i.e. delete) a volume, use the following command (when instance is stopped):

`sudo docker volume rm <name_of_volume>`  
(See the volumes folder to get the exact name of each one)

**CAUTION**: Removing a volume will delete all the data on that instance. Ensure you have exported a snapshot before removing a volume!

To remove all currently unused volumes at once:

`sudo docker volume prune`

### View logs

```bash
# don't need bash inside container for this
sudo docker exec -ti conforma-on-8000_app_1 cat /var/log/conforma/server.log
sudo docker exec -ti conforma-on-8000_app_1 cat /var/log/conforma/graphile.log
```

### Other image/container commands

- List local images

`sudo docker images`

- Stop a specific container

`sudo docker stop <name-or-container-id>`

- Remove a container or its image:

`sudo docker rm <name-or-id>`

- Remove all containers

`sudo docker rm $(sudo docker ps -a -q)`

- Remove all images

`sudo docker rmi -f $(sudo docker images -a -q)`

### Access command line within container:

You can access the command line of a particular container instance with the following:

`sudo docker exec -ti <name-or-container-id> /bin/bash`

From there the following commands might be useful:

- view environment variables: `printenv`
- check the server log: `tail -n 100 /var/log/conforma/server.log`

### Copy the log file to your local environment:

1. Log in to the server using terminal
2. Copy the log to an accessible place (in the server) i.e. if getting log for instance 5000:

   `sudo cp /var/lib/docker/volumes/conforma-on-8000_logs/_data/server.log demo_server/2022-11-15.log`

3. Now exit the server `exit` and on your local machine open the terminal
4. Navigate to a folder where you would like to save the log, run (some similar to this):
   ```
   scp -i $KEY_LOC ubuntu@conforma-demo.msupply.org:/home/ubuntu/demo_server/2022-11-15.log .
   ```

- KEY_LOC is an environment variable for the server public key :)
- And the `.` means it will create a file with the same file name on the current folder

## Managing CloudVPS hosted servers

Conforma live servers are usually hosted on ([CloudVPS](https://www.cloudvps.com/)), an ([openstack](https://www.openstack.org/)) server provider.

### Changing CloudVPS server settings

Prior to changing any settings on CloudVPS, it is essential to take a backup snapshot of the live server, and ensure back ups are logging correctly to dropbox or wherever you are keeping your backups.

1. Log in to ([CloudVPS](https://openstack.cloudvps.com/auth/login/?next=/)). Credentials are available through Bitwarden.
2. Access all instances through compute / instances
3. Search for the relevant instance ie “conforma”
4. Click “resize instance” from dropdown
5. Select new flavour package
6. Upon set up, conforma docker image will need environment variables set up again, and run, using the [commands above](Demo-Server-Guide.md#launching-instances-with-docker-compose). Images and snapshots will persist when a server's capacity is changed.