Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions docs/source/backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,19 @@ The WAL-G backup setup is integrated into the Postgres Docker image. It will do
WAL-G is installed alongside Postgres. See the Dockerfile of Postgres (`postgres/Dockerfile`) for details. In the docker-compose postgres command (`docker-compose.yml`) archive parameter of postgres are set needed to make archives.


## Restore setup
### Configuration

The WAL-G restore setup is realized in a dedicated Docker image (`postgres/Dockerfile-restore_backup`). It will create a new Postgres database cluster, fetch latest backup using `wal-g backup-fetch` and create a `recovery.conf` file in the new cluster during Docker build. `recovery.conf` is used by Postgres during first start to get the `restore_command`. Again the exact commands are to be found in `postgres/wal-g/restore_command.sh`. During first start Postgres will get WALs from backup server and restore the database.
To store backups in Google Cloud Storage, WAL-G requires that this variable is set: `WALG_GS_PREFIX` to specify where to store backups (eg. `gs://x4m-test-bucket/walg-folder`).
Please add this to the `.env` file at the root of MapSwipe Back-end (See `.example-env` for environment variables which have to be set)

WAL-G determines Google Cloud credentials using application-default credentials like other GCP tools. Get a Service Account Key file (`serviceAccountKey.json`) for your Google Cloud Storage (See [Google Cloud Docs](https://cloud.google.com/iam/docs/creating-managing-service-account-keys)) and save it to `postgres/serviceAccountKey.json`.

## Configuration

To store backups in Google Cloud Storage, WAL-G requires that this variable be set: `WALG_GS_PREFIX` to specify where to store backups (eg. `gs://x4m-test-bucket/walg-folder`).
Please add this to your `.env` file at the root of MapSwipe Back-end (See `.example-env` for environment variables wich needs to be set)
## Restore setup

WAL-G determines Google Cloud credentials using application-default credentials like other GCP tools. Get a Service Account Key file (`serviceAccountKey.json`) for your Google Cloud Storage (See [Google Cloud Docs](https://cloud.google.com/iam/docs/creating-managing-service-account-keys)) and save it to `postgres/serviceAccountKey.json`.
The WAL-G restore setup is realized in a dedicated Docker image (`postgres/recovery/Dockerfile`). The entrypoint is the `ini_recovery.sh` srcipt. This script will create a new Postgres database cluster, fetch latest backup using `wal-g backup-fetch` and create a `recovery.conf` file in the new cluster. `recovery.conf` is used by Postgres during first start to get the `restore_command`. Again the exact commands are to be found in `postgres/recovery/restore_command.sh`. During first start Postgres will get WALs from backup server and restore the database.


### Configuration

> The same configuration as for the backup setup is requiered. Except the Service Account Key has to be stored at `postgres/recovery/serviceAccountKey`.
43 changes: 40 additions & 3 deletions docs/source/contributing.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,49 @@
# Contributing

This document is a work progress. It should describe how to contribute (code or issues) to the MapSwipe back-end.

To contribute to the MapSwipe back-end please create dedicated feature branches.
## Clone from GitHub

```bash
git clone https://github.com/mapswipe/python-mapswipe-workers.git
cd python-mapswipe-workers
git checkout dev
```


## Install MapSwipe Workers

Create a Python virtual environment and activate it. Install MapSwipe Workers using pip:

```bash
python -m venv venv
source venv/bin/activate
pip install --editable .
```


## Feature Branch

To contribute to the MapSwipe back-end please create dedicated feature branches from dev:

```bash
git checkout dev
git checkout -b featureA
git commit -am 'add new project type'
git push -u origin featureA
git request-pull origin/master featureA
git request-pull origin/dev featureA
```

> Note: If a bug in production (master branch) needs fixing before a new versions of MapSwipe Workers gets released (merging dev into master branch), a hotfix branch should be created. In the hotfix branch the bug should be fixed and then merged back with master and also dev.


## Style Guide

This project uses [black](https://github.com/psf/black) and [flake8](https://gitlab.com/pycqa/flake8) to achieve a unified style.

Use [pre-commit](https://pre-commit.com/) to run `black` and `flake8` prior to any git commit. `pre-commit`, `black` and `flake8` should already be installed in your virtual environment since they are listed in `requirements.txt`. To setup pre-commit simply run:

```
pre-commit install
```

From now on `black` and `flake8` should run automatically whenever `git commit` is executed.