Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
967891b
fix typo
Nov 11, 2019
dc46de8
refactor init scripts for db
Nov 11, 2019
0562220
add comment
Nov 11, 2019
b24162d
add comments
Nov 13, 2019
655db3a
Merge branch 'dev' into postgres-row-count
Nov 13, 2019
dba287c
add data tables for manager dashboard
Hagellach37 Nov 19, 2019
0fbaf8a
add description to each project group
Hagellach37 Nov 19, 2019
5bb24c9
changed recovery setup
Nov 20, 2019
2d7e4c9
add style guide
Nov 20, 2019
6d93da5
Merge branch 'dev' into docs
Nov 20, 2019
4c5ff5a
use datatables add and remove row
Hagellach37 Nov 20, 2019
ac9a0f8
simplify user interactions
Hagellach37 Nov 20, 2019
e169305
change isFeatured
Hagellach37 Nov 21, 2019
e89a33b
Merge pull request #230 from mapswipe/feature-add-status-finished
Hagellach37 Nov 21, 2019
359e7aa
add sql query to get overall stats such as project count #203
Hagellach37 Nov 21, 2019
f7827a1
change filename to stats.csv #203
Hagellach37 Nov 21, 2019
71344ef
add event listener to data table
Hagellach37 Nov 21, 2019
c338002
Merge pull request #236 from mapswipe/hotfixes
Matthias-Schaub Nov 25, 2019
ce67205
use projects df to derive overall stats
Hagellach37 Nov 25, 2019
23223f7
replace newline characters in projects
Hagellach37 Nov 25, 2019
4076799
set status of new projects to 'inactive' after they have been created
Hagellach37 Nov 25, 2019
67d59bc
don't display new projects
Hagellach37 Nov 25, 2019
9040f94
Merge pull request #232 from mapswipe/feature-overall-stats
Hagellach37 Nov 25, 2019
22265be
add prefered branch to make a pull request. Add note about hotfix bra…
Nov 26, 2019
ba9f6c9
removed trigger functions for row counts
Nov 26, 2019
be97f9c
remove primary and foreign key constraint on results_temp table
Nov 26, 2019
8f39aa2
Merge pull request #239 from mapswipe/postgres_triggers
Matthias-Schaub Nov 26, 2019
c4afa22
Merge pull request #227 from mapswipe/docs
Hagellach37 Nov 26, 2019
95ffc59
work on comments #238
Hagellach37 Nov 26, 2019
b0cfb56
change tile_y in tutorial for build area #240
Hagellach37 Nov 26, 2019
65eae7f
adjust tile_z coordinate for change detection tutorial
Hagellach37 Nov 26, 2019
0a53717
add comment
Hagellach37 Nov 26, 2019
c2d31f7
new projects will be displayed as inactive
Hagellach37 Nov 26, 2019
489a3bb
Merge pull request #233 from mapswipe/fix-manager-dashboard-change-st…
Hagellach37 Nov 26, 2019
2787665
Merge pull request #241 from mapswipe/fix-tutorial-scale-bar
Hagellach37 Nov 26, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ services:
build:
context: postgres/
environment:
POSTGERS_PASSWORD: '${POSTGRES_PASSWORD}'
POSTGRES_PASSWORD: '${POSTGRES_PASSWORD}'
POSTGRES_USER: 'mapswipe_workers'
POSTGRES_DB: 'mapswipe'
PGDATA: '/var/lib/postgresql/mapswipe'
Expand All @@ -81,7 +81,7 @@ services:
# Set options for WAL-G (backup tool)
command: postgres -c archive_mode=on -c archive_timeout=60 -c archive_command="/archive_command.sh %p"
volumes:
- ./postgres-data:/var/lib/postgresql/mapswipe
- ./postgres-data:/var/lib/postgresql/mapswipe
restart: unless-stopped
ports:
- "5432:5432"
Expand Down
17 changes: 11 additions & 6 deletions docs/source/backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,19 @@ The WAL-G backup setup is integrated into the Postgres Docker image. It will do
WAL-G is installed alongside Postgres. See the Dockerfile of Postgres (`postgres/Dockerfile`) for details. In the docker-compose postgres command (`docker-compose.yml`) archive parameter of postgres are set needed to make archives.


## Restore setup
### Configuration

The WAL-G restore setup is realized in a dedicated Docker image (`postgres/Dockerfile-restore_backup`). It will create a new Postgres database cluster, fetch latest backup using `wal-g backup-fetch` and create a `recovery.conf` file in the new cluster during Docker build. `recovery.conf` is used by Postgres during first start to get the `restore_command`. Again the exact commands are to be found in `postgres/wal-g/restore_command.sh`. During first start Postgres will get WALs from backup server and restore the database.
To store backups in Google Cloud Storage, WAL-G requires that this variable is set: `WALG_GS_PREFIX` to specify where to store backups (eg. `gs://x4m-test-bucket/walg-folder`).
Please add this to the `.env` file at the root of MapSwipe Back-end (See `.example-env` for environment variables which have to be set)

WAL-G determines Google Cloud credentials using application-default credentials like other GCP tools. Get a Service Account Key file (`serviceAccountKey.json`) for your Google Cloud Storage (See [Google Cloud Docs](https://cloud.google.com/iam/docs/creating-managing-service-account-keys)) and save it to `postgres/serviceAccountKey.json`.

## Configuration

To store backups in Google Cloud Storage, WAL-G requires that this variable be set: `WALG_GS_PREFIX` to specify where to store backups (eg. `gs://x4m-test-bucket/walg-folder`).
Please add this to your `.env` file at the root of MapSwipe Back-end (See `.example-env` for environment variables wich needs to be set)
## Restore setup

WAL-G determines Google Cloud credentials using application-default credentials like other GCP tools. Get a Service Account Key file (`serviceAccountKey.json`) for your Google Cloud Storage (See [Google Cloud Docs](https://cloud.google.com/iam/docs/creating-managing-service-account-keys)) and save it to `postgres/serviceAccountKey.json`.
The WAL-G restore setup is realized in a dedicated Docker image (`postgres/recovery/Dockerfile`). The entrypoint is the `ini_recovery.sh` srcipt. This script will create a new Postgres database cluster, fetch latest backup using `wal-g backup-fetch` and create a `recovery.conf` file in the new cluster. `recovery.conf` is used by Postgres during first start to get the `restore_command`. Again the exact commands are to be found in `postgres/recovery/restore_command.sh`. During first start Postgres will get WALs from backup server and restore the database.


### Configuration

> The same configuration as for the backup setup is requiered. Except the Service Account Key has to be stored at `postgres/recovery/serviceAccountKey`.
43 changes: 40 additions & 3 deletions docs/source/contributing.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,49 @@
# Contributing

This document is a work progress. It should describe how to contribute (code or issues) to the MapSwipe back-end.

To contribute to the MapSwipe back-end please create dedicated feature branches.
## Clone from GitHub

```bash
git clone https://github.com/mapswipe/python-mapswipe-workers.git
cd python-mapswipe-workers
git checkout dev
```


## Install MapSwipe Workers

Create a Python virtual environment and activate it. Install MapSwipe Workers using pip:

```bash
python -m venv venv
source venv/bin/activate
pip install --editable .
```


## Feature Branch

To contribute to the MapSwipe back-end please create dedicated feature branches from dev:

```bash
git checkout dev
git checkout -b featureA
git commit -am 'add new project type'
git push -u origin featureA
git request-pull origin/master featureA
git request-pull origin/dev featureA
```

> Note: If a bug in production (master branch) needs fixing before a new versions of MapSwipe Workers gets released (merging dev into master branch), a hotfix branch should be created. In the hotfix branch the bug should be fixed and then merged back with master and also dev.


## Style Guide

This project uses [black](https://github.com/psf/black) and [flake8](https://gitlab.com/pycqa/flake8) to achieve a unified style.

Use [pre-commit](https://pre-commit.com/) to run `black` and `flake8` prior to any git commit. `pre-commit`, `black` and `flake8` should already be installed in your virtual environment since they are listed in `requirements.txt`. To setup pre-commit simply run:

```
pre-commit install
```

From now on `black` and `flake8` should run automatically whenever `git commit` is executed.
28 changes: 18 additions & 10 deletions manager_dashboard/manager_dashboard/create.html
Original file line number Diff line number Diff line change
Expand Up @@ -14,23 +14,31 @@
<meta name="twitter:card" content="summary">
<meta name="viewport" content="width=device-width, initial-scale=1">

<!-- add firebase -->
<script src="https://cdn.firebase.com/libs/firebaseui/4.0.0/firebaseui.js"></script>
<link type="text/css" rel="stylesheet" href="https://cdn.firebase.com/libs/firebaseui/4.0.0/firebaseui.css" />

<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
<script src="https://code.jquery.com/jquery-3.2.1.slim.min.js" integrity="sha384-KJ3o2DKtIkvYIK3UENzmM7KCkRr/rE9/Qpg6aAZGJwFDMVNA/GpGFF93hXpG5KkN" crossorigin="anonymous"></script>
<!-- add jquery, ajax -->
<script src="https://code.jquery.com/jquery-3.3.1.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.12.9/umd/popper.min.js" integrity="sha384-ApNbgh9B+Y1QKtv3Rn7W3mgPxhU9K/ScQsAP7hUibX39j7fakFPskvXusvfa0b4Q" crossorigin="anonymous"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/js/bootstrap.min.js" integrity="sha384-JZR6Spejh4U02d8jOt6vLEHfe/JQGiRRSQQxSfFWpi1MquVdAyjUar5+76PVCmYl" crossorigin="anonymous"></script>

<!-- Add Bootstrap -->
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js" integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM" crossorigin="anonymous"></script>

<!-- Add Data Tables -->
<link rel="stylesheet" href="https://cdn.datatables.net/1.10.20/css/dataTables.bootstrap.min.css">
<script src="https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js"></script>

<!-- add mapswipe styles -->
<link type="text/css" rel="stylesheet" href="css/mapswipe.css" />
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.5.1/dist/leaflet.css"
integrity="sha512-xwE/Az9zrjBIphAcBb3F6JVqxf46+CDLwfLMHloNu6KEQCAWi6HcDUbeOfBIptF7tcCzusKFjFw2yuvEpDL9wQ=="
crossorigin=""/>
<!-- Make sure you put this AFTER Leaflet's CSS -->
<script src="https://unpkg.com/leaflet@1.5.1/dist/leaflet.js"
integrity="sha512-GffPMF3RvMeYyc1LWMHtK8EbPv0iNZ8/oTtHPx9/cc2ILxQ+u905qIwdpULaqDkyBKgOaB57QTMg7ztg8Jm2Og=="
crossorigin=""></script>

<!-- add leaflet and turf.js -->
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.5.1/dist/leaflet.css" integrity="sha512-xwE/Az9zrjBIphAcBb3F6JVqxf46+CDLwfLMHloNu6KEQCAWi6HcDUbeOfBIptF7tcCzusKFjFw2yuvEpDL9wQ==" crossorigin=""/>
<script src="https://unpkg.com/leaflet@1.5.1/dist/leaflet.js" integrity="sha512-GffPMF3RvMeYyc1LWMHtK8EbPv0iNZ8/oTtHPx9/cc2ILxQ+u905qIwdpULaqDkyBKgOaB57QTMg7ztg8Jm2Og==" crossorigin=""></script>
<script src='https://npmcdn.com/@turf/turf/turf.min.js'></script>

<!-- add sentry -->
<script
src="https://browser.sentry-cdn.com/5.7.1/bundle.min.js"
integrity="sha384-KMv6bBTABABhv0NI+rVWly6PIRvdippFEgjpKyxUcpEmDWZTkDOiueL5xW+cztZZ"
Expand Down
29 changes: 28 additions & 1 deletion manager_dashboard/manager_dashboard/css/mapswipe.css
Original file line number Diff line number Diff line change
Expand Up @@ -134,4 +134,31 @@
}

#geometryMap { height: 300px; width: 100%}
#geometryChangeDetectionMap { height: 300px; width: 100%}
#geometryChangeDetectionMap { height: 300px; width: 100%}

div.project-data {
padding-bottom: 50px !important;
padding-top: 25px !important;
}


/* adjust data tables styles */
.table {
border-spacing: 0px !important;
}

th.sorting::after{
content: "\2195" !important;
}

th.sorting_asc::after{
content: "\2193" !important;
}

th.sorting_desc::after{
content: "\2191" !important;
}

.paginate_button {
padding: 2.5px !important;
}
23 changes: 20 additions & 3 deletions manager_dashboard/manager_dashboard/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,31 @@
<meta name="twitter:card" content="summary">
<meta name="viewport" content="width=device-width, initial-scale=1">

<!-- add firebase -->
<script src="https://cdn.firebase.com/libs/firebaseui/4.0.0/firebaseui.js"></script>
<link type="text/css" rel="stylesheet" href="https://cdn.firebase.com/libs/firebaseui/4.0.0/firebaseui.css" />

<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
<script src="https://code.jquery.com/jquery-3.2.1.slim.min.js" integrity="sha384-KJ3o2DKtIkvYIK3UENzmM7KCkRr/rE9/Qpg6aAZGJwFDMVNA/GpGFF93hXpG5KkN" crossorigin="anonymous"></script>
<!-- add jquery, ajax -->
<script src="https://code.jquery.com/jquery-3.3.1.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.12.9/umd/popper.min.js" integrity="sha384-ApNbgh9B+Y1QKtv3Rn7W3mgPxhU9K/ScQsAP7hUibX39j7fakFPskvXusvfa0b4Q" crossorigin="anonymous"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/js/bootstrap.min.js" integrity="sha384-JZR6Spejh4U02d8jOt6vLEHfe/JQGiRRSQQxSfFWpi1MquVdAyjUar5+76PVCmYl" crossorigin="anonymous"></script>

<!-- Add Bootstrap -->
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js" integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM" crossorigin="anonymous"></script>

<!-- Add Data Tables -->
<link rel="stylesheet" href="https://cdn.datatables.net/1.10.20/css/dataTables.bootstrap.min.css">
<script src="https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js"></script>

<!-- add mapswipe styles -->
<link type="text/css" rel="stylesheet" href="css/mapswipe.css" />

<!-- add leaflet and turf.js -->
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.5.1/dist/leaflet.css" integrity="sha512-xwE/Az9zrjBIphAcBb3F6JVqxf46+CDLwfLMHloNu6KEQCAWi6HcDUbeOfBIptF7tcCzusKFjFw2yuvEpDL9wQ==" crossorigin=""/>
<script src="https://unpkg.com/leaflet@1.5.1/dist/leaflet.js" integrity="sha512-GffPMF3RvMeYyc1LWMHtK8EbPv0iNZ8/oTtHPx9/cc2ILxQ+u905qIwdpULaqDkyBKgOaB57QTMg7ztg8Jm2Og==" crossorigin=""></script>
<script src='https://npmcdn.com/@turf/turf/turf.min.js'></script>

<!-- add sentry -->
<script
src="https://browser.sentry-cdn.com/5.7.1/bundle.min.js"
integrity="sha384-KMv6bBTABABhv0NI+rVWly6PIRvdippFEgjpKyxUcpEmDWZTkDOiueL5xW+cztZZ"
Expand Down
12 changes: 12 additions & 0 deletions manager_dashboard/manager_dashboard/js/bootstrap.min.js

Large diffs are not rendered by default.

Loading