City of Helsinki land lease system

If using Apple M1/M2 chip (or equivalent), you need to add platform: linux/amd64 to django service in docker-compose.yml file. When you want to run also Tunnistamo on local environment, see Connecting to Tunnistamo.

networks:
    default:
        name: tunnistamo_net
        external: true

1. Run docker-compose up

2. Run migrations if needed (if you have sanitized.sql file then skip 2. and 3. Continue to "Settings for development environment"):

   • docker exec mvj python manage.py migrate

3. Create superuser if needed:

   • docker exec -it mvj python manage.py createsuperuser

The project is now running at localhost:8000.

Known issues:

• runserver_plus not found/not working: replace command: python manage.py runserver_plus 0:8000 with command: python manage.py runserver 0:8000 command in docker-compose.yml.
• You can only start Tunnistamo or MVJ in Docker: change port to command: python manage.py runserver_plus 0:8000 command for example to 8001.

cd mvj
# copy sanitized.sql to root
docker-compose exec django bash
psql -h postgres -U mvj -d mvj < sanitized.sql
docker-compose exec django python manage.py migrate
docker-compose exec django python manage.py createsuperuser #(github email)

• Create TEST groups by:

python manage.py copy_groups_and_service_unit_mappings

• Add "TEST Pääkäyttäjä" -group to created user at Django admin.

If you have a Tunnistamo and an mvj-ui instance running with docker in separate docker-compose environments, you can set up a network to sync mvj, mvj-ui and tunnistamo together.

1. Add network definition to tunnistamo's docker-compose:

   version: '3'
   services:
       postgres:
           ...
           networks:
               - net
   
       django:
           ...
           networks:
               - net
   
   networks:
       net:
           driver: bridge
   

2. Connect mvj to tunnistamo's network, by adding this to mvj's and mvj-ui's docker-compose:

   networks:
       default:
           name: tunnistamo_net
           external: true
   

   The name tunnistamo_net comes from the name of the folder, where tunnistamo lives combined with the name of the network. Change those according to your setup, if needed.

3. Now you can access tunnistamo from other docker containers with tunnistamo-backend, i.e. Tunnistamo's django container's name. Connect mvj's OIDC logic to that like so:

       OIDC_API_TOKEN_AUTH = {
           ...
           'ISSUER': 'http://tunnistamo-backend:8001/openid',
           ...
       }
   

4. Add tunnistamo-backend to your computer's localhost aliases. To do this on UNIX-like systems open /etc/hosts and add it:

       127.0.0.1    localhost tunnistamo-backend
   

   This way callbacks to tunnistamo-backend URL will work locally.

5. Configure OIDC settings in Tunnistamo's admin panel. Might require help from other devs.

6. Configure some social auth application to allow requests from your local tunnistamo by using this URL in the settings http://tunnistamo-backend

docker-compose exec django python manage.py createsuperuser

At Django admin:

• Create new Login Method: yletunnus

• OpenID Connect Provider / Clients / Add client:

  • Name: mvj
  • Client Type: public
  • Response types: id_token token (Implicit Flow)
  • Redirect URIs: http://localhost:3000/callback http://localhost:3000/silent_renew.html
  • Client ID: https://api.hel.fi/auth/mvj
  • Site type: Development
  • Login methods: GitHub

• Oidc_Apis / APIs / Add API:

  • Domain: https://api.hel.fi/auth
  • Name: mvj
  • Required scopes: Sähköposti, Profile, Address, AD Groups
  • OIDC client: mvj

• Oidc_Apis / API scopes / Add API scope:

  • API: https://api.hel.fi/auth/mvj
  • Name: mvj
  • Description: lue ja modifioi
  • Allowed applications: mvj

• Copy docker-compose.env.template -> docker-compose.env

• Create a new OAuth app to Github (use http://tunnistamo-backend:8888/accounts/github/login/callback/ as callback url)

• Add SOCIAL_AUTH_GITHUB_KEY and SOCIAL_AUTH_GITHUB_SECRET from Github to docker-compose.env.yaml

Install PostgreSQL and PostGIS.

# Ubuntu 16.04
sudo apt-get install python3-dev libpq-dev postgresql postgis

# Ubuntu 16.04
sudo apt-get install binutils libproj-dev gdal-bin

Create a Python 3.x virtualenv either using the venv tool or using the great virtualenvwrapper toolset. Assuming the latter, once installed, simply do:

mkvirtualenv -p /usr/bin/python3 mvj

The virtualenv will automatically activate. To activate it in the future, just do:

workon mvj

• Run pip install pip-tools
• Run pip-compile requirements.in
• Run pip-compile requirements-dev.in

• Run pip-compile --upgrade requirements.in
• Run pip-compile --upgrade requirements-dev.in

• Run pip install -r requirements.txt
• For development also run pip install -r requirements-dev.txt

To setup a database compatible with the default database settings:

Create user and database

sudo -u postgres createuser -P -R -S mvj  # use password `mvj`
sudo -u postgres createdb -O mvj mvj

Enable PostGIS

sudo -u postgres psql -d "mvj" -c "CREATE EXTENSION IF NOT EXISTS postgis;"

Allow the mvj user to create databases when running tests

sudo -u postgres psql -d "mvj" -c "ALTER USER mvj CREATEDB;"

Tests also require that PostGIS extension is installed on the test database. This can be achieved the easiest by adding PostGIS extension to the default template which is then used when the test databases are created:

sudo -u postgres psql -d template1 -c "CREATE EXTENSION IF NOT EXISTS postgis;"

Environment variables are used to customize configuration in mvj/settings.py. If you wish to override any settings, you can place them in a local .env file which will automatically be sourced when Django imports the settings file.

Alternatively you can create a local_settings.py which is executed at the end of the mvj/settings.py in the same context so that the variables defined in the settings are available.

The "idc_attachments"-folder under the media root must be excluded when serving media files. The files are protected by permission checks on a different URL.

• Enable debug echo 'DEBUG=True' >> .env
• Run python manage.py migrate
• Run python manage.py loaddata */fixtures/*.json
• Run python manage.py runserver 0.0.0.0:8000

• Run pytest

• Run python manage.py app_makemessages

There are multiple management commands that are required to run. Either when first installing the software or regularly.

Sets permitted groups for each report type.

Sets the default model specific permissions (view, add, change, delete) to the pre-defined groups for the leasing models.

Sets field specific permissions (view, change) to the pre-defined groups.

Sets the default mappings for AD groups to user groups.

Sets the default user group to Service unit mappings. Creates a group for the Service unit if a group doesn't exist already. The service_unit.json fixture has to be loaded into the database before running this command.

Adds the default lessor contacts for all the Service units. The service_unit.json fixture has to be loaded into the database before running this command.

Adds the default receivable types for all the Service units. The service_unit.json should be loaded into the database before running this command.

Creates invoices for rents that are due in the next month.

Should be run on the first day of every month

Sends unsent invoices to Laske SAP for invoicing.

Should be run every night

Imports index from stat.fi.

Should be run monthly after stat.fi update day

Imports reference interest rates from the Bank of Finland.

Should be run after the Bank of Finland releases new rates

Creates credit notes or invoices if index number has changed after the invoices are sent.

Should be run once a year after the previous years index has been released

Imports leasehold transfers from National Land Survey of Finland (Maanmittauslaitos).

Should be run every week

No need to run.

You can resend the failed invoices in Django admin Invoices view by selecting the invoices and chosing Resend invoice action from dropdown.

Virre integration requires certificates to be installed to servers. Current certificates are installed using following commands.

cd /usr/local/share/ca-certificates/virre # Create folder, if it doesn't exist.
sudo touch virre_intermediate.crt
sudo touch virre_root.crt
sudo touch virre_server_certificate.crt
* Copy-paste content from the files provided to you to the files you just created.
sudo update-ca-certificates # This is the actual command that registers certificates.

Most important thing to remember when making a backup is to sanitize the data, when needed. Usually we need to take dump from the database only for the development/testing purposes so normally you should sanitize the data. We are using Django sanitized dump for sanitizing data so check the most recent instructions from the vendor.

When running backup for the staging/testing/development purposes, you should exclude few tables to limit the size of the backup. So remember to add --exclude-table-data 'public.auditlog_logentry' --exclude-table-data 'public.batchrun_jobrunlog*' --exclude-table-data 'public.django_q_task' to pg_dump command.

pg_dump mvj_api_prod | gzip > mvj-api-prod_$(date +%Y%m%d%H%m).sql.gz

To restore dump run psql -f mvj-api-prod-DATE_HERE.sql ${DATABASE_URL/postgis/postgres}.

For machine-to-machine integration Django REST Framework's token based authentication is in use. To create token for the user you need to run python manage.py drf_create_token <username> command. If you need to renew token, then you need to append -r option to the command. To make things secure robot user should not have password set (aka cannot log in with browser) and it should only have access to the certain API endpoint and nothing else.

From client perspective token is sent in HTTP headers (Authorization header) with Token keyword. As an example Authorization: Token abcdefghijklmnopqrstuvwxyz1234567890.