Merge pull request #1572 from open-zaak/feature/setup-config

Feature/setup config

Dockerfile

@@ -5,6 +5,7 @@ RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-reco
         pkg-config \
         build-essential \
         libpq-dev \
+        git \
     && rm -rf /var/lib/apt/lists/*
 
 WORKDIR /app
@@ -65,6 +66,7 @@ COPY ./bin/uninstall_adfs.sh \
     ./bin/uninstall_django_auth_adfs_db.sql \
     ./bin/dump_configuration.sh \
     /app/bin/
+COPY ./bin/setup_configuration.sh /setup_configuration.sh
 
 RUN mkdir /app/log /app/config /app/media /app/private-media
 # prevent writing to the container layer, which would degrade performance.

bin/setup_configuration.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+# setup initial configuration using environment variables
+# Run this script from the root of the repository
+
+set -e
+
+# wait for required services
+${SCRIPTPATH}/wait_for_db.sh
+
+src/manage.py migrate
+src/manage.py setup_configuration --no-selftest

docker-compose.yml

@@ -40,11 +40,31 @@ services:
       - CELERY_RESULT_BACKEND=redis://redis:6379/1
       - CELERY_LOGLEVEL=DEBUG
       - CELERY_WORKER_CONCURRENCY=${CELERY_WORKER_CONCURRENCY:-4}
+      # setup_configuration env vars
+      - OPENZAAK_DOMAIN=web:8000
+      - OPENZAAK_ORGANIZATION=OZ
+      - NOTIF_OPENZAAK_CLIENT_ID=notif-client-id
+      - NOTIF_OPENZAAK_SECRET=notif-secret
+      - NOTIF_API_ROOT=http://notifications:8000/api/v1/
+      - OPENZAAK_NOTIF_CLIENT_ID=oz-client-id
+      - OPENZAAK_NOTIF_SECRET=oz-secret
+      - DEMO_CONFIG_ENABLE=yes
+      - DEMO_CLIENT_ID=demo-client-id
+      - DEMO_SECRET=demo-secret
     ports:
       - 8000:8000
     volumes: &app-volumes
       - media:/app/media  # Shared media volume to get access to saved OAS files
       - private-media:/app/private-media
+    depends_on:
+      web-init:
+        condition: service_completed_successfully
+
+  web-init:
+    build: .
+    environment: *app-env
+    command: /setup_configuration.sh
+    volumes: *app-volumes
     depends_on:
       - db
       - redis

docs/installation/config/env_config.md

@@ -223,6 +223,13 @@ The following parameters control the CORS policy.
   included. The value of this variable is added to these already included headers.
   Defaults to an empty list.
 
+### Initial configuration
+
+Open Zaak supports `setup_configuration` management command, which allows configuration via
+environment variables. 
+All these environment variables are described at :ref:`command line <installation_configuration_cli>`.
+
+
 ## Specifying the environment variables
 
 There are two strategies to specify the environment variables:

docs/installation/config/openzaak_config_cli.rst

@@ -7,9 +7,8 @@ Open Zaak configuration (CLI)
 After deploying Open Zaak, it needs to be configured to be fully functional. The
 command line tool ``setup_configuration`` assist with this configuration:
 
-* It favours explicit configuration via options - you can integrate this with your
-  infrastructure tooling such as init containers and/or Kubernetes Jobs
-* In interactive mode, you receive prompts to fill out the requested information
+* It uses environment variables for all configuration choices, therefore you can integrate this with your
+  infrastructure tooling such as init containers and/or Kubernetes Jobs.
 * The command can self-test the configuration to detect problems early on
 
 You can get the full command documentation with:
@@ -18,31 +17,75 @@ You can get the full command documentation with:
 
     src/manage.py setup_configuration --help
 
-.. note::
-
-    For more explanation/feedback, run the command with increased verbosity:
-
-    .. code-block:: bash
-
-        src/manage.py setup_configuration --verbosity 2
-
-
 .. warning:: This command is declarative - if configuration is manually changed after
    running the command and you then run the exact same command again, the manual
    changes will be reverted.
 
 Preparation
 ===========
 
-You should prepare the following information:
+The command executes the list of pluggable configuration steps, and each step
+has required specific environment variables, that should be prepared.
+Here is the description of all available configuration steps and the environment variables, 
+used by each step. 
+
+Sites configuration
+------------------------
+
+Configure the domain where Open Zaak is hosted
+
+* ``SITES_CONFIG_ENABLE``: enable Site configuration. Defaults to ``True``.
+* ``OPENZAAK_DOMAIN``:  a ``[host]:[port]`` or ``[host]`` value. Required.
+* ``OPENZAAK_ORGANIZATION``: name of Open Zaak organization. Required.
+
+Notification authorization configuration
+----------------------------------------
+
+Open Notificaties uses Open Zaak Autorisaties API to check authorizations
+of its consumers, therefore Open Notificaties should be able to request Open Zaak
+
+* ``NOTIF_OPENZAAK_CONFIG_ENABLE``: enable Notification credentials configuration. Defaults
+  to ``True``.
+* ``NOTIF_OPENZAAK_CLIENT_ID``: a client id, which Open Notificaties uses to request
+  Open Zaak, for example, ``open-notificaties``. Required.
+* ``NOTIF_OPENZAAK_SECRET``: some random string. Required.
+
+Notification configuration
+--------------------------
+
+Open Zaak publishes notifications to the Open Notificaties.
+
+* ``OPENZAAK_NOTIF_CONFIG_ENABLE``: enable Notification configuration. Defaults to ``True``.
+* ``NOTIF_API_ROOT``: full URL to the Notificaties API root, for example
+  ``https://notificaties.gemeente.local/api/v1/``. Required.
+* ``NOTIF_API_OAS``: full URL to the Notificaties OpenAPI specification in YAML format.
+* ``OPENZAAK_NOTIF_CLIENT_ID``: a client id, which Open Zaak uses to request Open Notificaties,
+  for example, ``open-zaak``. Required.
+* ``OPENZAAK_NOTIF_SECRET``: some random string. Required.
+
+Selectielijst configuration
+---------------------------
 
-* organization name, e.g. ``ACME``
-* domain name where Open Zaak is deployed, e.g. ``open-zaak.gemeente.local``
-* Notifications API root, e.g. ``https://notificaties.gemeente.local/api/v1/``
-* A Client ID for Open Zaak to the Notifications API, e.g. ``open-zaak-acme``
-* A Client Secret for Open Zaak to the Notifications API, e.g. ``insecure-nrc-secret``
-* A Client ID for the Notifications API to Open Zaak, e.g. ``notificaties-api-acme``
-* A Client Secret for the Notifications API to Open Zaak, e.g. ``insecure-oz-secret``
+Open Zaak requests Selectielijst API in the Catalogi API component.
+The Selectielijst API is not expected to require any authentication.
+
+* ``OPENZAAK_SELECTIELIJST_CONFIG_ENABLE``: enable Selectielijst configuration. Defaults to ``True``.
+* ``SELECTIELIJST_API_ROOT``: full url to the Selectielijst API root. Defaults to
+  ``https://selectielijst.openzaak.nl/api/v1/``
+* ``SELECTIELIJST_API_OAS``: full url to the Selectielijst OpenAPI specification in YAML format. Defaults to
+  ``https://selectielijst.openzaak.nl/api/v1/schema/openapi.yaml``
+* ``SELECTIELIJST_ALLOWED_YEARS``: years, for which process types can be used. Defaults to ``[2017, 2020]``.
+* ``SELECTIELIJST_DEFAULT_YEAR`` = the default year from which process types will be used. Defaults to `2020`.
+
+Demo user configuration
+-----------------------
+
+Demo user can be created to check if Open Zaak APIs work. It has superuser permissions, 
+so its creation is not recommended on production environment.
+
+* ``DEMO_CONFIG_ENABLE``: enable demo user configuration. Defaults to the value of the ``DEBUG`` setting. 
+* ``DEMO_CLIENT_ID``: demo client id. Required.
+* ``DEMO_SECRET``: demo secret. Required.
 
 .. note:: You can generate these Client IDs and Secrets using any password generation
    tool, as long as you configure the same values in the Notifications API.
@@ -58,41 +101,29 @@ tested. For all the self-tests to succeed, it's important that the
 :ref:`Notifications API is configured <installation_configuration_notificaties_api>`
 correctly before calling this command.
 
-Alternatively, you can skip the self-tests by using the ``--no-self-test`` flag.
+.. code-block:: bash
+
+    src/manage.py setup_configuration
+
 
-The example command uses the example values from the preparation above:
+Alternatively, you can skip the self-tests by using the ``--no-selftest`` flag.
 
 .. code-block:: bash
 
-    src/manage.py setup_configuration \
-        -v 2 \
-        --organization ACME \
-        --domain open-zaak.gemeente.local \
-        --create-notifications-api-app \
-        --notifications-api-app-client-id notificaties-api-acme \
-        --notifications-api-app-secret insecure-oz-secret \
-        --notifications-api-root https://notificaties.gemeente.local/api/v1/ \
-        --notifications-api-client-id open-zaak-acme \
-        --notifications-api-secret insecure-nrc-secret \
-        --self-test \
-        --send-test-notification
+    src/manage.py setup_configuration --no-self-test
 
-.. note:: Due to a cache-bug in the underlying framework, you need to restart all
-   replicas for part of this change to take effect everywhere.
 
-.. note:: You can output the results as JSON which your configuration management can
-   then pick up and process:
+``setup_configuration`` command checks if the configuration already exists before changing it.
+If you want to change some of the values of the existing configuration you can use ``--overwrite`` flag.
+
+.. code-block:: bash
 
-   .. code-block:: bash
+    src/manage.py setup_configuration --overwrite
 
-      export LOG_LEVEL=CRITICAL
-      src/manage.py setup_configuration \
-        ...\
-        --skip-checks \
-        --json
 
-   The ``LOG_LEVEL`` environment variable ensures your output is not cluttered with
-   logs, while ``--skip-checks`` prevents system check output from appearing.
+.. note:: Due to a cache-bug in the underlying framework, you need to restart all
+   replicas for part of this change to take effect everywhere.
+
 
 Register notification channels
 ------------------------------

requirements/base.in

@@ -47,6 +47,7 @@ humanize
 drc-cmis
 djangorestframework-inclusions
 drf-spectacular
+django-setup-configuration
 
 # WSGI servers & monitoring - production oriented
 uwsgi

requirements/base.txt

@@ -100,6 +100,7 @@ django==3.2.23
     #   django-relativedelta
     #   django-rest-framework-condition
     #   django-sendfile2
+    #   django-setup-configuration
     #   django-simple-certmanager
     #   django-sniplates
     #   django-solo
@@ -159,6 +160,8 @@ django-rest-framework-condition==0.1.1
     # via commonground-api-common
 django-sendfile2==0.7.0
     # via django-privates
+django-setup-configuration==0.1.0
+    # via -r requirements/base.in
 django-simple-certmanager==1.3.0
     # via zgw-consumers
 django-sniplates==0.7.0

requirements/ci.txt

@@ -140,6 +140,7 @@ django==3.2.23
     #   django-relativedelta
     #   django-rest-framework-condition
     #   django-sendfile2
+    #   django-setup-configuration
     #   django-simple-certmanager
     #   django-sniplates
     #   django-solo
@@ -211,6 +212,8 @@ django-sendfile2==0.7.0
     # via
     #   -r requirements/base.txt
     #   django-privates
+django-setup-configuration==0.1.0
+    # via -r requirements/base.txt
 django-simple-certmanager==1.3.0
     # via
     #   -r requirements/base.txt

requirements/dev.txt

@@ -162,6 +162,7 @@ django==3.2.23
     #   django-relativedelta
     #   django-rest-framework-condition
     #   django-sendfile2
+    #   django-setup-configuration
     #   django-silk
     #   django-simple-certmanager
     #   django-sniplates
@@ -238,6 +239,8 @@ django-sendfile2==0.7.0
     # via
     #   -r requirements/ci.txt
     #   django-privates
+django-setup-configuration==0.1.0
+    # via -r requirements/ci.txt
 django-silk==4.3.0
     # via -r requirements/dev.in
 django-simple-certmanager==1.3.0

src/openzaak/conf/includes/base.py

@@ -156,6 +156,7 @@
     "mozilla_django_oidc",
     "mozilla_django_oidc_db",
     "log_outgoing_requests",
+    "django_setup_configuration",
     # Project applications.
     "openzaak",
     "openzaak.accounts",
@@ -708,6 +709,18 @@
 #
 LOG_OUTGOING_REQUESTS_DB_SAVE = config("LOG_OUTGOING_REQUESTS_DB_SAVE", default=False)
 
+
+#
+# Django setup configuration
+#
+SETUP_CONFIGURATION_STEPS = [
+    "openzaak.config.bootstrap.site.SiteConfigurationStep",
+    "openzaak.config.bootstrap.notifications.AuthNotificationStep",
+    "openzaak.config.bootstrap.notifications.NotificationsAPIConfigurationStep",
+    "openzaak.config.bootstrap.selectielijst.SelectielijstAPIConfigurationStep",
+    "openzaak.config.bootstrap.demo.DemoUserStep",
+]
+
 #
 # OpenZaak configuration
 #
@@ -730,3 +743,40 @@
 
 # Name of the cache used to store responses for requests made when importing catalogi
 IMPORT_REQUESTS_CACHE_NAME = config("IMPORT_REQUESTS_CACHE_NAME", "import_requests")
+
+# Settings for setup_configuration command
+# sites config
+SITES_CONFIG_ENABLE = config("SITES_CONFIG_ENABLE", default=True)
+OPENZAAK_ORGANIZATION = config("OPENZAAK_ORGANIZATION", "")
+# notif -> OZ config
+NOTIF_OPENZAAK_CONFIG_ENABLE = config("NOTIF_OPENZAAK_CONFIG_ENABLE", default=True)
+NOTIF_OPENZAAK_CLIENT_ID = config("NOTIF_OPENZAAK_CLIENT_ID", "")
+NOTIF_OPENZAAK_SECRET = config("NOTIF_OPENZAAK_SECRET", "")
+# OZ -> notif config
+OPENZAAK_NOTIF_CONFIG_ENABLE = config("OPENZAAK_NOTIF_CONFIG_ENABLE", default=True)
+NOTIF_API_ROOT = config("NOTIF_API_ROOT", "")
+if NOTIF_API_ROOT and not NOTIF_API_ROOT.endswith("/"):
+    NOTIF_API_ROOT = f"{NOTIF_API_ROOT.strip()}/"
+NOTIF_API_OAS = config("NOTIF_API_OAS", default=f"{NOTIF_API_ROOT}schema/openapi.yaml")
+OPENZAAK_NOTIF_CLIENT_ID = config("OPENZAAK_NOTIF_CLIENT_ID", "")
+OPENZAAK_NOTIF_SECRET = config("OPENZAAK_NOTIF_SECRET", "")
+# Selectielijst config
+OPENZAAK_SELECTIELIJST_CONFIG_ENABLE = config(
+    "OPENZAAK_SELECTIELIJST_CONFIG_ENABLE", default=True
+)
+SELECTIELIJST_API_ROOT = config(
+    "SELECTIELIJST_API_ROOT", default="https://selectielijst.openzaak.nl/api/v1/"
+)
+if SELECTIELIJST_API_ROOT and not SELECTIELIJST_API_ROOT.endswith("/"):
+    SELECTIELIJST_API_ROOT = f"{SELECTIELIJST_API_ROOT.strip()}/"
+SELECTIELIJST_API_OAS = config(
+    "SELECTIELIJST_API_OAS", default=f"{SELECTIELIJST_API_ROOT}schema/openapi.yaml"
+)
+SELECTIELIJST_ALLOWED_YEARS = config(
+    "SELECTIELIJST_ALLOWED_YEARS", default=[2017, 2020]
+)
+SELECTIELIJST_DEFAULT_YEAR = config("SELECTIELIJST_DEFAULT_YEAR", default=2020)
+# Demo user config
+DEMO_CONFIG_ENABLE = config("DEMO_CONFIG_ENABLE", default=DEBUG)
+DEMO_CLIENT_ID = config("DEMO_CLIENT_ID", "")
+DEMO_SECRET = config("DEMO_SECRET", "")

src/openzaak/config/bootstrap/__init__.py

@@ -36,7 +36,12 @@
       scopes so that the Notifications API can query the applications/permissions of
       a given client ID. (SCOPE_AUTORISATIES_LEZEN)
 
-- [ ] Create a demo API token and make a test request?
+- [x] Selectielijst API
+     - Open Zaak uses Selectielijst API for the Catalogi component. Specify the API
+       root of the Selectielijst API and default and allowed years to use. The API
+       is expected not to require any authentication.
+
+- [x] Create a demo API token
+
 
-- [ ] Configure NLX
 """