Fix DELAYED_EVAL not working for other than pavics compose.sh (#288)

## Fixes There are other scripts sourcing `default.env` and `env.local` and all those scripts have to expand the vars in `DELAYED_EVAL` list to have their actual values. Only scripts using the 3 variables in `DELAYED_EVAL` list are broken. `DELAYED_EVAL` was previously introduced in PR #272. ### Sample errors `fix-geoserver-data-dir-perm` (called at the end of `pavics-compose.sh`): ``` fix GeoServer data dir permission on first run only, when data dir do not exist yet. + DATA_DIR='${DATA_PERSIST_ROOT}/geoserver' + '[' -n ] + docker run --rm --name fix-geoserver-data-dir-perm --volume '${DATA_PERSIST_ROOT}/geoserver:/datadir' --env FIRST_RUN_ONLY bash:5.1.4 bash -xc 'if [ -z "$FIRST_RUN_ONLY" -o ! -f /datadir/global.xml ]; \ then chown -R 1000:10001 /datadir; else echo "No execute."; fi' docker: Error response from daemon: create ${DATA_PERSIST_ROOT}/geoserver: "${DATA_PERSIST_ROOT}/geoserver" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path. ``` `trigger-deploy-notebook` (broke notebook deploy job): ``` + TMP_SCRIPT=/tmp/notebookdeploy/notebookdeploy.XXXXXXIfafFK/deploy-notebook + cat + chmod a+x /tmp/notebookdeploy/notebookdeploy.XXXXXXIfafFK/deploy-notebook + docker run --rm --name deploy_tutorial_notebooks -u root -v /tmp/notebookdeploy/notebookdeploy.XXXXXXIfafFK/deploy-notebook:/deploy-notebook:ro -v /tmp/notebookdeploy/notebookdeploy.XXXXXXIfafFK/tutorial-notebooks:/tutorial-notebooks:ro -v '${DATA_PERSIST_ROOT}/jupyterhub_user_data:/notebook_dir:rw' --entrypoint /deploy-notebook bash:5.1.4 docker: Error response from daemon: create ${DATA_PERSIST_ROOT}/jupyterhub_user_data: "${DATA_PERSIST_ROOT}/jupyterhub_user_data" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path. ``` ### Explanation of the fix All scripts have to remember to call function `process_delayed_eval` in order to obtain the real value of each vars in `DELAYED_EVAL` list. Centralized all logic about reading configs (config files reading order, remember to call `process_delayed_eval`) to avoid mistake and to ease updating logic in the future. Too many scripts were reading the configs themselves and some are not doing it properly, ex: forgot to hide password when reading `env.local`. ### All scripts should do this going forward ```sh # Set variable COMPOSE_DIR to the dir containing pavics-compose.sh and docker-compose.yml. # Source the script providing function read_configs. # read_configs uses COMPOSE_DIR to find default.env and env.local. . $COMPOSE_DIR/read-configs.include.sh # Call function read_configs to read the various config files in the appropriate order and process delayed eval vars properly. read_configs ```
bird-house · Feb 13, 2023 · c88e68f · c88e68f
2 parents bcd1d88 + b78847a
commit c88e68f
Show file tree

Hide file tree

Showing 24 changed files with 278 additions and 142 deletions.
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 1.23.0
+current_version = 1.23.1
 commit = True
 tag = False
 tag_name = {new_version}
@@ -30,11 +30,11 @@ search = {current_version}
 replace = {new_version}
 
 [bumpversion:file:RELEASE.txt]
-search = {current_version} 2023-02-10T16:31:08Z
+search = {current_version} 2023-02-13T18:31:02Z
 replace = {new_version} {utcnow:%Y-%m-%dT%H:%M:%SZ}
 
 [bumpversion:part:releaseTime]
-values = 2023-02-10T16:31:08Z
+values = 2023-02-13T18:31:02Z
 
 [bumpversion:file(version):birdhouse/config/canarie-api/docker_configuration.py.template]
 search = 'version': '{current_version}'

diff --git a/CHANGES.md b/CHANGES.md
@@ -16,6 +16,67 @@
 
 [//]: # (list changes here, using '-' for each new entry, remove this when items are added)
 
+[1.23.1](https://github.com/bird-house/birdhouse-deploy/tree/1.23.1) (2023-02-13)
+------------------------------------------------------------------------------------------------------------------
+
+[//]: # (list changes here, using '-' for each new entry, remove this when items are added)
+
+## Fixes
+- Vars in `DELAYED_EVAL` list are not expanded properly outside of `pavics-compose.sh`
+
+  There are other scripts sourcing `default.env` and `env.local` and all those
+  scripts have to expand the vars in `DELAYED_EVAL` list to have their actual
+  values.
+
+  Only scripts using the 3 variables in `DELAYED_EVAL` list are broken.
+
+  `DELAYED_EVAL` was previously introduced in PR https://github.com/bird-house/birdhouse-deploy/pull/272.
+
+  **Sample errors**
+
+  `fix-geoserver-data-dir-perm` (called at the end of `pavics-compose.sh`):
+  ```
+  fix GeoServer data dir permission on first run only, when data dir do not exist yet.
+  + DATA_DIR='${DATA_PERSIST_ROOT}/geoserver'
+  + '[' -n  ]
+  + docker run --rm --name fix-geoserver-data-dir-perm --volume '${DATA_PERSIST_ROOT}/geoserver:/datadir' --env FIRST_RUN_ONLY bash:5.1.4 bash -xc 'if [ -z "$FIRST_RUN_ONLY" -o ! -f /datadir/global.xml ]; \
+      then chown -R 1000:10001 /datadir; else echo "No execute."; fi'
+  docker: Error response from daemon: create ${DATA_PERSIST_ROOT}/geoserver: "${DATA_PERSIST_ROOT}/geoserver" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
+  ```
+
+  `trigger-deploy-notebook` (broke notebook deploy job):
+  ```
+  + TMP_SCRIPT=/tmp/notebookdeploy/notebookdeploy.XXXXXXIfafFK/deploy-notebook
+  + cat
+  + chmod a+x /tmp/notebookdeploy/notebookdeploy.XXXXXXIfafFK/deploy-notebook
+  + docker run --rm --name deploy_tutorial_notebooks -u root -v /tmp/notebookdeploy/notebookdeploy.XXXXXXIfafFK/deploy-notebook:/deploy-notebook:ro -v /tmp/notebookdeploy/notebookdeploy.XXXXXXIfafFK/tutorial-notebooks:/tutorial-notebooks:ro -v '${DATA_PERSIST_ROOT}/jupyterhub_user_data:/notebook_dir:rw' --entrypoint /deploy-notebook bash:5.1.4
+  docker: Error response from daemon: create ${DATA_PERSIST_ROOT}/jupyterhub_user_data: "${DATA_PERSIST_ROOT}/jupyterhub_user_data" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
+  ```
+
+  **Explanation of the fix**
+
+  All scripts have to remember to call function `process_delayed_eval` in order
+  to obtain the real value of each vars in `DELAYED_EVAL` list.
+
+  Centralized all logic about reading configs (config files reading order,
+  remember to call `process_delayed_eval`) to avoid mistake and to ease updating
+  logic in the future.  Too many scripts were reading the configs  themselves and
+  some are not doing it properly, ex: forgot to hide password when reading
+  `env.local`.
+
+  **All scripts should do this going forward**
+
+  ```sh
+  # Set variable COMPOSE_DIR to the dir containing pavics-compose.sh and docker-compose.yml.
+
+  # Source the script providing function read_configs.
+  # read_configs uses COMPOSE_DIR to find default.env and env.local.
+  . $COMPOSE_DIR/read-configs.include.sh
+
+  # Call function read_configs to read the various config files in the appropriate order and process delayed eval vars properly.
+  read_configs
+  ```
+
 [1.23.0](https://github.com/bird-house/birdhouse-deploy/tree/1.23.0) (2023-02-10)
 ------------------------------------------------------------------------------------------------------------------
 

diff --git a/Makefile b/Makefile
@@ -1,7 +1,7 @@
 # Generic variables
 override SHELL       := bash
 override APP_NAME    := birdhouse-deploy
-override APP_VERSION := 1.23.0
+override APP_VERSION := 1.23.1
 
 # utility to remove comments after value of an option variable
 override clean_opt = $(shell echo "$(1)" | $(_SED) -r -e "s/[ '$'\t'']+$$//g")

diff --git a/README.rst b/README.rst
@@ -14,13 +14,13 @@ for a full-fledged production platform.
     * - releases
       - | |latest-version| |commits-since|
 
-.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/1.23.0.svg
+.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/1.23.1.svg
     :alt: Commits since latest release
-    :target: https://github.com/bird-house/birdhouse-deploy/compare/1.23.0...master
+    :target: https://github.com/bird-house/birdhouse-deploy/compare/1.23.1...master
 
-.. |latest-version| image:: https://img.shields.io/badge/tag-1.23.0-blue.svg?style=flat
+.. |latest-version| image:: https://img.shields.io/badge/tag-1.23.1-blue.svg?style=flat
     :alt: Latest Tag
-    :target: https://github.com/bird-house/birdhouse-deploy/tree/1.23.0
+    :target: https://github.com/bird-house/birdhouse-deploy/tree/1.23.1
 
 .. |readthedocs| image:: https://readthedocs.org/projects/birdhouse-deploy/badge/?version=latest
     :alt: ReadTheDocs Build Status (latest version)

diff --git a/RELEASE.txt b/RELEASE.txt
@@ -1 +1 @@
-1.23.0 2023-02-10T16:31:08Z
+1.23.1 2023-02-13T18:31:02Z
diff --git a/birdhouse/config/canarie-api/docker_configuration.py.template b/birdhouse/config/canarie-api/docker_configuration.py.template
@@ -17,8 +17,8 @@ SERVICES = {
         'info': {
             'name': 'Node',
             'synopsis': 'Nodes are data, compute and index endpoints accessed through the PAVICS platform or external clients. The Node service is the backend that allows: data storage, harvesting, indexation and discovery of local and federated data; authentication and authorization; server registration and management. Node service is therefore composed of several other services.',
-            'version': '1.23.0',
-            'releaseTime': '2023-02-10T16:31:08Z',
+            'version': '1.23.1',
+            'releaseTime': '2023-02-13T18:31:02Z',
             'institution': 'Ouranos',
             'researchSubject': 'Climatology',
             'supportEmail': '${SUPPORT_EMAIL}',
@@ -242,8 +242,8 @@ PLATFORMS = {
         'info': {
             'name': 'PAVICS',
             'synopsis': 'The PAVICS (Power Analytics for Visualization of Climate Science) platform is a collection of climate analysis services served through Open Geospatial Consortium (OGC) protocols. These services include data access, processing and visualization. Both data and algorithms can be accessed either programmatically, through OGC-compliant clients such as QGIS or ArcGIS, or a custom web interface.',
-            'version': '1.23.0',
-            'releaseTime': '2023-02-10T16:31:08Z',
+            'version': '1.23.1',
+            'releaseTime': '2023-02-13T18:31:02Z',
             'institution': 'Ouranos',
             'researchSubject': 'Climatology',
             'supportEmail': '${SUPPORT_EMAIL}',

diff --git a/birdhouse/deployment/certbotwrapper b/birdhouse/deployment/certbotwrapper
@@ -50,21 +50,10 @@ THIS_FILE="`realpath "$0"`"
 THIS_DIR="`dirname "$THIS_FILE"`"
 SAVED_PWD="`pwd`"
 
-# Default values
-. $THIS_DIR/../default.env
+. "$THIS_DIR/../read-configs.include.sh"
 
-ENV_LOCAL_FILE="$THIS_DIR/../env.local"
-
-set +x  # hide password in env.local
-
-if [ -e "$ENV_LOCAL_FILE" ]; then
-    # Override default values
-    . "$ENV_LOCAL_FILE"
-else
-    echo "WARNING: '$ENV_LOCAL_FILE' not found."
-fi
-
-set -x  # resume logging/tracing
+# Get PAVICS_FQDN_PUBLIC, PAVICS_FQDN, SUPPORT_EMAIL, SSL_CERTIFICATE.
+read_configs
 
 CERT_DOMAIN="$PAVICS_FQDN_PUBLIC"
 if [ -z "$CERT_DOMAIN" ]; then

diff --git a/birdhouse/deployment/deploy.sh b/birdhouse/deployment/deploy.sh
@@ -105,10 +105,10 @@ cd $COMPOSE_DIR
 START_TIME="`date -Isecond`"
 echo "deploy START_TIME=$START_TIME"
 
-. $COMPOSE_DIR/default.env
+. "$COMPOSE_DIR/read-configs.include.sh"
 
 # Read AUTODEPLOY_EXTRA_REPOS
-. $ENV_LOCAL_FILE
+read_basic_configs_only
 
 set -x
 
@@ -128,12 +128,7 @@ done
 
 cd $COMPOSE_DIR
 
-. ./default.env
-
-set +x  # hide passwd in env.local in logs
-# reload again after default.env since env.local can override default.env
-. $ENV_LOCAL_FILE
-set -x
+read_basic_configs_only
 
 # stop all to force reload any changed config that are volume-mount into the containers
 ./pavics-compose.sh stop
@@ -169,13 +164,11 @@ done
 cd $COMPOSE_DIR
 
 # reload again after git pull because this file could be changed by the pull
-. ./default.env
+. "$COMPOSE_DIR/read-configs.include.sh"
 
-set +x  # hide passwd in env.local in logs
 # reload again after default.env since env.local can override default.env
 # (ex: JUPYTERHUB_USER_DATA_DIR)
-. $ENV_LOCAL_FILE
-set -x
+read_basic_configs_only
 
 # restart everything, only changed containers will be destroyed and recreated
 ./pavics-compose.sh up -d

diff --git a/birdhouse/deployment/fix-geoserver-data-dir-perm b/birdhouse/deployment/fix-geoserver-data-dir-perm
@@ -14,15 +14,11 @@ THIS_DIR="`dirname "$THIS_FILE"`"
 # Go to repo root.
 cd $THIS_DIR/../..
 
-# Default values
+. birdhouse/read-configs.include.sh
+
 # Get BASH_IMAGE
 # Get GEOSERVER_DATA_DIR
-. birdhouse/default.env
-
-if [ -e "birdhouse/env.local" ]; then
-    # Override default values
-    . birdhouse/env.local
-fi
+read_configs
 
 set -x
 

diff --git a/birdhouse/deployment/fix-write-perm b/birdhouse/deployment/fix-write-perm
@@ -43,13 +43,10 @@ THIS_DIR="`dirname "$THIS_FILE"`"
 # Go to repo root.
 cd $THIS_DIR/../..
 
-# Default values
-. birdhouse/default.env
+. birdhouse/read-configs.include.sh
 
-if [ -e "birdhouse/env.local" ]; then
-    # Override default values
-    . birdhouse/env.local
-fi
+# Get GEOSERVER_DATA_DIR, JUPYTERHUB_USER_DATA_DIR, MAGPIE_PERSIST_DIR
+read_configs
 
 DEFAULT_EXTRA_DATA_DIR="$GEOSERVER_DATA_DIR/ $JUPYTERHUB_USER_DATA_DIR/ \
     $MAGPIE_PERSIST_DIR/"

diff --git a/birdhouse/deployment/install-deploy-notebook b/birdhouse/deployment/install-deploy-notebook
@@ -29,12 +29,10 @@ if [ ! -e "$REPO_ROOT/birdhouse/deployment/trigger-deploy-notebook" ]; then
     exit 2
 fi
 
-. "$REPO_ROOT/birdhouse/default.env"
+. "$REPO_ROOT/birdhouse/read-configs.include.sh"
 
-if [ -f "$REPO_ROOT/birdhouse/env.local" ]; then
-    # allow override of JUPYTERHUB_USER_DATA_DIR
-    . "$REPO_ROOT/birdhouse/env.local"
-fi
+# Get JUPYTERHUB_USER_DATA_DIR
+read_configs
 
 set -x
 

diff --git a/birdhouse/deployment/trigger-deploy-notebook b/birdhouse/deployment/trigger-deploy-notebook
@@ -35,13 +35,12 @@ set -x
 if [ -z "$COMPOSE_DIR" ]; then
     COMPOSE_DIR="$(dirname -- "$(dirname -- "$(realpath "$0")")")"
 fi
-if [ -e "$COMPOSE_DIR/default.env" ]; then
-    . "$COMPOSE_DIR/default.env"  # default JUPYTERHUB_USER_DATA_DIR
-fi
-if [ -e "$COMPOSE_DIR/env.local" ]; then
-    set +x  # do not leak password in logs
-    . "$COMPOSE_DIR/env.local"  # optional override JUPYTERHUB_USER_DATA_DIR
-    set -x
+
+if [ -f "$COMPOSE_DIR/read-configs.include.sh" ]; then
+    . "$COMPOSE_DIR/read-configs.include.sh"
+
+    # Get JUPYTERHUB_USER_DATA_DIR
+    read_configs
 fi
 
 NOTEBOOK_DIR_MNT="/notebook_dir"

diff --git a/birdhouse/deployment/triggerdeploy.sh b/birdhouse/deployment/triggerdeploy.sh
@@ -173,10 +173,10 @@ START_TIME="`date -Isecond`"
 echo "==========
 triggerdeploy START_TIME=$START_TIME"
 
-. $COMPOSE_DIR/default.env
+. "$COMPOSE_DIR/read-configs.include.sh"
 
 # Read AUTODEPLOY_EXTRA_REPOS
-. $ENV_LOCAL_FILE
+read_basic_configs_only
 
 set -x
 

diff --git a/birdhouse/pavics-compose.sh b/birdhouse/pavics-compose.sh
@@ -101,38 +101,11 @@ cd $(dirname $(readlink -f $0 || realpath $0))
 # container and manually from the host.
 COMPOSE_DIR="`pwd`"
 
-. ./default.env
+. "$COMPOSE_DIR/read-configs.include.sh"
+read_configs
 
-# we source local configs, if present
-# we don't use usual .env filename, because docker-compose uses it
-[ -f env.local ] && . ./env.local
 . ./scripts/get-components-json.include.sh
 
-for adir in ${EXTRA_CONF_DIRS}; do
-  if [ ! -e "$adir" ]; then
-    # Do not exit to not break unattended autodeploy since no human around to
-    # fix immediately.
-    # The new adir with typo will not be active but at least all the existing
-    # will still work.
-    echo "WARNING: '$adir' in EXTRA_CONF_DIRS does not exist" 1>&2
-  fi
-  COMPONENT_DEFAULT_ENV="$adir/default.env"
-  if [ -f "$COMPONENT_DEFAULT_ENV" ]; then
-    echo "reading '$COMPONENT_DEFAULT_ENV'"
-    . "$COMPONENT_DEFAULT_ENV"
-  fi
-done
-
-# Re-read env.local to make sure it can override ALL defaults from all
-# components.
-[ -f env.local ] && . ./env.local
-
-for i in ${DELAYED_EVAL}; do
-  v="`eval "echo \\$${i}"`"
-  eval 'export ${i}="`eval "echo ${v}"`"'
-  echo "delayed eval '`env |grep ${i}=`'"
-done
-
 for i in ${VARS}
 do
   v="${i}"