Hardcoded Database Connections and Init Container Dependencies

[emmanuelmathot]

Description

Found two critical bugs in the Helm charts that can prevent proper deployment in certain configurations:

1. Hardcoded pgbouncer connection in pgstacbootstrap job

   • In helm-chart/eoapi/templates/pgstacboostrap/job.yaml (starting line 23), there's an initContainer that waits for eoapi-pgbouncer with a hardcoded address:

   initContainers:
     - name: wait-for-db
       image: busybox
       command:
         {{ if .Values.testing }}
         ['sh', '-c', 'until nc -z {{ $.Release.Name }}-pgbouncer 5432; do echo waiting for db; sleep 10; done;']
         {{ else }}
         ['sh', '-c', 'until nc -z eoapi-pgbouncer 5432; do echo waiting for db; sleep 10; done;']
         {{ end }}

2. Waiting for pgstacbootstrap job in service deployments

   • In helm-chart/eoapi/templates/services/deployment.yaml (around line 41), there's an initContainer that waits for the pgstacbootstrap job to complete:

   initContainers:
   - name: wait-for-pgstacbootstrap
     image: bitnami/kubectl:latest
     command:
     - /bin/sh
     - -c
     - |
       echo "Waiting for pgstacbootstrap job to complete..."
       while ! kubectl -n {{ $.Release.Namespace }} wait --for=condition=complete job/pgstacbootstrap --timeout=5s; do
         echo "pgstacbootstrap job not completed yet. Checking again in 10 seconds..."
         sleep 10
       done
       echo "pgstacbootstrap job completed successfully"

Problems

1. For the pgbouncer connection:

   • What if pgbouncer isn't being used? Some deployments might use a direct database connection or a different connection pooler.
   • The hardcoded address doesn't work if the PostgreSQL endpoint is at a different address.
   • No flexibility to configure the connection details through Helm values.

2. For the job dependency:

   • If the pgstacbootstrap job never completes, the services will never start.
   • This creates a hard dependency that can be problematic in some deployment scenarios.
   • As discussed in issue #203, checking job completion in init containers is not the recommended approach.

Proposed Solution

As mentioned in issue #203, we should use Helm hooks instead of init containers for managing these dependencies:

1. For the database connection:

   • Make the database connection configurable through values in the chart.
   • Add a template function to determine the correct database endpoint based on the deployment configuration (Issue TBC)

2. For the job dependency:

   • Use Helm hooks to establish the proper execution order.
   • Specifically, use helm.sh/hook: post-install,post-upgrade for the pgstacbootstrap job.
   • Use helm.sh/hook-weight to ensure execution order.
   • This would run the job after Helm installs or upgrades the release, and the services won't need to wait for it.
   • See Avoid checking job completion in init containers #203

Impact

These issues can cause deployment failures in the following scenarios:

• Using a different database connection than pgbouncer
• Using a different database host name
• When the pgstacbootstrap job fails or hangs indefinitely

Fixing these issues will improve the flexibility and reliability of the helm charts, particularly in custom deployment scenarios.

Related Issues

• Avoid checking job completion in init containers #203 - Avoid checking job completion in init containers

[emmanuelmathot]

fixed with #207