Release/3.1.0

.github/actions/publish_charts/action.yaml

@@ -0,0 +1,31 @@
+name: Publish Charts
+description: Push charts to GitHub Pages
+inputs:
+  github-token:
+    description: 'GitHub token for pushing to gh-pages branch'
+    required: true
+runs:
+  using: 'composite'
+  steps:
+    - name: Configure Git
+      shell: bash
+      run: |
+        git config user.name "$GITHUB_ACTOR"
+        git config user.email "$GITHUB_ACTOR@users.noreply.github.com"
+
+    - name: Install Helm
+      uses: azure/setup-helm@v4.2.0
+
+    - name: Add Helm Repos for Dependencies
+      shell: bash
+      run: |
+        helm repo add bitnami https://charts.bitnami.com/bitnami
+
+    - name: Run chart-releaser
+      uses: helm/chart-releaser-action@v1.6.0
+      with:
+        charts_dir: deploy/charts
+        skip_existing: 'true'
+        packages_with_index: 'true'
+      env:
+        CR_TOKEN: '${{ inputs.github-token }}'

.github/workflows/release.yaml

@@ -0,0 +1,21 @@
+name: Release New Version
+on:
+  push:
+    branches:
+      - main
+  workflow_dispatch:
+    branches:
+      - main
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Latest Changes
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Package and Publish Charts
+        uses: ./.github/actions/publish_charts
+        with:
+          github-token: '${{ secrets.GITHUB_TOKEN }}'

.gitignore

@@ -30,6 +30,7 @@ share/python-wheels/
 .installed.cfg
 *.egg
 MANIFEST
+deploy/charts/*/charts
 
 # Installer logs
 pip-log.txt

deploy/charts/README.md

@@ -0,0 +1,134 @@
+# Overview
+
+These are the Helm charts to install DataOps TestGen on a Kubernetes
+environment. This README includes instructions for installing TestGen on a
+minikube instance, including the application database engine, added by the
+`testgen-services` charts. The application itself is installed by the
+`testgen-app` charts. When installing on cloud, it's likely that the
+application database will be provisioned by a cloud provider, and therefore
+installing the `testgen-services` charts is not necessary.
+
+# Preparing the Environment
+
+Here are the instructions to create a local minikube cluster where DataOps
+TestGen and its services will be installed. When deploying to a cloud /
+production environment, this step should be adjusted accordingly.
+
+Assuming that minikube is installed, the following command will create a
+cluster named dk-testgen and configure the local kubectl tool to issue commands
+against the cluster. When running helm commands against an existing cluster,
+it's important to make sure that the kubectl tool is pointing to the correct
+profile.
+
+```shell
+minikube start -p dk-testgen --namespace datakitchen
+```
+
+# Configuration
+
+Whenever a helm command is used to generate manifests from templates
+(`install`, `upgrade`), it will rely on pre-configured values to do so. Helm
+provides a variety of ways to define these values. You can find the complete
+configuration set that is also used as a default in each charts’ folder, in the
+values.yaml file.
+
+No additional configuration is recommended for the `testgen-services` charts.
+
+The `testgen-app` charts require some configuration to be fine tuned. The best
+way to do that is to save it in a values file, so that the same configuration
+set can be easily used on the first install and future upgrades.
+
+The following configuration is recommended for experimental installations, but
+you're free to adjust it for your needs. The next installation steps assumes
+that a file named tg-values.yaml exists with this configuration.
+
+```yaml
+testgen:
+
+  # Password that will be assigned to the 'admin' user during the database preparation
+  uiPassword: "admin"
+
+  # Whether to trust the target database certificates.
+  trustTargetDatabaseCertificate: true
+
+  # Whether to run the SSL certificate verifications when connecting to DataOps Observability
+  observabilityVerifySsl: false
+
+image:
+
+  # DataOps TestGen version to be installed / upgraded
+  tag: v3.0
+```
+
+# Installing
+
+The following command will install the required services for DataOps TestGen,
+which currently is the Postgres database. This step is not needed when the
+database is provisioned in a cloud environment.
+
+```shell
+helm install --create-namespace --wait testgen-services deploy/charts/testgen-services/
+```
+
+The following command will install the application. As part of the process, a
+one-time database configuration will be automatically performed. Note that the
+custom configuration values are being used.
+
+```shell
+helm install --wait -f tg-values.yaml testgen-app deploy/charts/testgen-app/
+```
+
+At this point you should have a fully functional instance of DataOps TestGen
+installed. If you're looking to secure the UI password, you can delete it from
+the values file.
+
+# Accessing the Application
+
+In order to use your TestGen instance, you have to use your web browser to
+login to it. Your username will be admin and the password the one you
+configured earlier.
+
+It may be needed that you forward the application HTTP port in order to be able
+to access it. This is especially necessary if you are using the docker driver
+for minikube installed into a mac OS. The following command forwards the TesGen
+“http” port to the host's 8501 port.
+
+```shell
+kubectl port-forward svc/testgen-app 8501:http
+```
+
+# Upgrading
+
+When you already have a running instance of TestGen and want to upgrade it to a
+newer version, you can do so without losing any data by running the
+`upgrade-system-version` TestGen command from the target version's image. The
+helm charts will do this automatically when you upgrade. You should edit your
+values file to update the image tag value to the desired version, and then run
+the following command.
+
+```shell
+helm upgrade --wait -f tg-values.yaml testgen-app deploy/charts/testgen-app/
+```
+
+# Uninstalling
+
+The following commands will uninstall almost everything that was created
+through the helm templates. It usually preserves data, which is helpful to
+avoid unplanned data loss, but will likely break a future install attempt.
+
+```shell
+helm uninstall testgen-app
+```
+
+```shell
+helm uninstall testgen-services
+```
+
+If you're uninstalling to re-install from scratch, and the data previously
+generated can be purged, issue the following command to delete the database
+volume so that the next installation will re-create and re-populate it
+successfully.
+
+```shell
+kubectl delete pvc data-testgen-services-postgresql-0
+```

deploy/charts/testgen-app/Chart.yaml

@@ -0,0 +1,24 @@
+apiVersion: v2
+name: testgen
+description: A Helm chart for Kubernetes
+
+# A chart can be either an 'application' or a 'library' chart.
+#
+# Application charts are a collection of templates that can be packaged into versioned archives
+# to be deployed.
+#
+# Library charts provide useful utilities or functions for the chart developer. They're included as
+# a dependency of application charts to inject those utilities and functions into the rendering
+# pipeline. Library charts do not define any templates and therefore cannot be deployed.
+type: application
+
+# This is the chart version. This version number should be incremented each time you make changes
+# to the chart and its templates, including the app version.
+# Versions are expected to follow Semantic Versioning (https://semver.org/)
+version: 0.1.0
+
+# This is the version number of the application being deployed. This version number should be
+# incremented each time you make changes to the application. Versions are not expected to
+# follow Semantic Versioning. They should reflect the version the application is using.
+# It is recommended to use it with quotes.
+appVersion: "v3"

deploy/charts/testgen-app/templates/_environment.yaml

@@ -0,0 +1,40 @@
+{{- define "testgen.environment" -}}
+- name: TG_DECRYPT_SALT
+  valueFrom:
+    secretKeyRef:
+      name: {{ .Values.testgen.databaseDecryptSecret.name | quote }}
+      key: "decrypt-salt"
+- name: TG_DECRYPT_PASSWORD
+  valueFrom:
+    secretKeyRef:
+      name: {{ .Values.testgen.databaseDecryptSecret.name | quote }}
+      key: "decrypt-password"
+- name: TG_METADATA_DB_HOST
+  value: {{ .Values.testgen.databaseHost | quote }}
+- name: TG_METADATA_DB_NAME
+  value: {{ .Values.testgen.databaseName | quote }}
+- name: TG_METADATA_DB_SCHEMA
+  value: {{ .Values.testgen.databaseSchema | quote }}
+- name: TG_METADATA_DB_USER
+  value: {{ .Values.testgen.databaseUser | quote }}
+- name: TG_METADATA_DB_PASSWORD
+  valueFrom:
+    secretKeyRef:
+      name: {{ .Values.testgen.databasePasswordSecret.name | quote }}
+      key: {{ .Values.testgen.databasePasswordSecret.key | quote }}
+- name: TG_TARGET_DB_TRUST_SERVER_CERTIFICATE
+  value: {{ .Values.testgen.trustTargetDatabaseCertificate | ternary "yes" "no" | quote }}
+- name: TG_EXPORT_TO_OBSERVABILITY_VERIFY_SSL
+  value: {{ .Values.testgen.observabilityVerifySsl | ternary "yes" "no" | quote }}
+- name: TG_DOCKER_RELEASE_CHECK_ENABLED
+  value: {{ .Values.testgen.releaseCheck | ternary "yes" "no" | quote }}
+{{- end -}}
+
+{{- define "testgen.hookEnvironment" -}}
+{{- if .Values.testgen.uiPassword -}}
+- name: TESTGEN_USERNAME
+  value: {{ .Values.testgen.uiUser | quote }}
+- name: TESTGEN_PASSWORD
+  value: {{ .Values.testgen.uiPassword | quote }}
+{{- end -}}
+{{- end -}}

deploy/charts/testgen-app/templates/_helpers.tpl

@@ -0,0 +1,62 @@
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "testgen.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Create a default fully qualified app name.
+We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
+If release name contains chart name it will be used as a full name.
+*/}}
+{{- define "testgen.fullname" -}}
+{{- if .Values.fullnameOverride }}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- $name := default .Chart.Name .Values.nameOverride }}
+{{- if contains $name .Release.Name }}
+{{- .Release.Name | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
+{{- end }}
+{{- end }}
+{{- end }}
+
+{{/*
+Create chart name and version as used by the chart label.
+*/}}
+{{- define "testgen.chart" -}}
+{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Common labels
+*/}}
+{{- define "testgen.labels" -}}
+helm.sh/chart: {{ include "testgen.chart" . }}
+{{ include "testgen.selectorLabels" . }}
+{{- if .Chart.AppVersion }}
+app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
+{{- end }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+{{- end }}
+
+{{/*
+Selector labels
+*/}}
+{{- define "testgen.selectorLabels" -}}
+app.kubernetes.io/name: {{ include "testgen.name" . }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+{{- end }}
+
+{{/*
+Create the name of the service account to use
+*/}}
+{{- define "testgen.serviceAccountName" -}}
+{{- if .Values.serviceAccount.create }}
+{{- default (include "testgen.fullname" .) .Values.serviceAccount.name }}
+{{- else }}
+{{- default "default" .Values.serviceAccount.name }}
+{{- end }}
+{{- end }}

deploy/charts/testgen-app/templates/deployment.yaml

@@ -0,0 +1,66 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ include "testgen.fullname" . }}
+  labels:
+    {{- include "testgen.labels" . | nindent 4 }}
+spec:
+  replicas: {{ .Values.replicaCount }}
+  selector:
+    matchLabels:
+      {{- include "testgen.selectorLabels" . | nindent 6 }}
+  template:
+    metadata:
+      annotations:
+        # Used to force Helm to trigger a re-deploy despite this manifest haven't changed
+        timestamp: {{ now | quote }}
+        {{- with .Values.podAnnotations }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+      labels:
+        {{- include "testgen.labels" . | nindent 8 }}
+        {{- with .Values.podLabels }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+    spec:
+      {{- with .Values.imagePullSecrets }}
+      imagePullSecrets:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      serviceAccountName: {{ include "testgen.serviceAccountName" . }}
+      securityContext:
+        {{- toYaml .Values.podSecurityContext | nindent 8 }}
+      containers:
+        - name: {{ .Chart.Name }}
+          securityContext:
+            {{- toYaml .Values.securityContext | nindent 12 }}
+          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          ports:
+            - name: http
+              containerPort: {{ .Values.service.port }}
+              protocol: TCP
+          resources:
+            {{- toYaml .Values.resources | nindent 12 }}
+          env:
+            {{- include "testgen.environment" . | nindent 12 }}
+          {{- with .Values.extraVolumeMounts }}
+          volumeMounts:
+            {{ toYaml . | nindent 12 }}
+          {{- end }}
+      {{- with .Values.extraVolumes }}
+      volumes:
+        {{ toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.nodeSelector }}
+      nodeSelector:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.affinity }}
+      affinity:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.tolerations }}
+      tolerations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}