GDB-7986: Overhauled the configurations structure

CHANGELOG.md

@@ -4,51 +4,70 @@
 
 TODO: short motivational paragraph about the major version
 TODO: short info about being decoupled from GraphDB
+TODO: short section about the most notable changes (decoupling, naming, plugins, etc.)
+
+### Breaking
+
+TODO: decide how detailed we want this to be
+
+- Resource names are no longer hardcoded and are using the templates for `nameOverride` and `fullnameOverride`
+- Renamed `extraLabels` to just `labels`
+- Renamed GraphDB storage PVC prefix to `graphdb-storage` and server import folder to `graphdb-server-import`
+- Removed setting FQDN as hostnames in GraphDB and the proxy in favor of dynamically resolving and configuring the hostnames in the init containers
+- Renamed `messageSize` to `messageSizeKB` in the cluster creation configuration
+- Renamed `java_args` to `javaArguments`
+- Removed `global.storageClass` in favor of using by default the default storage class in the cluster. Templates will no longer
+  use `global.storageClass`.
+- Updated the GraphDB deployment URL to be http://graphdb.127.0.0.1.nip.io/ by default
+- Removed the default value from `global.imageRegistry`, the chart now uses the value from `image.registry`
+- Updated the ingress to be agnostic to the ingress implementation. It will no longer assume that NGINX is the ingress controller in the
+  cluster and will no longer deploy NGINX specific annotations by default. Removed anything related to NGINX as configurations.
+- Moved all proxy configurations from `graphdb.clusterProxy` to just `proxy`
+  - Renamed `proxy.persistence.enablePersistence` toggle to just `enabled`
+  - Moved `proxy.serviceType` to `proxy.service.type`
+- Configmaps from `graphdb.configs` are now under `configuration`, `repositories` and `security` with a different structure allowing better reuse
+  of existing configmaps
+- Moved job related configurations from `graphdb` (e.g. `graphdb.jobResources`) to a new root section `jobs`
+- Moved `images.graphdb` configurations to just `image`
+- Moved `deployment.imagePullPolicy` to `image.pullPolicy` and `deployment.imagePullSecret` to `image.pullSecrets`
+  - Updated `imagePullSecret` to be a list, e.g. `imagePullSecrets`
+- Moved `graphdb.import_directory_mount` configurations to `import.volumeMount`
+- Moved `deployment.ingress` to just `ingress`
+  - Moved `deployment.tls` to `ingress.tls`
+- Renamed `graphdb.node.service` to `headlessService`
+- Moved `graphdb` and `graphdb.node` configurations on the root level
+  - Moved `graphdb.pdb` to `pdb`
+- Moved `graphdb.clusterConfig` configurations
+  - Moved `graphdb.clusterConfig.nodesCount` to `replicas`
+  - Moved the rest of `graphdb.clusterConfig` configurations under `cluster` and `cluster.config`
+- Moved `graphdb.security` configurations to `security`
 
 ### New
 
 - Added `annotations` for common annotations across resources
-- Added `graphdb.serviceAccount` allowing you to create or use an existing service account for GraphDB pods.
+- Added `serviceAccount` allowing you to create or use an existing service account for GraphDB pods.
 - Added separate `labels` and `annotations` for the cluster proxy
 - Added GraphDB and GraphDB proxy hostnames resolution in the init containers
-- Added `properties` for inserting additional GraphDB configurations in the properties configmap
-- Added `images.graphdb.sha` to optionally provide an expected SHA checksum of the image
-- Added `graphdb.node.persistence.enabled` toggle flag for enabling or disabling the persistence of GraphDB
-- Added new configuration options for the default ingress `deployment.ingress`:
-  - Ability to override the `host` and `path` for GraphDB from `deployment.host` and `graphdb.workbench.subpath` 
+- Added `conpfiguration.properties` for inserting additional GraphDB configurations in the properties configmap
+- Added `image.sha` to optionally provide an expected SHA checksum of the image
+- Added `persistence.enabled` toggle flag for enabling or disabling the persistence of GraphDB
+- Added new configuration options for the default ingress `ingress`:
+  - Ability to override the `host` and `path` for GraphDB from `configuration.host` and `configuration.path` 
   - Changing the `pathType` 
   - Inserting additional hosts and TLS configurations with `extraHosts` and `extraTLS`
 - Added `labels` for each service resource for insertion of additional labels 
 - Added `containerPorts` and `proxy.containerPorts` for mapping the ports on which GraphDB listens on
 - Added `ports` mappings in each service 
 - Added `extraContainerPorts` and `proxy.extraContainerPorts`
+- Added `imagePullPolicy` to the jobs containers
 
 ### Updates
 
 - GraphDB properties and logback configuration configmaps are now applied by default
 - Values in `labels`, `annotations` and `imagePullSecrets` are now evaluated as templates
 - Removed unused busybox image configurations from `images.busybox`
-- Updated the ingress resource to be agnostic to the ingress implementation. It will no longer assume that NGINX is the ingress controller in the
-  cluster
 - Service resources and probes now refer to the target ports by their nicknames
-
-### Breaking
-
-- Renamed `extraLabels` to just `labels`
-- Renamed GraphDB storage PVC prefix to `graphdb-storage` and server import folder to `graphdb-server-import`
-- Resource names are no longer hardcoded and are using the templates for `nameOverride` and `fullnameOverride`
-- Removed setting FQDN as hostnames in GraphDB and the proxy in favor of dynamically resolving and configuring the hostnames in the init containers
-- Configmaps from `graphdb.configs` are now under `configuration` and with a different structure allowing better reuse of existing configmaps
-- Updated `workbench.subpath` to serve GraphDB at context path `/` by default
-- Updated `deployment.imagePullSecret` to be a list, e.g. `deployment.imagePullSecrets`
-- Removed the default value from `global.imageRegistry`, the chart now uses the value from `images.graphdb.registry`
-- Removed `global.storageClass` in favor of using by default the default storage class in the cluster. Templates will no longer
-  use `global.storageClass`.
-- Renamed `graphdb.clusterProxy.persistence.enablePersistence` toggle to just `enabled`
-- Removed `maxRequestSize` and `timeout` configurations from `deployment.ingress` as they were specific to the ingress controller implementation of
-  nginx
 - Renamed the port mappings of GraphDB and GraphDB proxy to `http` and `rpc`
-- Renamed `graphdb.node.service` to `headlessService`
 
 ## Version 10.6.0-R2
 

files/config/cluster-config.json

@@ -1,9 +1,9 @@
 {
-  "electionMinTimeout": {{ .Values.graphdb.clusterConfig.electionMinTimeout | int }},
-  "electionRangeTimeout": {{ .Values.graphdb.clusterConfig.electionRangeTimeout | int }},
-  "heartbeatInterval": {{ .Values.graphdb.clusterConfig.heartbeatInterval | int }},
-  "verificationTimeout": {{ .Values.graphdb.clusterConfig.verificationTimeout | int }},
-  "messageSizeKB": {{ .Values.graphdb.clusterConfig.messageSize | int }},
-  "transactionLogMaximumSizeGB": {{ .Values.graphdb.clusterConfig.transactionLogMaximumSizeGB | quote }},
+  "electionMinTimeout": {{ .Values.cluster.config.electionMinTimeout | int }},
+  "electionRangeTimeout": {{ .Values.cluster.config.electionRangeTimeout | int }},
+  "heartbeatInterval": {{ .Values.cluster.config.heartbeatInterval | int }},
+  "verificationTimeout": {{ .Values.cluster.config.verificationTimeout | int }},
+  "messageSizeKB": {{ .Values.cluster.config.messageSizeKB | int }},
+  "transactionLogMaximumSizeGB": {{ .Values.cluster.config.transactionLogMaximumSizeGB | quote }},
   "nodes": {{- include "graphdb.cluster.nodes.json" . | indent 2 }}
 }

files/config/settings.js

@@ -2,7 +2,7 @@
   "import.server" : { },
   "import.local" : { },
   "properties" : {
-  {{- if .Values.graphdb.security.enabled }}
+  {{- if .Values.security.enabled }}
     "security.enabled" : true,
   {{- end }}
     "current.location" : ""

files/config/users.js

@@ -13,9 +13,9 @@
       },
       "dateCreated" : 1618403171751
     },
-    "{{ .Values.graphdb.security.provisioningUsername }}" : {
-      "username" : "{{ .Values.graphdb.security.provisioningUsername }}",
-      "password" : "{bcrypt}{{ htpasswd .Values.graphdb.security.provisioningUsername .Values.graphdb.security.provisioningPassword | trimPrefix (printf "%s:" .Values.graphdb.security.provisioningUsername) }}",
+    "{{ .Values.security.provisioningUsername }}" : {
+      "username" : "{{ .Values.security.provisioningUsername }}",
+      "password" : "{bcrypt}{{ htpasswd .Values.security.provisioningUsername .Values.security.provisioningPassword | trimPrefix (printf "%s:" .Values.security.provisioningUsername) }}",
       "grantedAuthorities" : [ "ROLE_ADMIN" ],
       "appSettings" : {
         "DEFAULT_INFERENCE" : true,

templates/NOTES.txt

@@ -10,14 +10,14 @@
                       |_|
 -------------------------------------------------------------------------------
 version: {{ .Chart.AppVersion }}
-GDB cluster: {{ gt (int .Values.graphdb.clusterConfig.nodesCount) 1 }}
+GDB cluster: {{ gt ( int .Values.replicas ) 1 }}
 
 ** Please be patient while the chart is being deployed and services are available **
 You can check their status with kubectl get pods
 
-{{- if and (gt (int .Values.graphdb.clusterConfig.nodesCount) 1) (not .Values.graphdb.node.license) }}
-
+{{- if and ( gt ( int .Values.replicas) 1 ) ( not .Values.license.existingSecret ) }}
 WARNING: You are attempting to make a cluster without providing a license secret!
-{{ end }}
+{{- end }}
+
 Endpoints:
-* GraphDB workbench: {{ include "graphdb.url.public" . }}
+* GraphDB workbench: {{ .Values.configuration.externalUrl }}

templates/_helpers.tpl

@@ -1,33 +1,26 @@
-{{/*
-Renders the URL address at which GraphDB would be accessed
-*/}}
-{{- define "graphdb.url.public" -}}
-  {{- printf "%s://%s%s" .Values.deployment.protocol .Values.deployment.host .Values.graphdb.workbench.subpath -}}
-{{- end }}
-
 {{/*
 Combined image pull secrets
 */}}
 {{- define "graphdb.combinedImagePullSecrets" -}}
-  {{- $secrets := concat .Values.global.imagePullSecrets .Values.deployment.imagePullSecrets }}
+  {{- $secrets := concat .Values.global.imagePullSecrets .Values.image.pullSecrets }}
   {{- tpl ( toYaml $secrets ) . -}}
 {{- end -}}
 
 {{/*
 Renders the container image for GraphDB
 */}}
 {{- define "graphdb.image" -}}
-  {{- $repository := .Values.images.graphdb.repository -}}
-  {{- $tag := .Values.images.graphdb.tag | default .Chart.AppVersion | toString -}}
+  {{- $repository := .Values.image.repository -}}
+  {{- $tag := .Values.image.tag | default .Chart.AppVersion | toString -}}
   {{- $image := printf "%s:%s" $repository $tag -}}
   {{/* Add registry if present */}}
-  {{- $registry := .Values.global.imageRegistry | default .Values.images.graphdb.registry -}}
+  {{- $registry := .Values.global.imageRegistry | default .Values.image.registry -}}
   {{- if $registry -}}
     {{- $image = printf "%s/%s" $registry $image -}}
   {{- end -}}
   {{/* Add SHA if provided */}}
-  {{- if .Values.images.graphdb.sha -}}
-    {{- $image = printf "%s@sha256:%s" $image .Values.images.graphdb.sha -}}
+  {{- if .Values.image.sha -}}
+    {{- $image = printf "%s@sha256:%s" $image .Values.image.sha -}}
   {{- end -}}
   {{- $image -}}
 {{- end -}}
@@ -38,9 +31,9 @@ Renders the gRPC address of each GraphDB node that is part of the cluster as a J
 {{- define "graphdb.cluster.nodes.json" -}}
   {{- $pod_name := include "graphdb.fullname" . -}}
   {{- $service_name := include "graphdb.fullname.service.headless" . -}}
-  {{- $service_rpc_port := .Values.graphdb.node.headlessService.ports.rpc -}}
+  {{- $service_rpc_port := .Values.headlessService.ports.rpc -}}
   {{- $nodes := list -}}
-  {{- range $i, $node_index := until (int .Values.graphdb.clusterConfig.nodesCount) -}}
+  {{- range $i, $node_index := until (int .Values.replicas) -}}
     {{- $nodes = append $nodes (printf "%s-%s.%s.%s.svc.cluster.local:%s" $pod_name (toString $node_index) $service_name $.Release.Namespace (toString $service_rpc_port)) -}}
   {{- end -}}
   {{- toPrettyJson $nodes -}}
@@ -52,10 +45,10 @@ Renders the HTTP address of each GraphDB node that is part of the cluster, joine
 {{- define "graphdb-proxy.cluster.nodes" -}}
   {{- $pod_name := include "graphdb.fullname" . -}}
   {{- $service_name := include "graphdb.fullname.service.headless" . -}}
-  {{- $service_http_port := .Values.graphdb.node.headlessService.ports.http -}}
-  {{- range $i, $node_index := until (int $.Values.graphdb.clusterConfig.nodesCount) -}}
+  {{- $service_http_port := .Values.headlessService.ports.http -}}
+  {{- range $i, $node_index := until (int $.Values.replicas) -}}
     http://{{ $pod_name }}-{{ $node_index }}.{{ $service_name }}.{{ $.Release.Namespace }}.svc.cluster.local:{{ $service_http_port }}
-    {{- if gt (sub (int $.Values.graphdb.clusterConfig.nodesCount) 1 ) $node_index -}}
+    {{- if gt (sub (int $.Values.replicas) 1 ) $node_index -}}
       {{- ", " -}}
     {{- end -}}
   {{- end -}}

templates/_labels.tpl

@@ -57,9 +57,9 @@ app.kubernetes.io/instance: {{ .Release.Name }}
 Create the name of the service account to use
 */}}
 {{- define "graphdb.serviceAccountName" -}}
-  {{- if .Values.graphdb.serviceAccount.create }}
-    {{- default (include "graphdb.fullname" .) .Values.graphdb.serviceAccount.name }}
+  {{- if .Values.serviceAccount.create }}
+    {{- default (include "graphdb.fullname" .) .Values.serviceAccount.name }}
   {{- else }}
-    {{- default "default" .Values.graphdb.serviceAccount.name }}
+    {{- default "default" .Values.serviceAccount.name }}
   {{- end }}
 {{- end }}

templates/graphdb/configmap-logback.yaml

@@ -1,4 +1,4 @@
-{{- if not .Values.configuration.logback.existingConfigmap }}
+{{- if not .Values.configuration.logbackConfig.existingConfigmap }}
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -10,6 +10,6 @@ metadata:
     {{- tpl ( toYaml . ) $ | nindent 4 }}
   {{- end }}
 data:
-  {{ .Values.configuration.logback.configmapKey }}: |-
+  {{ .Values.configuration.logbackConfig.configmapKey }}: |-
     {{- tpl ( .Files.Get "files/config/logback.xml" ) . | nindent 4 }}
 {{- end }}

templates/graphdb/configmap-properties.yaml

@@ -1,4 +1,4 @@
-{{- if not .Values.configuration.properties.existingConfigmap }}
+{{- if not .Values.configuration.propertiesConfig.existingConfigmap }}
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -10,11 +10,11 @@ metadata:
     {{- tpl ( toYaml . ) $ | nindent 4 }}
   {{- end }}
 data:
-  {{ .Values.configuration.properties.configmapKey }}: |-
+  {{ .Values.configuration.propertiesConfig.configmapKey }}: |-
     {{- tpl ( .Files.Get "files/config/graphdb.properties" | trim ) . | nindent 4 }}
-    {{- if .Values.properties }}
+    {{- if .Values.configuration.properties }}
     ##### Overrides from values.yaml #####
-    {{ range $key, $val := .Values.properties }}
+    {{ range $key, $val := .Values.configuration.properties }}
     {{- if ne $val nil }}
     {{- $key }}={{ tpl ($val | toString) $ | quote }}
     {{- end }}

templates/graphdb/configmap-settings.yaml

@@ -1,4 +1,4 @@
-{{- if and .Values.graphdb.security.enabled (not .Values.provision.settings.existingConfigmap) }}
+{{- if and .Values.security.enabled (not .Values.configuration.initialSettingsConfig.existingConfigmap) }}
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -10,6 +10,6 @@ metadata:
     {{- tpl ( toYaml . ) $ | nindent 4 }}
   {{- end }}
 data:
-  {{ .Values.provision.settings.configmapKey }}: |-
+  {{ .Values.configuration.initialSettingsConfig.configmapKey }}: |-
     {{- tpl ( .Files.Get "files/config/settings.js" ) . | nindent 4 }}
 {{- end }}

templates/graphdb/configmap-users.yaml

@@ -1,4 +1,4 @@
-{{- if and .Values.graphdb.security.enabled (not .Values.provision.users.existingConfigmap) }}
+{{- if and .Values.security.enabled (not .Values.security.initialUsersConfig.existingConfigmap) }}
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -10,6 +10,6 @@ metadata:
     {{- tpl ( toYaml . ) $ | nindent 4 }}
   {{- end }}
 data:
-  {{ .Values.provision.users.configmapKey }}: |-
+  {{ .Values.security.initialUsersConfig.configmapKey }}: |-
     {{- tpl ( .Files.Get "files/config/users.js" ) . | nindent 4 }}
 {{- end }}

templates/graphdb/configmap.yaml

@@ -18,7 +18,7 @@ data:
     -Dgraphdb.workbench.importDirectory=/opt/graphdb/home/graphdb-import
     -Dgraphdb.home=/opt/graphdb/home
     -Dgraphdb.ontop.jdbc.path=/opt/graphdb/home/jdbc-driver
-{{- if gt (int $.Values.graphdb.clusterConfig.nodesCount) 1 }}
-    -Dgraphdb.auth.token.secret={{ $.Values.graphdb.clusterConfig.clusterSecret | quote }}
+{{- if gt (int $.Values.replicas) 1 }}
+    -Dgraphdb.auth.token.secret={{ $.Values.cluster.clusterSecret | quote }}
 {{- end }}
-    {{ default $.Values.graphdb.node.java_args}}
+    {{ default $.Values.configuration.javaArguments}}

templates/graphdb/pdb.yaml

@@ -1,4 +1,4 @@
-{{- if .Values.graphdb.pdb.create }}
+{{- if .Values.pdb.create }}
 apiVersion: policy/v1
 kind: PodDisruptionBudget
 metadata:
@@ -10,11 +10,11 @@ metadata:
     {{- tpl ( toYaml . ) $ | nindent 4 }}
   {{- end }}
 spec:
-  {{- if .Values.graphdb.pdb.minAvailable }}
-  minAvailable: {{ .Values.graphdb.pdb.minAvailable }}
+  {{- if .Values.pdb.minAvailable }}
+  minAvailable: {{ .Values.pdb.minAvailable }}
   {{- end }}
-  {{- if .Values.graphdb.pdb.maxUnavailable }}
-  maxUnavailable: {{ .Values.graphdb.pdb.maxUnavailable }}
+  {{- if .Values.pdb.maxUnavailable }}
+  maxUnavailable: {{ .Values.pdb.maxUnavailable }}
   {{- end }}
   selector:
     matchLabels:

templates/graphdb/service-headless.yaml

@@ -4,10 +4,10 @@ metadata:
   name: {{ include "graphdb.fullname.service.headless" . }}
   labels:
     {{- include "graphdb.labels" . | nindent 4 }}
-    {{- with .Values.graphdb.node.headlessService.labels }}
+    {{- with .Values.headlessService.labels }}
       {{- tpl ( toYaml . ) $ | nindent 4 }}
     {{- end }}
-  {{- with (mergeOverwrite (deepCopy .Values.annotations) .Values.graphdb.node.headlessService.annotations) }}
+  {{- with (mergeOverwrite (deepCopy .Values.annotations) .Values.headlessService.annotations) }}
   annotations:
     {{- tpl ( toYaml . ) $ | nindent 4 }}
   {{- end }}
@@ -17,12 +17,12 @@ spec:
     {{- include "graphdb.selectorLabels" . | nindent 4 }}
   ports:
     - name: http
-      port: {{ .Values.graphdb.node.headlessService.ports.http }}
+      port: {{ .Values.headlessService.ports.http }}
       targetPort: http
       protocol: TCP
-    {{- if gt (int $.Values.graphdb.clusterConfig.nodesCount) 1 }}
+    {{- if gt (int $.Values.replicas) 1 }}
     - name: rpc
-      port: {{ .Values.graphdb.node.headlessService.ports.rpc }}
+      port: {{ .Values.headlessService.ports.rpc }}
       targetPort: rpc
       protocol: TCP
     {{- end }}

templates/graphdb/serviceaccount.yaml

@@ -1,11 +1,11 @@
-{{- if .Values.graphdb.serviceAccount.create -}}
+{{- if .Values.serviceAccount.create -}}
 apiVersion: v1
 kind: ServiceAccount
 metadata:
   name: {{ include "graphdb.serviceAccountName" . }}
   labels:
     {{- include "graphdb.labels" . | nindent 4 }}
-  {{- with (mergeOverwrite (deepCopy .Values.annotations) .Values.graphdb.serviceAccount.annotations) }}
+  {{- with (mergeOverwrite (deepCopy .Values.annotations) .Values.serviceAccount.annotations) }}
   annotations:
     {{- tpl ( toYaml . ) $ | nindent 4 }}
   {{- end -}}