[EPIC]: Add chart tests #125
Comments
|
Based on #133 (comment) we create the following decision matrix. Please be aware that I tried to be objective as possible, but there might be some points more subjective like I wanted to be, e.g understandability etc..
For further details please see the comments and field reports below. Conclusion: For the template / unit tests, see #125 (comment) we will go with a combination of both.
For the integration tests, see #125 (comment) I think it would be a good choice for now to go with terratest as well.
|
Template Test: TerratestI reused the example from #133 provided by @npepinpe 🙇 I will share here my experience with Terratest. SetupIn general the setup is quite simple and easy to understand, see #133. Run TestsIf you have setup go on your machine then it is not a problem to run the test either via After migrating the tests to the After deleting the The output: I move the tests to the root chart, here it seems to work! I had to change the paths obviously. func TestStatefulSetTemplate(t *testing.T) {
t.Parallel()
chartPath, err := filepath.Abs("../")
require.NoError(t, err)
suite.Run(t, &statefulSetTemplateTest{
chartPath: chartPath,
release: "ccsm-helm-test",
namespace: "zeebe-" + strings.ToLower(random.UniqueId()),
templates: []string{"charts/zeebe/templates/statefulset.yaml"},
})
}Failure ReportWhat I really like is that the complete rendered manifest is printed out so we can see what values are set etc. TemplateTestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: Running command helm with args [template --namespace zeebe-jxgflc --namespace zeebe-jxgflc --set image.repository=helm/zeebe --set image.tag=a.b.c --show-only charts/zeebe/templates/statefulset.yaml ccsm-helm-test /home/zell/goPath/src/github.com/zeebe-io/camunda-cloud-helm/charts/ccsm-helm]
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: ---
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: # Source: ccsm-helm/charts/zeebe/templates/statefulset.yaml
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: apiVersion: apps/v1
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: kind: StatefulSet
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: metadata:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: name: "ccsm-helm-test-zeebe"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: labels:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app: camunda-cloud-self-managed
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/name: zeebe
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: helm.sh/chart: zeebe-0.0.11
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/instance: ccsm-helm-test
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/managed-by: Helm
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/version: 1.3.1
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/part-of: camunda-cloud-self-managed
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/component: zeebe-broker
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: annotations:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: spec:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: replicas: 3
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: selector:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: matchLabels:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app: camunda-cloud-self-managed
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/name: zeebe
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: helm.sh/chart: zeebe-0.0.11
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/instance: ccsm-helm-test
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/managed-by: Helm
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/version: 1.3.1
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/part-of: camunda-cloud-self-managed
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/component: zeebe-broker
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: serviceName: "ccsm-helm-test-zeebe"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: updateStrategy:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: type: RollingUpdate
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: podManagementPolicy: Parallel
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: template:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: metadata:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: labels:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app: camunda-cloud-self-managed
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/name: zeebe
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: helm.sh/chart: zeebe-0.0.11
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/instance: ccsm-helm-test
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/managed-by: Helm
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/version: 1.3.1
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/part-of: camunda-cloud-self-managed
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: app.kubernetes.io/component: zeebe-broker
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: annotations:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: spec:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: initContainers:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: containers:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: zeebe
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: image: "camunda/zeebe:1.3.1"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: imagePullPolicy: IfNotPresent
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: env:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: LC_ALL
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: C.UTF-8
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: K8S_NAME
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: valueFrom:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: fieldRef:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: fieldPath: metadata.name
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: K8S_SERVICE_NAME
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "ccsm-helm-test-zeebe"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: K8S_NAMESPACE
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: valueFrom:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: fieldRef:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: fieldPath: metadata.namespace
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_BROKER_NETWORK_ADVERTISEDHOST
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "$(K8S_NAME).$(K8S_SERVICE_NAME).$(K8S_NAMESPACE).svc.cluster.local"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: $(K8S_SERVICE_NAME)-0.$(K8S_SERVICE_NAME).$(K8S_NAMESPACE).svc.cluster.local:26502,
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: $(K8S_SERVICE_NAME)-1.$(K8S_SERVICE_NAME).$(K8S_NAMESPACE).svc.cluster.local:26502,
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: $(K8S_SERVICE_NAME)-2.$(K8S_SERVICE_NAME).$(K8S_NAMESPACE).svc.cluster.local:26502,
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_BROKER_CLUSTER_CLUSTERNAME
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: ccsm-helm-test-zeebe
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_LOG_LEVEL
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "info"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "3"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_BROKER_CLUSTER_CLUSTERSIZE
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "3"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "3"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_BROKER_THREADS_CPUTHREADCOUNT
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "2"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_BROKER_THREADS_IOTHREADCOUNT
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "2"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_BROKER_GATEWAY_ENABLE
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "false"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "io.camunda.zeebe.exporter.ElasticsearchExporter"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "http://elasticsearch-master:9200"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_BROKER_NETWORK_COMMANDAPI_PORT
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "26501"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_BROKER_NETWORK_INTERNALAPI_PORT
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "26502"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: ZEEBE_BROKER_NETWORK_MONITORINGAPI_PORT
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "9600"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: K8S_POD_NAME
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: valueFrom:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: fieldRef:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: fieldPath: metadata.name
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: JAVA_TOOL_OPTIONS
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: value: "-XX:MaxRAMPercentage=25.0 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/data -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log -XX:+ExitOnOutOfMemoryError"
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: ports:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - containerPort: 9600
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: name: http
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - containerPort: 26501
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: name: command
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - containerPort: 26502
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: name: internal
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: readinessProbe:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: httpGet:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: path: /ready
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: port: 9600
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: periodSeconds: 10
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: successThreshold: 1
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: timeoutSeconds: 1
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: resources:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: limits:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: cpu: 1000m
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: memory: 4Gi
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: requests:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: cpu: 500m
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: memory: 2Gi
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: volumeMounts:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: config
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: mountPath: /usr/local/bin/startup.sh
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: subPath: startup.sh
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: data
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: mountPath: /usr/local/zeebe/data
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: exporters
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: mountPath: /exporters
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: securityContext:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: {}
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: volumes:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: config
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: configMap:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: name: ccsm-helm-test-zeebe
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: defaultMode: 0744
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - name: exporters
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: emptyDir: {}
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: volumeClaimTemplates:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: - metadata:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: name: data
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: spec:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: accessModes: [ReadWriteOnce]
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: storageClassName:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: resources:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: requests:
TestStatefulSetTemplate/TestContainerSpecImage 2022-02-09T06:34:57+01:00 logger.go:66: storage: "10Gi"
The test failure output is easy understandable: statefulset_test.go:56:
Error Trace: statefulset_test.go:56
Error: Not equal:
expected: "camunda/zeebe:1.3.1"
actual : "helm/zeebe:a.b.c"
Diff:
--- Expected
+++ Actual
@@ -1 +1 @@
-camunda/zeebe:1.3.1
+helm/zeebe:a.b.c
Test: TestStatefulSetTemplate/TestContainerSpecImage
--- FAIL: TestStatefulSetTemplate/TestContainerSpecImage (0.19s)
Expected :camunda/zeebe:1.3.1
Actual :helm/zeebe:a.b.cDue to the setup and using Goland debugging is also possible. One issue we still have is that it just calls
Testing DefaultsI added another test to verify the defaults we set. The test you can see below: func (s *statefulSetTemplateTest) TestContainerDefaults() {
options := &helm.Options{
KubectlOptions: k8s.NewKubectlOptions("", "", s.namespace),
ExtraArgs: map[string][]string{"template": {"--debug"}, "install": {"--debug"}},
}
output := helm.RenderTemplate(s.T(), options, s.chartPath, s.release, s.templates)
var statefulSet appsv1.StatefulSet
helm.UnmarshalK8SYaml(s.T(), output, &statefulSet)
expectedReplicas := int32(3)
replicas := *statefulSet.Spec.Replicas
s.Require().Equal(replicas, expectedReplicas)
}One thing to note is that we need to be aware of the different go types. We need to dereference the pointers ( It becomes more interesting if we also want to verify whether the right env vars are set. I expected that this would be really a pain. Because I experienced that in the zeebe controller, but it was not. func (s *statefulSetTemplateTest) TestContainerDefaults() {
options := &helm.Options{
KubectlOptions: k8s.NewKubectlOptions("", "", s.namespace),
ExtraArgs: map[string][]string{"template": {"--debug"}, "install": {"--debug"}},
}
output := helm.RenderTemplate(s.T(), options, s.chartPath, s.release, s.templates)
var statefulSet appsv1.StatefulSet
helm.UnmarshalK8SYaml(s.T(), output, &statefulSet)
expectedEnv := v1.EnvVar{Name: "ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT", Value: "3"}
envs := statefulSet.Spec.Template.Spec.Containers[0].Env
s.Require().Contains(envs, expectedEnv)
}Asserting the env var is quite easy. What I wanted to verify how it looks like if I change the default now or the property is no longer used (e.g. env is not set or not equal value). The failure output is not so nice anymore if it doesn't contain the value, but sure I still can find it somehow and at least it fails :). I think for the for such test (like testing all defaults) it is worth to setup a golden file. |
Template Test: Golden FilesSetupI first thought about using my example https://github.com/camunda-community-hub/camunda-cloud-helm/commits/zell-test-goldenfile Which just adds this scripts and golden file. Script: #!/bin/bash
set -euox pipefail
helm template test . --set elasticsearch.enabled=false > actual
diff goldenfile actual
rm actualIt is quite simple, doesn't need any go setup. But the golden file is rather larger, since it contains all manifests. AlternativeIt came to my mind to combine both. Terratest and the go files, also based on this blog post https://medium.com/@jarifibrahim/golden-files-why-you-should-use-them-47087ec994bf func (s *statefulSetTemplateTest) TestContainerDefaultsStatefulsetGolden() {
options := &helm.Options{
KubectlOptions: k8s.NewKubectlOptions("", "", s.namespace),
}
output := helm.RenderTemplate(s.T(), options, s.chartPath, s.release, s.templates)
actual := []byte(output)
goldenFile := "statefulset.golden.json"
update := false
if (update) {
err := ioutil.WriteFile(goldenFile, actual, 0644)
if err != nil {
return
}
}
expected, err := ioutil.ReadFile(goldenFile)
if err != nil {
return
}
s.Require().Equal(string(expected), output)
}The test is fairly simple. If necessary it also allows to generate new golden files, similar how we do it in the zeebe controller. Run TestsIt can be easily also run by the CI and local, if helm is installed. AlternativeAs described in the Terratest comment above, we need helm and go installed. If both avail. running tests are also quite simple either via Failure ReportSee #133 (comment) If running the script from above the following would be the output, which makes it quite clear what the difference is: AlternativeIf we run the Terratest with goldenfile the output on failure is also quite nice and useful: Testing defaultsI think here lies the strength of the golden files, this is something which shouldn't change and we can verify with a previous generated manifest and don't need to verify the properties one by one. Small note: Might make sense to compare the objects instead of the strings. ConclusionI think we could combine both golden files and Terratest, and if we want to verify specific properties (or conditions) we can use the direct property tests we saw above #125 (comment) So we would have all in one place, tests specific to a manifest (for each one test), can run and edit easily locally via ide etc. |
Integration TestsI first thought about what we actually want to test here. I think it is two things:
Other things like broken templates, values are not set correctly etc. are caught by the tests above. So to turn it around, what are potential failure cases we can find.
The first case we could also solve with other tools, which validates manifests based on the k8 api, but not the second one. For the second case, I could imagine as MVP one simple test which allows to ensure most things are running. We deploy a process model with an user task, start an instance, verify via Operate API that an instance exist and verify via tasklist API that an user task exists. This ensures that:
This test should be run with the default values and with values which configures and enables everything, so we can be sure that all work together and all configurations/templates are accepted by k8. For simplicity and test the tools I will concentrate one the first failure case, to provoke some failures and understand how test failures look like etc. |
IT: TerratestSetupSetup is as in #133 described. I ported them to ccsm, similar I described above. I had to move them to the parent chart. I think if we use for unit tests also terratest it is also quite easy if we have the same for integration tests, since it is the same setup etc. Run TestsSimilar to described above, via Failure ReportAs described above I changed something in the templates to produce an error, which is only detectable via integration tests, but not with the template test. Diff: --- a/charts/ccsm-helm/charts/zeebe/templates/statefulset.yaml
+++ b/charts/ccsm-helm/charts/zeebe/templates/statefulset.yaml
@@ -28,6 +28,8 @@ spec:
{{ $key }}: {{ $value | quote }}
{{- end }}
spec:
+ resources:
+ {{- toYaml .Values.resources | nindent 10 }}
{{- if .Values.priorityClassName }}
priorityClassName: {{ .Values.priorityClassName | quote }}
{{- end }}
@@ -122,8 +124,6 @@ spec:
periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
successThreshold: {{ .Values.readinessProbe.successThreshold }}
timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
- resources:
- {{- toYaml .Values.resources | nindent 10 }}
volumeMounts:
- name: config
mountPath: /usr/local/bin/startup.shI move the resources under the spec, which is valid yaml but not valid for k8 api. Be aware this is just an example, sure normally we would catch this with the normal golden files, but if we have a property which is only enabled/rendered if certain properties are set then maybe not. The test fails really fast, since the helm install immediately fails. The error message also clearly mentions what the issue is, === RUN TestIntegration/TestGatewayConnection
TestIntegration/TestGatewayConnection 2022-02-10T07:07:52+01:00 client.go:42: Configuring Kubernetes client using config file /home/zell/.kube/config with context gke_zeebe-io_europe-west1-b_zeebe-cluster
TestIntegration/TestGatewayConnection 2022-02-10T07:07:52+01:00 logger.go:66: Running command helm with args [install --kube-context gke_zeebe-io_europe-west1-b_zeebe-cluster --namespace zeebe-eaukyg --set image.tag=1.3.1 zeebe-cluster-helm-it /home/zell/goPath/src/github.com/zeebe-io/camunda-cloud-helm/charts/ccsm-helm]
TestIntegration/TestGatewayConnection 2022-02-10T07:07:53+01:00 logger.go:66: Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(StatefulSet.spec.template.spec): unknown field "resources" in io.k8s.api.core.v1.PodSpec
install.go:16:
Error Trace: install.go:16
integration_test.go:69
Error: Received unexpected error:
error while running command: exit status 1; Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(StatefulSet.spec.template.spec): unknown field "resources" in io.k8s.api.core.v1.PodSpec
Test: TestIntegration/TestGatewayConnection
TestIntegration/TestGatewayConnection 2022-02-10T07:07:53+01:00 client.go:42: Configuring Kubernetes client using config file /home/zell/.kube/config with context gke_zeebe-io_europe-west1-b_zeebe-cluster
--- FAIL: TestIntegration/TestGatewayConnection (1.20s) |
IT: CT and Helm TestSetupSetup is fairly simple, locally you can download the binary from the release page of https://github.com/helm/chart-testing for CI you can use the corresponding github action https://github.com/helm/chart-testing-action It doesn't need much else, just access to the kubeconfig to deploy the chart to (similar to terratest). Run TestsThe run is simple as Unfortunately on the first try it failed during awaiting to become ready. This caused to print all logs from all pods and deployments (k describe), quite nice and insightful. Increasing the timeout hasn't helped, it seems to always fail with some timing issues. The namespace is always deleted after the test run. Failure ReportSimilar to the IT: Terratest, I changed something in the templates to produce an error, which is only detectable via integration tests, but not with the template test. The test fails really fast, since the helm install immediately fails. As we can see it shows the same message, as the Terratest. Since both just print the error message from the K8 API, it is quite clear what the issue is. ConclusionOne issue I see with the chart-testing tool is that we have not much possibility to configure the complete testing process (or have not much control at all). Available options you can find here https://github.com/helm/chart-testing/blob/main/doc/ct_install.md but they are quite limited. We can configure the helm install itself, but it seems that the chart-testing times out after 3 minutes. Or at least I was not able to run one successful test with In general looking at the other failure cases (#125 (comment)) we need to test, I feel it will become much harder to find out the issues and reproduce it, if this fails in the CI. You have to run the My conclusion out of this would be to use terratest also for the integration tests (for now). The benefit here is that we have one tool for both (unit and it tests) and more control over it. It makes it easy to run and debug the tests. In general the tests were also quite simple. I think we can always revisit this decision, I don't expect that we will write a lot of integration tests here, and of course I'm always open for feedback and concerns which might be raised regarding this. P.S.: I tried it again and checked what happens in the K8, and I saw actually that ct is already running the tests: I forwared the output of the tool into a separate log file and found at the begin of the logging the following: After that information it prints ~4800 lines of logs, for all pods etc. Which caused that I can't see it in my shell. Well this was not optimal I would say 😅 |
We need to test our charts, to verify and ensure functionality and to guarantee that we not break them on adding more features or fixing bugs.
The text was updated successfully, but these errors were encountered: