How to package and run chainsaw tests as user-facing E2E tests running in production clusters ?

[gberche-orange]

background

AFAIK, chainsaw is currently targetting the testing of k8s operators, and currently focusing on the persona of a k8s operator contributor. The k8s operator contributor is testing his operator code in a CI/CD environment using Chainsaw CLI and chainsaw github action, helping taking decision as whether a k8s operator code base can be released when test suite passes.

Another useful personal is the k8s operator user who is responsible for configuring, installating and operating a k8s operator.

Here is a typical user story

As a k8s operator user,
In order to make sure that the installed k8s operator is properly installed and running,
I need to run end-to-end tests in my production system, triggered from different criterias:

• a change to the operator installation or configuration
• a change to one of the dependency of the operator (e.g. a S3 object storage upgrade)
• a periodic schedule (e.g. once an hour) in order to measure user-facing SLOs (K8S operator reconciliation success, time to reconciliate; ...)

Let's take the example of a managed database operator such as cnpg which is planning this use-case in cloudnative-pg/cloudnative-pg#9220 CNPG tests are currently written in go (see https://github.com/cloudnative-pg/cloudnative-pg/tree/main/tests/e2e and https://github.com/cloudnative-pg/cloudnative-pg/blob/release-1.24/docs/src/e2e.md) and don't yet leverage the chainsaw declarative approach.

Kyverno seems to package such E2E tests as helm tests at https://github.com/kyverno/kyverno/tree/main/charts/kyverno/templates/tests, in such as kyverno/kyverno#6085 They seem limited in scope and not yet leveraging chainsaw.

Prior art ?

Is the prior art to using chainsaw for user-facing E2E tests that I missed ?

Possible enhancements

Here are some ideas of potential features to support user-facing E2E running in production

helm chart

package chainsaw as an helm chart:

• deploying CronJob to trigger tests (scheduled or manually triggered), getting success status as job output, and test output as stdout
• test cases files are provided as configmap, mounted as volume in containers running chainsaw
• users configure chainsaw command line options through values.yaml

k8s controller

introduce a chainsaw k8s controller that reconciles CRDs, such as TestExecution.chainsaw.kyverno.io/v1alpha2 that surfaces the chainsaw test cli command

• tests case source as specified in a reference supporting
  • configmap
  • remote artefacts (git, http, s3, oci image...) leveraging off-the-shelf CRs such as
    • FluxToolkit sources https://fluxcd.io/flux/components/source/
    • Argo artifact see https://argo-workflows.readthedocs.io/en/latest/walk-through/hardwired-artifacts/
• users trigger a new test execution by deploying a TestExecution.chainsaw.kyverno.io CR and watching its status
• The detailed execution logs (and generated chainsaw documentation) can be outputted in
  • A k8s container stdout (typically a k8s job associated with the TestExecution)
  • a file in persistent volume, optionally surfaced as an HTTP endpoint
  • a remote artefact (such as an object in a S3 bucket)

I have previously worked on a similar approach with kuttl and the following custom scaffolding

• kuttl test cases are deployed as configmap
• carvel kapp controller detects change in configmap on the cluster to trigger a job from cronjob

This experiment suffered from limited user experience to leverage kuttl outputs

See also related attempt to leverage in cluster test-kube tool with kuttl in https://github.com/kubeshop/testkube/issues/1142 and kubeshop/testkube#4069

Schedule within argoworflow

[camaeel]

Would like to ask the same question.
IMHO helm hooks after install may be a way to execute tests within the cluster but it isn't that simple (I wanted to implement sth like that, but never had enough time to play around with that).
Another option is to use Argo Workflows/Tekton (any other in-cluster job runner). Should be also fine and should provide visibility if the tests succeeded on the level of helm release.
Final option is to run it outside of clsuter, but then monitoring of the execution has to be done additionally.