feat: functional end-to-end testing environment#4
Merged
Conversation
This introduces a complete and functional testing environment for testing the new activity service in a local kind cluster. The test environment is built on top of the [test-infra] cluster that includes base services like flux, envoy gateway, and a telemetry stack. The test environment includes a Vector, NATS, Clickhouse pipeline that automatically collects audit logs emitted from the test-infra kind cluster. The apiserver deployment manifests are structured as a standard `base` kustomize deployment that includes the kubernetes Deployment, Service, and RBAC resources. The following kustomize components have been introduced to provide optional functionality that can be enabled in environments when necessary. - api-registration: Configures the APIService registration with the k8s apiserver to proxy requests to the activity apiserver - cert-manager-ca: Configures a namespaced cert issuer to use with the activity apiserver - grafana-clickhouse: Configures a new Grafana datasource to connect to the deployed clickhouse instance - namespace: Creates a namespace to use for the system's deployment - nats-stream: Creates a new nats JetStream to use for the audit log pipeline - tracing: Configures the APIserver with a tracing configuration - vector-aggregator: Deploys an aggregated version of Vector that ingests audit logs from NATS and writes them to Clickhouse - vector-sidecar: Deployment of Vector that runs on every node in the cluster that's responsible for collecting audit logs from apiservers and writing them to NATS Also included is deployment automation for deploying the system's dependencies: - Clickhouse operator: Manages deployments of Clickhouse through CRDs - NATS: Deploys an instance of NATS and NACK to configure NATS through CRDs [test-infra]: https://github.com/datum-cloud/test-infra
scotwells
requested review from
JoseSzycho,
drewr,
ecv,
mattdjenkinson and
zachsmith1
ecv
approved these changes
Dec 17, 2025
scotwells
added a commit
that referenced
this pull request
Dec 17, 2025
## Summary The Activity CLI is available for consumers to easily build CLIs that can interact with the Activity API. The activity CLI includes the following commands: - `activity` - Entrypoint to all commands available with the activity CLI - `query` - Command to execute queries against the API This approach gives users the option of how they want to integrate the CLI and provide a native experience to their users. CLIs (e.g. [datumctl](https://github.com/datum-cloud/datumctl)) can choose to import just the `query` command if they wish. This also includes a [kubectl-plugin] named `activity` so that kubectl users can use `kubectl activity` to interact with the activity API. I've also included an auto-generated Golang client that can be used to interact with the activity API. [kubectl-plugin]: https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/ ## Details I was able to test this by installing the `kubectl-activity` command to my `$PATH` and used kubectl to query the test environment created in #4. ```shell $ go build -o ~/go/bin/kubectl-activity ./cmd/kubectl-activity $ kubectl activity query --filter='objectRef.resource != "leases"' --limit 10 TIMESTAMP VERB USER NAMESPACE RESOURCE NAME STATUS 2025-12-17 17:22:56 watch system:kube-controller-manager updaterequests 200 2025-12-17 17:22:56 create system:serviceaccount:activity-system:activity-apiserver subjectaccessreviews 201 2025-12-17 17:22:55 get system:anonymous 200 2025-12-17 17:22:55 get system:serviceaccount:kyverno:kyverno-admission-controller 200 2025-12-17 17:22:54 watch system:kube-controller-manager volumeattachments 200 2025-12-17 17:22:54 watch system:apiserver persistentvolumes 200 2025-12-17 17:22:53 get system:anonymous 200 2025-12-17 17:22:53 get system:serviceaccount:kyverno:kyverno-admission-controller kyverno secrets kyverno-svc.kyverno.svc.kyverno-tls-pair 200 2025-12-17 17:22:53 get system:serviceaccount:kyverno:kyverno-admission-controller kyverno secrets kyverno-svc.kyverno.svc.kyverno-tls-ca 200 2025-12-17 17:22:53 watch system:serviceaccount:telemetry-system:telemetry-system-vm-kube-state-metrics services 200 More results available. Use --continue-after 'eyJ0IjoiMjAyNS0xMi0xN1QyMzoyMjo1My4xOTM3MjhaIiwiYSI6Ijc1ZTc4ZGVjLTA3ODQtNGY2OS1hY2NlLWM4OGFhOTQ0ZDUzNyIsImgiOiJDZ1FrUHh5S2NCT2NkTEUyNm9meDhBPT0iLCJpIjoiMjAyNS0xMi0xN1QyMzoyMzowMi4xODM3NjU5MjhaIn0=' to get the next page. Or use --all-pages to fetch all results automatically. ``` ## Up Next - CI / build pipeline - Operational dashboards & performance testing --- Relates to https://github.com/datum-cloud/engineering/issues/90
Contributor
|
After a while my VM finally seemed to get through the setup, BUT it doesn't think it wasn't successful. Thoughts? |
Contributor
Author
|
That error is because I removed the observability stack before committing the changes because I want the observability stack to be its own PR. Seems like everything deployed fine otherwise |
Contributor
|
Copy that! |
Contributor
|
None of the commands seem to work yet: Poked around the test-infra repo to see if I could reverse engineer the intent but there aren't any PRs or branches there to match. Is it supposed to just reference the root Taskfile there? |
Contributor
Author
|
That's from me merging this PR datum-cloud/test-infra#18 earlier today. Just need to cut a new release and adjust the taskfile in the activity repo to use the tagged version. |
Contributor
Author
|
Fixed in #7 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR builds on datum-cloud/engineering#1 and datum-cloud/engineering#2 to introduce a complete and functional end-to-end testing environment for testing the new activity service in a local kind cluster. The test environment is built on top of a test-infra cluster that includes base services like flux, envoy gateway, and a telemetry stack.
The test environment includes a Vector, NATS, Clickhouse pipeline that automatically collects audit logs emitted from the test-infra kind cluster.
graph LR APIServer[Activity API Server<br/>Generates audit logs] Vector1[Vector Sidecar<br/>Publishes events] NATS[NATS JetStream<br/>Event storage & routing] Vector2[Vector Aggregator<br/>Batching & persistence] CH[ClickHouse<br/>Long-term storage] QueryAPI[Activity API Server<br/>Query interface] Client[Clients<br/>kubectl/API] APIServer -->|writes| Vector1 Vector1 -->|publish| NATS NATS -->|push| Vector2 Vector2 -->|insert| CH Client -->|query| QueryAPI QueryAPI -->|CEL → SQL| CH CH -->|results| QueryAPI style APIServer fill:#e1f5ff style NATS fill:#fff3e0 style CH fill:#f3e5f5 style QueryAPI fill:#e8f5e9 style Vector1 fill:#fff9c4 style Vector2 fill:#fff9c4Details
The apiserver deployment manifests are structured as a standard
basekustomize deployment that includes the kubernetes Deployment, Service, and RBAC resources.The following kustomize components have been introduced to provide optional functionality that can be enabled in environments when necessary.
Also included is deployment automation for deploying the system's dependencies:
After deploying a fresh test environment, I used a kubectl-plugin to query the activity API and retrieve audit logs that have been collected through the pipeline.
Here's a dashboard I'm working on that was loaded into the local environment to visualize the health and performance of the pipeline. The source for this dashboard will be included in a follow up PR.
Up Next
Relates to datum-cloud/enhancements#536