Skip to content

Releases: tronyx/caretta

v0.2.1

Choose a tag to compare

@github-actions github-actions released this 13 Oct 14:56

Caretta and its chart work exactly as in v0.2.0. This is the first release that is signed, with an SBOM and a provenance record for each image. It also adds the documentation pages and the wiki, tests on each long-term kernel from 5.4, and tested steps to upgrade in place from groundcover's chart.

Project

  • examples/astronomy-shop-values.yaml deploys the OpenTelemetry Demo (the "Astronomy Shop") with Helm: a production-like shop of about 15 services in 10 languages, with gRPC, an Envoy edge proxy, Kafka, PostgreSQL, a Valkey cache, feature flags and a load generator, trimmed to fit on one node next to Caretta. The README now lists both example apps.
  • The Markdown files pass markdownlint, with the rules in .markdownlint.json. The README's broken "made by groundcover" link is fixed.
  • CI lints the Markdown files on every pull request, so these problems can't come back.
  • The README no longer says Docker for Mac is unsupported: Caretta runs on Docker Desktop, whose recent releases have a Linux kernel with BTF, and it's tested there on a kind cluster.
  • Every container in both example apps has a CPU limit, so one runaway service can't starve the rest of the cluster. One did: after a restart, the Astronomy Shop's flagd leaked connections and climbed to about 14 CPUs. The Astronomy Shop also gets CPU requests close to what each service uses under load, since without them Kubernetes would reserve the full limits; the limits leave room for its Java and .NET services to start.
  • scripts/build/download_libbpf_headers.sh passes shellcheck, and a failed download of the libbpf headers now always stops the build with a clear message. Before, the failure only showed if tar happened to notice the broken download.
  • The README is rewritten in plain English, in the order a new user needs it: what Caretta does, requirements, installing, opening Grafana, configuration, metrics, upgrading and the example apps. It adds steps for clusters that enforce Pod Security Admission, using your own Prometheus, Caretta's own health metrics and the upgrade from groundcover's chart. Its example queries now work. The three in the old README didn't parse.
  • The documentation is split into pages in docs/: Installation, Upgrading, Configuration, Metrics, How it works, Troubleshooting and How it is tested. They add what the README didn't have room for: every chart value, every metric and label, the capabilities the agent needs and why, what each startup error and log message means, and what Caretta can't see. The README is shorter and links to each page.
  • The pages in docs/ are copied to the GitHub Wiki each time they change on main, so the wiki always matches the latest release. Edit the pages in docs/, since changes made in the wiki are overwritten.
  • CI also runs the eBPF tests on other kernels, each in a VM made by vimto: every long-term kernel from 5.4 to 6.18, and the newest stable kernel. Before, they only ran on the kernels of GitHub's runners, which are all 6.x. All of them pass.
  • The docs no longer say that Caretta runs on kernels 4.16 to 5.7. The new tests showed that kernels before 5.4 usually have no BTF, and Caretta can't load without it. Kernels 5.4 to 5.7 work, with SYS_ADMIN instead of the default capabilities.
  • Releases are signed with cosign, without a key: the signature proves that this repository's release workflow built the image or chart from the release's tag. Each image also carries an SBOM and a provenance record, which the signature covers. The develop- images are signed the same way. The Installation page shows how to check them.
  • The upgrade in place from groundcover's last chart, 0.0.16, is now tested, and the Upgrading page has the steps. It works with --reset-values and your values file, without groundcover's image and Pod Security Policy values. With --reset-then-reuse-values, a pinned groundcover image tag stays, and the upgrade fails.

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 12 Oct 16:14

Bug fixes found by a new test suite, which now checks every change on real amd64 and arm64 kernels and on a kind cluster before anything is published.

Upgrading: if you set serviceAccount.create: false, also set serviceAccount.name to the account caretta should use. Until now the chart created its own account regardless; without a name, caretta now runs as the namespace's default account, which can't read the cluster.

Fixed

  • When two links ended up with the same labels, caretta_links_observed showed only one of them, picked at random on each poll, so its value could jump between scrapes and look like counter resets. Their byte counts are now added together.
  • A PROMETHEUS_PORT of 0, a negative number or one above 65535 is ignored, like other invalid settings, instead of starting the metrics server on a random port that every probe then fails to reach.
  • The chart's serviceAccount values now take effect. serviceAccount.annotations are added to the service account, for example for cloud workload identity such as EKS IAM roles. serviceAccount.create: false uses an existing account, named by serviceAccount.name, instead of creating one anyway. By default the account is still called caretta, so existing installs don't change.

Project

  • Unit tests now cover the health checks, the metrics server, configuration parsing, the poll loop and shutdown order (the eBPF maps are never unloaded while a poll is reading them), loopback filtering, and the resolver's handling of deleted pods, nodes and services. Coverage goes from 50% to 77%. A bug in the resolver's test harness, which never applied modified workload resources, is fixed.
  • CI loads the real eBPF programs into the kernels of an amd64 and an arm64 runner and checks what they record for real TCP connections: both sides of a connection with the right roles, ports, process and byte counts, closed connections marked as such, and a poll turning them into links and deleting them from the kernel's map. It also catches Go and C struct layouts that don't match. Until now, nothing tested the eBPF code before release.
  • The Helm chart has tests (chart/tests, run with helm-unittest in CI). They check caretta's security context and capabilities, the read-only tracefs mount, the configuration it's given, the probes, RBAC, the service account, every NetworkPolicy case (including that one never lets in traffic from anywhere), pinned images with no latest, and Grafana's generated password and disabled anonymous access.
  • An end-to-end test runs on a kind cluster in CI, with the image built in the same run. It installs the chart with the NetworkPolicies on, starts the demo app, and checks that caretta maps each of its connections, that VictoriaMetrics scrapes it, that Grafana requires a login and can query VictoriaMetrics, and that caretta shuts down without errors. Images are only published once it passes. The same test runs locally with scripts/e2e.sh, and the README explains how to run every kind of test.

v0.1.0

Choose a tag to compare

@github-actions github-actions released this 12 Oct 14:51

The fork's first release.

Changed

  • Every container image is pinned to an explicit, current version, and none uses latest. Grafana goes from 9.3.1 to 13.2.3 (distroless) and VictoriaMetrics from v1.85.3 to v1.153.0, both now pulled from their upstream registries instead of groundcover's Quay mirrors, which stopped being updated in 2023. Grafana's helper images (k8s-sidecar, busybox, curl, bats and the image renderer, which defaulted to latest) are pinned too. The Grafana subchart moves to the grafana-community repository, where it has lived since January 2026.
  • The caretta image is built from golang:1.27.2-trixie instead of a 2022 builder image that shipped Go 1.18 or older, and runs on gcr.io/distroless/static-debian13 (pinned by digest) instead of Alpine 3.17, which is end-of-life. It starts with ENTRYPOINT ["/caretta"], so stop signals reach caretta itself instead of a shell.
  • All Go dependencies are current, among them cilium/ebpf v0.22.0 and the Kubernetes client v0.37.1. Go now sizes its thread pool to the pod's CPU limit, instead of the node's core count.
  • golang.org/x/net is at v0.60.0, which fixes five HTTP/2 vulnerabilities Trivy found in v0.59.0, the worst a high-severity denial of service (CVE-2026-78669).
  • Caretta no longer runs as a privileged container. It runs as root, which loading eBPF probes requires, but with every capability dropped except BPF, PERFMON and SYS_RESOURCE, a read-only root filesystem, no privilege escalation and the RuntimeDefault seccomp profile. It no longer mounts the host's /proc and debugfs, only tracefs, read-only, at a path set by the new tracefsPath value. This needs kernel 5.8 or newer; the README says what to change for older kernels.
  • VictoriaMetrics runs as an unprivileged user (65534) with no privilege escalation, every capability dropped, a read-only root filesystem and the RuntimeDefault seccomp profile. It ran as root before.
  • Caretta's ClusterRole grants only what it reads: list and watch on pods, nodes, Services, ReplicaSets, DaemonSets, StatefulSets, Deployments, Jobs and CronJobs. It could read about 40 kinds of objects across the cluster before, among them every ConfigMap (where credentials often end up), pod logs, events and ServiceAccounts, which anyone who took over a caretta pod could have read too.
  • The bundled Grafana no longer has the fixed admin password caretta, which anyone who knew Caretta could look up: a random one is generated into the caretta-grafana Secret at install time and kept across upgrades, and grafana.admin.existingSecret can name a Secret of your own instead. Anonymous access is off, so the dashboards need a login; it can be turned back on in grafana.ini. An upgrade keeps the password already in the Secret, so an existing install keeps caretta until it's rotated; the README says how, and to upgrade with --reset-then-reuse-values, since --reuse-values would also keep the old password setting and anonymous access.
  • Caretta has /livez and /readyz endpoints, and the chart sets startup, readiness and liveness probes. Readiness waits until the cluster's state is loaded and the eBPF probes are attached; liveness only checks that caretta is still polling, never the Kubernetes API, so an API server outage can't restart caretta on every node at once.
  • Caretta is ready in seconds: it waited a fixed 10 seconds before attaching its probes, and now attaches them as soon as it has loaded the cluster's state.
  • Caretta follows the cluster through client-go informers instead of nine hand-written watches, and caches only the fields it uses from each object, so each node holds far less of the cluster in memory. The flood of couldn't retrieve owner warnings at startup is gone, since every cache is loaded before the first pod is resolved. A deleted pod's IP is forgotten 2 minutes after the pod is gone, and a deleted node's or Service's at once, so a reused IP never keeps an old name.
  • A link leaves the metrics once it hasn't been seen for an hour, set by the new linkTTL value (0 keeps links for as long as caretta runs, as before). A link that comes back after that starts again from zero, which rate() and increase(), and so the bundled dashboard, handle as a counter reset.
  • Caretta logs one JSON object per line, with a level and the details in separate fields ({"time":…,"level":"WARN","msg":"Skipping poll","error":…}), so log collectors such as Fluent Bit or Promtail can filter and parse it; the Kubernetes client's own messages come out the same way. The new logFormat value switches to plain key=value text. Warnings and errors are marked as such, where every line used to be the same plain text.
  • The new caretta_poll_duration_seconds histogram shows how long each poll takes, so a poll that's slowing down, for example as a node's connections grow, shows up before it trips the liveness probe.
  • The new networkPolicy.enabled value adds NetworkPolicies: only the bundled VictoriaMetrics can scrape caretta, only the bundled Grafana can query VictoriaMetrics, and both can only reach DNS and the Kubernetes API server. Other scrapers or dashboards can be let in with networkPolicy.metricsFrom and networkPolicy.queryFrom. Off by default, since it needs a network plugin that enforces NetworkPolicy.
  • How many connections caretta can track per node is the new maxConnections value, 1,000,000 by default as before. The kernel reserves memory for all of them when caretta starts, about 192 MiB at the default, which is most of caretta's memory, so lowering it saves memory on every node.
  • Caretta's resources match what it uses: a 256Mi memory request (it was 50Mi, below the eBPF maps alone, which made caretta an early eviction target), a 512Mi limit, and a 500m CPU limit instead of 150m, so a poll of a busy node isn't throttled. Go's memory limit is set to 256MiB by the new goMemLimit value, so Go frees memory before the pod reaches its limit.

Fixed

  • Every graceful stop exited with code 1, so each rolling update, drain or eviction looked like a crash: the metrics server's normal shutdown was reported as a fatal error. Caretta now exits with code 0, and a port that's already taken is reported at startup instead of later.
  • Shutdown closed the eBPF maps while a poll could still be reading them, and had no time limit. Caretta now stops polling and waits for the poll to finish first, and finishes within 20 seconds, inside the pod's 30-second grace period.
  • When re-opening a dropped watch failed, the resolver crashed on its next event, which could take caretta down on every node at once during an API server blip.
  • A new Service that reused a deleted Service's cluster IP was never mapped, and stayed under the old Service's name until caretta restarted.
  • POLL_INTERVAL=0 made caretta crash at startup; a value of 0 or less is now ignored.
  • When reading the kernel's connection map failed partway, which can happen on a busy node when the map keeps changing during the read, caretta published what it had read as if it were everything, so links it hadn't reached dropped out of the metrics for that poll. Such a poll is now skipped: nothing is published or removed from the map, the metrics keep the previous poll's numbers, and the error is logged and counted in the new caretta_failed_map_iterations metric. If every poll keeps failing, the liveness probe restarts caretta after 30 seconds.
  • A slow or unreachable DNS server could stall caretta's polling: with enableDnsResolving on, the default, each new external IP was looked up in the middle of a poll, one at a time and with no time limit, so a node seeing many new external IPs at once, as right after startup, could go minutes without a poll, until the liveness probe restarted caretta and it started over. Lookups now run in the background with a 2-second limit, and a poll never waits for one: a new external IP shows under its address until its name arrives, usually by the next poll, and traffic counted under the address before then stays there until linkTTL drops it. Names are looked up again after an hour, so a changed DNS record is picked up (the old name stays in use until the new one arrives), and a failed lookup is retried after 10 minutes instead of on every poll.
  • Grafana 13 uses about 290 MiB on its own, which left it at the edge of its 300Mi limit, so a dashboard with heavy queries or a grafana cli command got it killed. Its limit is now 512Mi with a 320Mi request, and VictoriaMetrics' memory request is 96Mi, closer to the 70 MiB it uses.
  • Caretta's memory, and the number of series in VictoriaMetrics, grew for as long as it ran: caretta_links_observed and caretta_tcp_states kept every label set they had ever seen, so each pod that came and went and each closed connection stayed in the metrics, until a busy cluster's caretta pods ran out of memory. The metrics now show only the latest poll: a closed connection leaves caretta_tcp_states at the next poll, and an idle link leaves caretta_links_observed after linkTTL. The metric names and labels are unchanged.

Removed

  • The Go profiler (/debug/pprof) is no longer served on the metrics port, where anyone who could reach the pod could pull heap dumps or run CPU profiles.
  • The caretta_watcher_resets_count metric, which informers have no equivalent for. caretta_watcher_events_count is still there, for pod, node and Service events.
  • Support for batch/v1beta1 CronJobs, which Kubernetes removed in 1.25.
  • The chart's PodSecurityPolicy, its ClusterRole rule and the rbac.pspEnabled value. Kubernetes removed PodSecurityPolicy in 1.25, so they rendered on no supported cluster; Pod Security Admission replaces it, and the README says how to label caretta's namespace for it.

Project

  • Releases publish the image to both GHCR and Docker Hub, and the Helm chart to GHCR as an OCI artifact: `helm install caretta oci://ghcr.io/tronyx/charts/caretta --versio...
Read more