Governor Telemetry Tap: dramatically lighter, safer, and self-healing

What's new

Your tap traffic is now ~85% smaller — and it's not just compression.

• The tap now samples what it captures instead of mirroring everything. Host metrics volume dropped ~85% (per-CPU cores, loopback disks, and container network interfaces are pruned; one CPU core, physical disks, and real NICs are kept so nothing important disappears). You get the same picture of your fleet at a fraction of the cost.
• Everything the tap sends is now binary + zstd compressed instead of JSON. Wire size dropped again on top of the sampling, and your collectors spend noticeably less CPU serializing it.

Net effect: turning on the tap for 1,000 collectors no longer means 1,000 parallel firehoses hitting the Governor endpoint. You can leave it on for longer, on more collectors, without thinking about it.

The tap can no longer break your collectors.

• Before any tap config reaches your collector, it's now validated against your actual collector binary. If your collector is too old to support a feature (or runs a vendor distribution missing a component), the tap automatically simplifies itself or steps aside — it never blocks your telemetry, and it never crash-loops your collector.
• If a tap misconfiguration somehow gets through, the supervisor rolls it back automatically and keeps your primary pipelines running. The tap is, and will remain, optional garnish — never in the critical path.

One-click onboarding groundwork (visible soon).

The tap pipeline now reports exactly what's running on each node — which receivers, processors, and exporters are active — laying the foundation for config recommendations in an upcoming release.

Bug fixes

• Fixed a defect where collectors receiving a tap config for the first time (no pre-existing sampler) could get an incomplete tap pipeline. All collectors now receive the full, correct tap configuration on first apply.
• Fixed a flaky internal verification test; the live-binary test suite — which validates real collector versions v0.70 through v0.161 — now passes deterministically. More credible CI, fewer false alarms for you.

Under the hood (no action needed)

• We're running a 7-day evaluation of the OpenTelemetry Arrow protocol on a small canary cohort, measuring whether it beats our current protobuf+zstd wire. This runs in an isolated sidecar process — zero impact on your collectors or data, even if it crashes. Results will drive a future decision; we'll only ship it fleet-wide if it measurably wins against pre-registered thresholds.
• The tap now supports collectors from v0.70.0 through the latest contrib, with automatic per-version feature negotiation (modern vs. legacy config syntax, compression tiers).

Upgrading

• No config changes required. Deploy the CollectorCtrl server first, then the supervisors — the tap upgrades itself in place, and old tap configs continue to work during the transition.
• Canary users: you may briefly see two tap formats in flight; both are accepted, and the mix decays to the new format automatically.