Repository navigation
Releases
Release v0.12.0
Released: 2026-11-16 · Branch: main · Previous: v0.11.12 (2026-10-31) · Tracking: #1206
A minor release that gets the operator ready to publish as open source. It adds the license and contributor documents and a public README, and it removes Pixie and the observability kinds built on it. It also carries security fixes for tenant isolation, datastore credentials and Zetta's secrets, moves Orama's own cluster resources out of the repository, and pins zetta-tunnel v0.2.0. It was cut without a planned release epic, as part of the open-source release (#1148).
| Measured | Value |
|---|---|
| Resource kinds | 49 CRDs in chart/crds (54 in v0.11.12; five removed) |
| Pull requests | 32 merged since v0.11.12 |
| CRD changes | removed: ObservabilityProvider, ObservabilityHub, ObservabilityAgent, PixieCloud, PixieVizier. Added fields: OramaZettaCluster spec.egressAllowConfigMap and the identity-root rotation; Lake fields for the tenant-isolation and service changes; OramaCloudV1 spec.components.controlDB.referenceRows and spec.components.pgAdmin.email |
| License | MIT, with COPYRIGHT, SPDX headers on every source file, THIRD_PARTY_LICENSES and an SBOM |
| zetta-tunnel | v0.2.0 (main-10acaa2, by digest), from v0.1.0 |
| Tests | see Verification |
What it includes
Open-source release
- License and notices (#1194): MIT
LICENSEandCOPYRIGHT(OramaSearch, Inc.),SPDX-License-Identifier: MITon every source file (tools/spdx-headers.py),THIRD_PARTY_LICENSESfromgo-licenses, andsbom.spdx.json. - Contributor documents:
SECURITY.md, a private reporting address and response times (#1161);CONTRIBUTING.md,CODE_OF_CONDUCT.mdandCLA.md(#1190). - A public README (#1202) with a quickstart that was run on a throwaway cluster.
make runnow passes--control-plane-bind-address="", so it starts without a control-plane certificate. - No internal material in the tree: comments, CRD descriptions, samples and chart values (#1193), test fixtures (#1195, #1188), and the public docs (#1196). The design log, the security reports and the research notes moved out of the repository (#1190).
- Cluster resources are an example (#1149):
config/clusters/holds a cleansed example deployment. The tests read it, or another directory named byORAMA_CLUSTERS_DIR. - Cloud v1 configuration instead of embedded data. A stack's plan catalog is
spec.components.controlDB.referenceRows. The built-in rows are a made-up example that refuses to run on a database that already has a catalog (#1185). pgAdmin's sign-in isspec.components.pgAdmin.email,admin@example.comby default (#1189).
Pixie removed (#1199)
ObservabilityProvider, ObservabilityHub, ObservabilityAgent, PixieCloud and PixieVizier are removed, with their controllers, CRDs, RBAC, the vendored Pixie manifests and the Pixie build material. The hub Prometheus and its federation, and the per-cluster alert routing, go with them; nothing replaces them in this release. The platform layer's Prometheus, Alertmanager and Perses stay, with each Cloud v1 stack's rules and dashboards. The manager also loses its alertmanagerconfigs and prometheuses grants. make check-crds now fails while an object of a removed kind remains (hack/retired-crds.txt).
Security
- A person's service launch gets its own token, never the Lake's agent service token (#1150).
- An explicit CIDR egress rule reaches public addresses only (#1153), and a tenant-named peer remote must be public (#1154).
- The default images that run tenant code are pinned by digest (#1155).
- The operator verifies a workspace Postgres's certificate before signing in as superuser (#1156).
- Image builders never mount the platform pull Secret (#1159).
- An app's bindings copy only its own workspace's services (#1160).
- Tenant services never run in the Lake namespace. Leftovers there are reported as a condition and a Warning event, never deleted (#1181).
- Five findings in one batch (#1162):
- TiDB's root password is set while TiDB bootstraps;
- a lake secret's projection is deleted when lake says the secret is gone;
- a row's workspace is checked against its customer;
- methodology rows and agent pods are bounded on the operator's side;
- mesh projections carry the pod UID and namespace inode.
- TiDB and Cassandra tenants get a user of their own, never the administrator (#1171). Each service's administrator password is then rotated once (#1173).
- Zetta: a connector-credential root Secret mounted outside the data volume, and the egress allowlist as a file the engine re-reads (#1167). Identity-root rotation as a spec generation, and the enrollment CA (#1177). Each is switched on by what the image's own
--helplists.
Datastores and Cloud v1
- lake-duckdb v0.2.0 is the default DuckDB image (#1204). It fixes lake-duckdb's security findings, whose reproductions are public. lake-sqlite v0.2.0 is the default SQLite image (#1203). Both are pinned by digest.
- Cloud v1:
Other
- zetta-tunnel v0.2.0 (#1138), which the mesh projection needs.
- Seventeen tests the untested merges left failing, fixed on
devbefore the cut:
What it does not include
- Any replacement for Pixie, the hub metrics, or alert delivery.
- Public images. Every default image, and the operator's own, is in a private registry (#1186).
- These open pull requests:
- #1164 with #1170, #1175 and #1176: apps stop being served until each Lake has its own wildcard domain.
- #1178: moves DNS for every Cloud v1 host.
- #1163 and #1157: their lake halves are not merged.
- #1180: waits on a zetta-tunnel change.
- #1137: denies the cluster's own addresses in the egress gateway snapshot, not yet run against zetta-tunnel v0.2.0.
- #857, #860 and #864: on hold for a planned rollout.
- The backup system: #1182 to #1184.
Before it rolls
Each step needs the cluster owner's consent, in this order:
- Remove the observability resources under the operator that still ships them (v0.11.12). Delete the
ObservabilityAgents,ObservabilityHubs and theObservabilityProviderfrom their committed files as they were before this release, and wait until they are gone. The managing operator tears down what each one wrote on its clusters. - Apply the Cloud v1 set files that carry the new fields before the operator rolls:
controlDB.referenceRowswith each stack's current catalog. Otherwise each stack's reference-data Job fails its guard, and its catalog stays unchanged.pgAdmin.emailon the stacks that run pgAdmin. Otherwise pgAdmin signs in asadmin@example.comfrom its next restart.
make check-crds CONTEXT=<context>on every cluster. It fails while an object of a removed kind remains.- Roll the operator, with the CRDs applied widened first and exactly after (
docs/operations/crd-rollout.md). Then apply the platform layer by both installation paths: kube-state-metrics stops listing the Pixie kinds. - lake must carry its half of #1150 (lake#1276) in the same roll. Until both run it, a person's service launch answers 401.
- Roll this operator before the Zetta images that read the connection-root file and the identity rotation (zetta#1509, zetta#1512, zetta#1514).
What changes on the roll:
...
Release v0.11.12
Released: 2026-10-31 · Branch: main · Previous: v0.11.11 (2026-10-31).
A patch release: one Pixie hub shows every cluster, and Pixie keeps each user's settings. Production's hub on the managing
cluster now takes the managing cluster itself and nonprod, as well as orama-production-cluster-02 and -03 (Steve, 2026-10-31,
replacing #456's one hub per purpose). The welcome tour no longer comes back on every page load: the bootstrap now writes the
per-user rows Pixie's settings are saved in.
| Measured | Value |
|---|---|
| Resource kinds | 54 CRDs in chart/crds (unchanged) |
| CRD changes | ObservabilityAgent: spec.clusterRef optional (empty is the managing cluster), new optional spec.alerting |
| Operator code | the Pixie bootstrap (per-user rows), the observability purpose rule, agent alert routing |
| Lake schema | nothing vendored |
| Tests | see Verification |
What it includes
| PR | Change |
|---|---|
| #1124 | Settings (#1121). The bootstrap gives each Pixie user its user_attributes and user_settings rows, which Pixie's own user creation writes and its settings updates need; existing users get them on the next reconcile. The bootstrap role gains SELECT, INSERT on the two tables. The managing cluster's agent (#1122). An ObservabilityAgent with no clusterRef runs on the managing cluster, beside production's hub without a link, registered under the hub's name. Nonprod on production's hub (#1122). The managing cluster's hub accepts nonprod agents; production still never reports to a nonprod hub. An agent may name its own alert receivers (spec.alerting); a nonprod agent on production's hub pages only through those, so production's receiver Secrets never reach nonprod. config/clusters: the managing cluster's agent in observability-hub.yaml; nonprod's agent reports to production's hub and keeps #monitoring-stg; nonprod's own hub is retired from the file |
What it does not include
- Any change to the platform charts, Lake or Cloud v1.
- Applying the new agent files or deleting nonprod's hub: each needs Steve's consent after the roll.
- The fix for the hub Prometheus not reaching the production leaves
(#1111).
Upgrading
- Apply the CRDs first:
ObservabilityAgentdropsclusterReffromrequiredand gainsalerting. Additive; existing agents
are unchanged. - No pod restarts from the operator code itself; the hub's bootstrap writes two rows per user on its next pass.
- After the roll, with consent: apply
config/clusters/observability-hub.yaml(the managing cluster's agent: Pixie's node agents
on the managing cluster) andconfig/clusters/orama-nonprod-cluster-01/observability.yaml(nonprod's agent moves to production's
hub: its Vizier re-registers there and its in-memory trace history restarts), then deleteObservabilityHub/nonprodon the
managing cluster, which removes nonprod's Pixie Cloud and its databases (sign-ins only). - What one hub costs: nonprod gets a PeerLink to production's hub like each production cluster: vzconn only, its own deploy
key, the hub's CA; the hub reads back its read-only metrics proxy. Nonprod's "cluster lost" alerts come from the hub's rules and go
to#monitoring.
Known issues
- #1111:
ObservabilityClusterUnreachablefires falsely for the
production leaves. - #1104: kuberesolver 401 log noise on the hub's API server.
- Pixie shows clusters as degraded ("non-operator version of Pixie"): cosmetic, because this operator installs Vizier instead of
Pixie's own operator. - Licensing: Pixie's eBPF programs are GPL-2.0 (orama-bridge#268).
Verification
| Check | Result |
|---|---|
go build ./..., go vet ./... |
pass |
go test ./... with envtest (Kubernetes 1.30.3) |
not run for this release, on Steve Sperandeo's instruction ("we can skip the test this time"); stopped while running. On #1124's branch, go test ./... -count=1 passed (7 packages, no failures; run without envtest assets, so envtest suites may have skipped) |
go test ./cmd/ |
passes on the release branch, TestReleaseVersionIsDocumented included |
helm lint chart |
1 chart(s) linted, 0 chart(s) failed |
| Pull request | #1124 approved by Steve Sperandeo on 2026-10-31 |
| Live | rolled after the release; recorded in techops-docs |
Release v0.11.11
Released: 2026-10-31 · Branch: main · Previous: v0.11.10 (2026-10-31).
A patch release: the Pixie UI can run scripts. Signed in and with scripts loaded, running px/cluster failed with
"Server error, http status code: 0". Envoy serves gRPC-web on the proxy's port 4444, and its CORS allowed only upstream's
dev.withpixie.dev, so the browser's preflight for ExecuteScript got 403. The release also makes a configuration change roll
the pods that read it: v0.11.10's script bundle URL reached the API server but not the proxy, which kept serving the old one.
| Measured | Value |
|---|---|
| Resource kinds | 54 CRDs in chart/crds (unchanged) |
| CRD changes | none |
| Operator code | the PixieCloud rendering (proxy-envoy-config), and a ConfigMap fingerprint on every Pixie pod template (Cloud and Vizier) |
| Lake schema | nothing vendored |
| Tests | see Verification |
What it includes
| PR | Change |
|---|---|
| #1117 | Envoy's CORS origin becomes the hub's domain (suffix, so work.<domain> is covered); rendering fails if the vendored origin ever disappears. orama.cloud/pixie-config on every Pixie pod template is a hash of the rendered ConfigMaps it reads, so a change rolls exactly those pods. TestPixieCloudRendersNoUpstreamDomain (no rendered object names withpixie.dev, withpixie.ai, px.dev or gcr.io/pixie-oss) and TestPixieConfigChangeRollsThePodsThatReadIt |
What it does not include
- Any change to the platform charts, Lake or Cloud v1.
- The fix for the hub Prometheus not reaching the production leaves
(#1111).
Upgrading
- No CRD changes.
- Every Pixie pod restarts once: the hub on the managing cluster and on nonprod, and the agent on all three leaves, because each
pod template gains the fingerprint. Pixie keeps its traces in memory, so each node's recent trace data is lost. No Lake or Cloud v1
workload changes. - People reaching the UI forward the proxy's port 4444 too (script runs), as well as 443.
Known issues
- #1111:
ObservabilityClusterUnreachablefires falsely for the
production leaves. - #1104: kuberesolver 401 log noise.
- Licensing: Pixie's eBPF programs are GPL-2.0 (orama-bridge#268).
Verification
| Check | Result |
|---|---|
go build ./..., go vet ./... |
pass |
go test ./... with envtest (Kubernetes 1.30.3) |
3,198 passed, 0 failed, 22 skipped with subtests (1,221 top-level tests). Skipped: the opt-in suites that need a local Postgres, a live zetta-tunnel, a live chart download, and a Prometheus binary |
helm lint chart |
1 chart(s) linted, 0 chart(s) failed |
| Not run before the release | a script run through a live hub; checked after the roll |
| Pull request | #1117 approved by Steve Sperandeo on 2026-10-31 |
| Live | rolled after the release; recorded in techops-docs |
Release v0.11.10
Released: 2026-10-31 · Branch: main · Previous: v0.11.9 (2026-10-31).
A patch release: the Pixie UI and the hub's API server load the PxL script bundle. Once people could sign in (v0.11.9), the UI
showed "Failed to Load Script Bundles (Network Error (SCRIPT_BUNDLES_OVERRIDDEN))":
- The UI's fetch was cross-origin. It runs on
work.<domain>, but it was givenhttps://<domain>/pxl_scripts/bundle.json,
another origin that sends no CORS headers, so the browser refused it. - The API server couldn't resolve the address. It was given the same URL, which nothing in the cluster resolves (#1112).
| Measured | Value |
|---|---|
| Resource kinds | 54 CRDs in chart/crds (unchanged) |
| CRD changes | none |
| Operator code | the PixieCloud rendering: pl-script-bundles-config and the API server's environment |
| Lake schema | nothing vendored |
| Tests | see Verification |
What it includes
| PR | Change |
|---|---|
| #1113 | The UI gets SCRIPT_BUNDLE_URLS=["/pxl_scripts/bundle.json"], a path on its own origin that the proxy already serves. The API server gets its own SCRIPT_BUNDLE_URLS, the bundle server's Service (http://pixie-scripts.plc.svc:8080/pixie-scripts/bundle.json). A render test pins both. No Pixie image changes |
What it does not include
- Any change to the platform charts, Lake or Cloud v1.
- The fix for the hub Prometheus not reaching the production leaves
(#1111). Until thenObservabilityClusterUnreachablefires for
-02and-03although both are healthy.
Upgrading
- No CRD changes.
- What rolls on each cluster: the operator and the zetta-tunnel mesh pods. On the hubs, the proxy and the API server roll
with their new configuration. No Lake or Cloud v1 workload changes. - A browser that set
px-custom-script-bundle-pathsto work around the error can remove it:
localStorage.removeItem('px-custom-script-bundle-paths').
Known issues
- #1111: the hub cannot federate the production leaves, so
ObservabilityClusterUnreachablefires for them. - #1104: kuberesolver 401 log noise.
- Otherwise unchanged from v0.11.9.
Verification
| Check | Result |
|---|---|
go build ./..., go vet ./... |
pass |
go test ./... with envtest (Kubernetes 1.30.3) |
not run for this release, on Steve Sperandeo's instruction ("skip the test suite... deploy everywhere"); stopped while running. #1113's own tests (go test ./internal/controller/ -run Pixie, go test ./cmd/) passed on its branch |
helm lint chart |
1 chart(s) linted, 0 chart(s) failed |
| Not run before the release | the UI listing scripts on a live hub; checked after the roll |
| Pull request | #1113 approved by Steve Sperandeo on 2026-10-31 |
| Live | rolled after the release; recorded in techops-docs |
Release v0.11.9
Released: 2026-10-31 · Branch: main · Previous: v0.11.8 (2026-10-31).
A patch release: the hub Prometheus writes its configuration, so it federates every cluster and evaluates the hub's alert
rules. After v0.11.7 both hubs and all three agents were Ready, but the hub Prometheus had no scrape targets: prometheus-operator
v0.94 writes no configuration for a Prometheus that sets none of the monitor selectors, so additionalScrapeConfigs and the rule
selector were ignored (Reconciled=True ConfigurationUnmanaged), and every agent reported metricsConnected=false.
v0.11.8 was released but never rolled. This release carries it too: the clusters go from v0.11.7 straight to v0.11.9, with
both fixes.
| Measured | Value |
|---|---|
| Resource kinds | 54 CRDs in chart/crds (unchanged from v0.11.8) |
| CRD changes | none since v0.11.8; from v0.11.7, ObservabilityHub.spec.users and PixieCloud.spec.users |
| Operator code | the hub Prometheus's selectors |
| Lake schema | nothing vendored |
| Tests | see Verification |
What it includes
| PR | Change |
|---|---|
| #1102 | The hub Prometheus sets a scrapeConfigSelector (and its namespace selector) for the hub's own ScrapeConfigs, of which there are none. It is there so prometheus-operator writes the configuration: the federation jobs and the hub's rules then load. TestHubPrometheusIsPinnedAndScoped fails without it |
| #1103, from v0.11.8 | People can sign in to the Pixie hubs: the organisation's domain, and spec.users (v0.11.8) |
What it does not include
- Any change to the platform charts, Lake or Cloud v1.
- Live traces through a PxL script are still unproven (observability.md).
Upgrading
- From v0.11.7, follow v0.11.8's upgrade notes: its two optional CRD fields are widened by the CRD
rollout (crd-rollout.md), and each hub's file is applied after the roll to add itsusers. - What rolls on each cluster with the operator: the operator, and the zetta-tunnel mesh pods, which carry a container from
the operator image. No Lake or Cloud v1 workload changes. On the hubs, the hub Prometheus reloads its configuration.
Known issues
Verification
| Check | Result |
|---|---|
go build ./..., go vet ./... |
pass |
go test ./... with envtest (Kubernetes 1.30.3) |
3,196 passed, 0 failed, 22 skipped with subtests (1,219 top-level tests). Skipped: the opt-in suites that need a local Postgres, a live zetta-tunnel, a live chart download, and a Prometheus binary |
helm lint chart |
1 chart(s) linted, 0 chart(s) failed |
| Not run before the release | the hub Prometheus's targets on a live hub; checked after the roll |
| Pull requests | #1102 and #1103 approved by Steve Sperandeo on 2026-10-31 |
| Live | rolled after the release; recorded in techops-docs |
Release v0.11.8
Released: 2026-10-31 · Branch: main · Previous: v0.11.7 (2026-10-31).
A patch release: people can sign in to the Pixie hubs. After v0.11.7 both hubs ran, but Pixie turned their
administrator away with "Our system found an issue with your account". Pixie's sign-in refuses a member of an
organisation that has no domain, unless the organisation is named after that member's email
(src/cloud/auth/controllers/login.go, loginSimple). The operator had created the organisation orama without a
domain.
| Measured | Value |
|---|---|
| Resource kinds | 54 CRDs in chart/crds (unchanged) |
| CRD changes | ObservabilityHub.spec.users and PixieCloud.spec.users, optional lists of emails |
| Operator code | the PixieCloud bootstrap: the organisation's domain, and one sign-in per user |
| Lake schema | nothing vendored |
| Tests | see Verification |
What it includes
| PR | Change |
|---|---|
| #1103 | The organisation gets the administrator's email domain, and a hub bootstrapped without one has it set: the bootstrap role may update orgs.domain_name, and nothing else of an existing row. spec.users: each email gets a generated password in plc/pixie-user-<email>, an identity in Kratos and a user in the organisation; an email removed loses its identity, then its Secret. Both hubs' files list steve@orama.com. Docs: the bootstrap, the Secrets, the CRD reference, and why the UI must be reached on port 443 |
What it does not include
- Any change to the platform charts, Lake or Cloud v1.
- Live traces through a PxL script are still unproven (observability.md).
Upgrading
- CRDs: two optional fields are added. The CRD rollout (crd-rollout.md) widens them before
the operator upgrade, as for any release. - What rolls on each cluster with the operator: the operator, and the zetta-tunnel mesh pods, which carry a container from
the operator image. No Lake or Cloud v1 workload changes. - On each hub, on the next pass after the roll: the organisation gets its domain, and sign-in works for the administrator.
- Then apply each hub's file (
config/clusters/observability-hub.yaml,config/clusters/orama-nonprod-cluster-01/observability.yaml)
to add itsusers. The operator writes each user's Secret and sign-in within five minutes.
Known issues
- Unchanged from v0.11.7.
- #1104: the hub's
auth-server,profile-serverandapi-server
log a kuberesolver 401 from the Kubernetes API every second. It does not stop sign-in or registration. - The proxy redirects
<domain>tohttps://work.<domain>/without the port, so the UI must be reached on port 443 (techops-docs'
observability runbook gives two ways).
Verification
| Check | Result |
|---|---|
go build ./..., go vet ./... |
pass |
go test ./... with envtest (Kubernetes 1.30.3) |
3,195 passed, 1 failed, 22 skipped with subtests (1,218 top-level tests). The one failure was TestReleaseVersionIsDocumented: the suite started before this page existed. go test ./cmd/ rerun after it was written: pass. Skipped: the opt-in suites that need a local Postgres, a live zetta-tunnel, a live chart download, and a Prometheus binary |
| The PR's own tests | TestPixieCloudBootstrapLetsTheAdministratorAndEveryUserSignIn (new), TestPixieCloudBootstrapIssuesAndRevokesPerClusterKeys and TestPixieBootstrapMatchesVendoredSchema (now covers orgs.domain_name) |
helm lint chart |
1 chart(s) linted, 0 chart(s) failed |
| Not run before the release | the domain update and the Kratos delete against a live hub; checked after the roll by signing in |
| Pull request | #1103 approved by Steve Sperandeo on 2026-10-31 |
| Live | rolled after the release; recorded in techops-docs |
Release v0.11.7
Released: 2026-10-31 · Branch: main · Previous: v0.11.6 (2026-10-31).
A patch release: Pixie's metadata index is created as soon as OpenSearch is ready, so the indexer starts and the
observability hubs can become Ready. After v0.11.6 both hubs were stuck at 19/20: the indexer exits when its index
md_entities_9 is missing, the hub is not Ready without the indexer, and the operator created the index only once the hub was
Ready.
| Measured | Value |
|---|---|
| Resource kinds | 54 CRDs in chart/crds (unchanged) |
| CRD changes | none |
| Operator code | the PixieCloud bootstrap's order |
| Lake schema | nothing vendored |
| Tests | see Verification |
What it includes
| PR | Change |
|---|---|
| #1098 | The bootstrap takes OpenSearch's readiness apart from the hub's: it creates the metadata index once pixie-search is Ready, and does the rest (the administrator, the organisation, the deploy keys) once the hub is Ready, as before. TestPixieCloudBootstrapIssuesAndRevokesPerClusterKeys now expects the index and no keys with only OpenSearch ready |
| #1072 | Docs: how people reach Perses, Prometheus, Alertmanager and the Pixie UI (kubectl port-forward, nothing public), and Verify records that the platform charts run on all four clusters |
What it does not include
- Any change to the platform charts, Lake or Cloud v1.
- The Viziers connecting to their hub and live traces are still unproven
(observability.md).
Upgrading
- No CRD changes. The CRD rollout (crd-rollout.md) runs as for any release and changes nothing.
- What rolls on each cluster with the operator: the operator, and the zetta-tunnel mesh pods, which carry a container from
the operator image. No Lake or Cloud v1 workload changes. On the hubs, the operator then creates the index and the indexer
starts.
Known issues
- Unchanged from v0.11.6.
- #1096: a CloudflareR2Bucket already gone keeps its finalizer;
v1-dev-uploadsgets 403 on permission groups.
Verification
| Check | Result |
|---|---|
Full suite (go test ./... with envtest) |
not run for this release, on Steve Sperandeo's instruction ("skip testing again for this one... let's get it deployed asap") |
| The PR's own tests | go test ./cmd/ ./internal/controller/ passed on #1098's branch (per the PR); the changed test fails without the fix |
go test ./cmd/ -run TestReleaseVersionIsDocumented |
pass, on the release branch |
helm lint chart |
1 chart(s) linted, 0 chart(s) failed |
| Pull requests | #1098 and #1072 approved by Steve Sperandeo on 2026-10-31 |
| Live | rolled to every cluster after the release; recorded in techops-docs |
Release v0.11.6
Released: 2026-10-31 · Branch: main · Previous: v0.11.5 (2026-10-31).
A patch release: the faults that stopped both observability hubs after Pixie's release was applied. With v0.11.5 and the
provider applied, each hub's Prometheus crash-looped and Pixie Cloud never started. Three operator faults are fixed here; the
fourth, the Postgres image's missing tag, was fixed in the committed provider on dev (1e912c0f) and applied the same day.
| Measured | Value |
|---|---|
| Resource kinds | 54 CRDs in chart/crds (unchanged) |
| CRD changes | none |
| Operator code | the hub Prometheus, the Pixie release check, the Pixie prune, Pixie Cloud's dependency ConfigMaps |
| Lake schema | nothing vendored |
| Tests | see Verification |
What it includes
| PR | Change |
|---|---|
| #1091 | The hub Prometheus owns its volume. It set no pod securityContext, so its Azure disk stayed root's and Prometheus (non-root) failed with permission denied on /prometheus. It now runs as uid 1000 with fsGroup 2000, like the platform's kube-prometheus-stack. A Pixie Postgres image must carry its tag: CloudNativePG reads the PostgreSQL version from the tag and refuses an image given by digest alone, so the provider is not Ready until postgres is repo:<version>@sha256:…. config/clusters/observability-provider.yaml already names pixie-postgres:16.15@sha256:34cd4159… (1e912c0f) |
| #1092 | The Pixie prune spares objects another controller owns. CloudNativePG copies the pixie-db Cluster's inherited labels, orama.cloud/pixie: cloud included, onto the ServiceAccount, Services and budget it creates; the prune listed them by that label and deleted them on every pass, so the database pod's token stopped working (Unauthorized). It also tried to delete CNPG's Role and RoleBinding (refused). It now skips an object whose controller is not an orama.cloud kind. pl-sentry-dsn-config is rendered, its three keys empty (Sentry off), as upstream ships it in cloud_deps; without it config-manager stopped at CreateContainerConfigError. Two tests: every ConfigMap key a rendered Cloud pod needs is rendered, and the prune leaves a CNPG-owned object alone |
What it does not include
- Any change to the platform charts, Lake or Cloud v1.
- Pixie Cloud has not yet started on a cluster: past the database, Kratos and Hydra, the bootstrap, the deploy keys and the
Viziers are still unproven (observability.md).
Upgrading
- No CRD changes. The CRD rollout (crd-rollout.md) runs as for any release and changes nothing.
- What rolls on each cluster with the operator: the operator, and the zetta-tunnel mesh pods, which carry a container from
the operator image. On the two hubs (the managing cluster and nonprod), the hub Prometheus is re-created with its new
securityContext. No Lake or Cloud v1 workload changes. - A
pixie-db-1pod started before the upgrade may still hold a token for a ServiceAccount the old prune deleted. If it keeps
failing withUnauthorizedonce CNPG has recreated the account, the pod must be replaced.
Known issues
- Unchanged from v0.11.5.
- Nonprod's
lakecontroller errors on workspace namespaces that no longer exist
(#1090).
Verification
| Check | Result |
|---|---|
Full suite (go test ./... with envtest) |
not run for this release, on Steve Sperandeo's instruction ("please skip the tests for this one and cut and deploy") |
| The PRs' own tests | go test ./cmd/ ./internal/controller/ passed on each PR's own branch (#1091 per its description, #1092 locally); each new test fails without its fix. The merged tree of both was not tested |
go test ./cmd/ -run TestReleaseVersionIsDocumented |
pass, on the release branch |
helm lint chart |
1 chart linted, 0 failed |
| Pull requests | #1091 and #1092 approved by Steve Sperandeo on 2026-10-31 ("release and deploy these") |
| Live | rolled to every cluster after the release; recorded in techops-docs |
Release v0.11.5
Released: 2026-10-31 · Branch: main · Previous: v0.11.4 (2026-10-30).
A patch release: Pixie's release is committed, so the observability hubs and agents can install. Until now every
ObservabilityHub and ObservabilityAgent stayed Pending on ObservabilityProvider pixie does not exist, which is why
ObservabilityAgentNotReady and ObservabilityHubUnreachable kept firing for orama-production-cluster-02 and -03.
| Measured | Value |
|---|---|
| Resource kinds | 54 CRDs in chart/crds (unchanged) |
| CRD changes | none |
| Operator code | none (one new test) |
| Lake schema | nothing vendored |
| Tests | see Verification |
What it includes
| PR | Change |
|---|---|
| #1085 | config/clusters/observability-provider.yaml: the ObservabilityProvider pixie, built from oramasearch/pixie at the vendored commit 4b445e0b7907 plus hack/pixie-patches/0001–0004: Vizier 0.14.16-pre-orama.1, Cloud 0.1.10-pre-orama.1, CLI 0.8.9-pre-orama.1, and 27 images, each a private oramasearch/pixie-* repository pinned by digest (the 8 dependency images are mirrors with their upstream's digest). The build patches 0002 (every image named pixie-*), 0003 (143 Go licences recorded, so the build does not depend on pkg.go.dev) and 0004 (the UI bundle's stamp values quoted for Bazel 7), and the build recipe that made the release. TestCommittedPixieReleaseIsInstallable fails when the operator's Pixie commit or image keys move away from the committed release. The sample drops cloud-plugin-load-db, a key the operator refuses |
(#455 tracked the build.)
What it does not include
- Any change to the operator's code, the hubs, the agents or the platform charts.
- Pixie has not run on a cluster before this release (observability.md).
Upgrading
- No CRD changes. The CRD rollout (crd-rollout.md) runs as for any release and changes nothing.
- The provider is applied to the managing cluster from
config/clusters/observability-provider.yaml. Once it exists,
every hub and every agent installs at once: there is no setting that holds production back while nonprod goes first. - What rolls on each cluster with the operator: the operator, and the zetta-tunnel mesh pods, which carry a container
from the operator image. No Lake or Cloud v1 workload changes.
Known issues
- Unchanged from v0.11.1.
- The managing cluster has a single node; the production hub requests about 1.8 CPU and 7 GiB of the about 2.3 CPU and 18 GiB
free when the provider was applied.
Verification
| Check | Result |
|---|---|
go build ./..., go vet ./... |
pass |
go test ./... with envtest (Kubernetes 1.30.3) |
3,193 passed, 0 failed, 22 skipped with subtests (1,216 top-level tests) |
| Skipped | the opt-in suites that need a local Postgres, a live zetta-tunnel, a live chart download, and a Prometheus binary |
helm lint chart |
1 chart linted, 0 failed |
| Pull request | reviewed by Steve Sperandeo on 2026-10-31; not re-tested in review, so this run is the first of the merged tree |
| Live | rolled to every cluster after the release, and the provider applied; recorded in techops-docs |
Release v0.11.4
Released: 2026-10-30 · Branch: main · Previous: v0.11.3 (2026-10-30).
A patch release: kube-prometheus-stack no longer raises KubeCPUQuotaOvercommit and KubeMemoryQuotaOvercommit on any
cluster. They fire when the namespaces' quotas add up to more than 1.5 times the nodes. Each Lake workspace gets a
workspace-defaults quota of 4 CPU and 8 GiB, a ceiling and not a reservation, so the quotas are meant to add up to more than
the cluster: on orama-nonprod-cluster-01, 19 workspaces come to 4.9 times its CPU and 2.9 times its memory.
| Measured | Value |
|---|---|
| Resource kinds | 54 CRDs in chart/crds (unchanged) |
| CRD changes | none |
| Operator code | none |
| Lake schema | nothing vendored |
| Tests | see Verification |
What it includes
| PR | Change |
|---|---|
| #1081 | Two more entries in the defaultRules.disabled list from v0.11.3, set the same way in helmfile.yaml and every leaf's config/clusters/*/platform.yaml, with the reason beside them and in observability.md. A workspace that reaches its own quota still alerts (KubeQuotaExceeded, warning; KubeQuotaFullyUsed and KubeQuotaAlmostFull, info), and a pod that cannot be placed pages as KubePodNotReady after 15 minutes |
What it does not include
- Any change to quotas, node pools, requests or other rules.
- Pixie itself (#455).
Upgrading
- No CRD changes. The CRD rollout (crd-rollout.md) runs as for any release and changes nothing.
- The platform values: the leaves take them when their
ClusterHelmReleases are applied from
config/clusters/*/platform.yaml; the managing cluster withhelmfile sync -l name=kube-prometheus-stack. Only the
PrometheusRuleholding the rules changes; Prometheus reloads its rules without restarting. - What rolls on each cluster with the operator: the operator, and the zetta-tunnel mesh pods, which carry a container
from the operator image. No Lake or Cloud v1 workload changes.
Known issues
- Unchanged from v0.11.1.
Verification
| Check | Result |
|---|---|
go build ./..., go vet ./... |
pass |
go test ./... with envtest (Kubernetes 1.30.3) |
3,192 passed, 0 failed, 22 skipped with subtests (1,215 top-level tests) |
| Skipped | the opt-in suites that need a local Postgres, a live zetta-tunnel, a live chart download, and a Prometheus binary |
helm lint chart |
1 chart linted, 0 failed |
| Pull request | reviewed by Steve Sperandeo on 2026-10-30; not re-tested in review, so this run is the first of the merged tree |
| Live | rolled to every cluster after the release; recorded in techops-docs |