Skip to content

Releases

Release v0.12.0

Choose a tag to compare

@homer6 homer6 released this 16 Nov 08:45
075b6d1

Released: 2026-11-16 · Branch: main · Previous: v0.11.12 (2026-10-31) · Tracking: #1206

A minor release that gets the operator ready to publish as open source. It adds the license and contributor documents and a public README, and it removes Pixie and the observability kinds built on it. It also carries security fixes for tenant isolation, datastore credentials and Zetta's secrets, moves Orama's own cluster resources out of the repository, and pins zetta-tunnel v0.2.0. It was cut without a planned release epic, as part of the open-source release (#1148).

Measured Value
Resource kinds 49 CRDs in chart/crds (54 in v0.11.12; five removed)
Pull requests 32 merged since v0.11.12
CRD changes removed: ObservabilityProvider, ObservabilityHub, ObservabilityAgent, PixieCloud, PixieVizier. Added fields: OramaZettaCluster spec.egressAllowConfigMap and the identity-root rotation; Lake fields for the tenant-isolation and service changes; OramaCloudV1 spec.components.controlDB.referenceRows and spec.components.pgAdmin.email
License MIT, with COPYRIGHT, SPDX headers on every source file, THIRD_PARTY_LICENSES and an SBOM
zetta-tunnel v0.2.0 (main-10acaa2, by digest), from v0.1.0
Tests see Verification

What it includes

Open-source release

  • License and notices (#1194): MIT LICENSE and COPYRIGHT (OramaSearch, Inc.), SPDX-License-Identifier: MIT on every source file (tools/spdx-headers.py), THIRD_PARTY_LICENSES from go-licenses, and sbom.spdx.json.
  • Contributor documents: SECURITY.md, a private reporting address and response times (#1161); CONTRIBUTING.md, CODE_OF_CONDUCT.md and CLA.md (#1190).
  • A public README (#1202) with a quickstart that was run on a throwaway cluster. make run now passes --control-plane-bind-address="", so it starts without a control-plane certificate.
  • No internal material in the tree: comments, CRD descriptions, samples and chart values (#1193), test fixtures (#1195, #1188), and the public docs (#1196). The design log, the security reports and the research notes moved out of the repository (#1190).
  • Cluster resources are an example (#1149): config/clusters/ holds a cleansed example deployment. The tests read it, or another directory named by ORAMA_CLUSTERS_DIR.
  • Cloud v1 configuration instead of embedded data. A stack's plan catalog is spec.components.controlDB.referenceRows. The built-in rows are a made-up example that refuses to run on a database that already has a catalog (#1185). pgAdmin's sign-in is spec.components.pgAdmin.email, admin@example.com by default (#1189).

Pixie removed (#1199)

ObservabilityProvider, ObservabilityHub, ObservabilityAgent, PixieCloud and PixieVizier are removed, with their controllers, CRDs, RBAC, the vendored Pixie manifests and the Pixie build material. The hub Prometheus and its federation, and the per-cluster alert routing, go with them; nothing replaces them in this release. The platform layer's Prometheus, Alertmanager and Perses stay, with each Cloud v1 stack's rules and dashboards. The manager also loses its alertmanagerconfigs and prometheuses grants. make check-crds now fails while an object of a removed kind remains (hack/retired-crds.txt).

Security

  • A person's service launch gets its own token, never the Lake's agent service token (#1150).
  • An explicit CIDR egress rule reaches public addresses only (#1153), and a tenant-named peer remote must be public (#1154).
  • The default images that run tenant code are pinned by digest (#1155).
  • The operator verifies a workspace Postgres's certificate before signing in as superuser (#1156).
  • Image builders never mount the platform pull Secret (#1159).
  • An app's bindings copy only its own workspace's services (#1160).
  • Tenant services never run in the Lake namespace. Leftovers there are reported as a condition and a Warning event, never deleted (#1181).
  • Five findings in one batch (#1162):
    • TiDB's root password is set while TiDB bootstraps;
    • a lake secret's projection is deleted when lake says the secret is gone;
    • a row's workspace is checked against its customer;
    • methodology rows and agent pods are bounded on the operator's side;
    • mesh projections carry the pod UID and namespace inode.
  • TiDB and Cassandra tenants get a user of their own, never the administrator (#1171). Each service's administrator password is then rotated once (#1173).
  • Zetta: a connector-credential root Secret mounted outside the data volume, and the egress allowlist as a file the engine re-reads (#1167). Identity-root rotation as a spec generation, and the enrollment CA (#1177). Each is switched on by what the image's own --help lists.

Datastores and Cloud v1

  • lake-duckdb v0.2.0 is the default DuckDB image (#1204). It fixes lake-duckdb's security findings, whose reproductions are public. lake-sqlite v0.2.0 is the default SQLite image (#1203). Both are pinned by digest.
  • Cloud v1:
    • the embeddings server requests the 4Gi it holds, limited to 5Gi (#1147);
    • a stack's analytics going stale raises OramaAnalyticsStale;
    • ClickHouse's own memory cap is three quarters of its pod's limit (#1141). The default renders the same file as before.

Other

  • zetta-tunnel v0.2.0 (#1138), which the mesh projection needs.
  • Seventeen tests the untested merges left failing, fixed on dev before the cut:
    • seven in #1205, one of them a real fix: spec.env can no longer override the two Zetta file variables the operator mounts;
    • ten in #1208: fixtures for #1162's workspace routing check and its revocation.

What it does not include

  • Any replacement for Pixie, the hub metrics, or alert delivery.
  • Public images. Every default image, and the operator's own, is in a private registry (#1186).
  • These open pull requests:
    • #1164 with #1170, #1175 and #1176: apps stop being served until each Lake has its own wildcard domain.
    • #1178: moves DNS for every Cloud v1 host.
    • #1163 and #1157: their lake halves are not merged.
    • #1180: waits on a zetta-tunnel change.
    • #1137: denies the cluster's own addresses in the egress gateway snapshot, not yet run against zetta-tunnel v0.2.0.
    • #857, #860 and #864: on hold for a planned rollout.
    • The backup system: #1182 to #1184.

Before it rolls

Each step needs the cluster owner's consent, in this order:

  1. Remove the observability resources under the operator that still ships them (v0.11.12). Delete the ObservabilityAgents, ObservabilityHubs and the ObservabilityProvider from their committed files as they were before this release, and wait until they are gone. The managing operator tears down what each one wrote on its clusters.
  2. Apply the Cloud v1 set files that carry the new fields before the operator rolls:
    • controlDB.referenceRows with each stack's current catalog. Otherwise each stack's reference-data Job fails its guard, and its catalog stays unchanged.
    • pgAdmin.email on the stacks that run pgAdmin. Otherwise pgAdmin signs in as admin@example.com from its next restart.
  3. make check-crds CONTEXT=<context> on every cluster. It fails while an object of a removed kind remains.
  4. Roll the operator, with the CRDs applied widened first and exactly after (docs/operations/crd-rollout.md). Then apply the platform layer by both installation paths: kube-state-metrics stops listing the Pixie kinds.
  5. lake must carry its half of #1150 (lake#1276) in the same roll. Until both run it, a person's service launch answers 401.
  6. Roll this operator before the Zetta images that read the connection-root file and the identity rotation (zetta#1509, zetta#1512, zetta#1514).

What changes on the roll:
...

Read more

Release v0.11.12

Choose a tag to compare

@homer6 homer6 released this 01 Nov 04:02
f74654d

Released: 2026-10-31 · Branch: main · Previous: v0.11.11 (2026-10-31).

A patch release: one Pixie hub shows every cluster, and Pixie keeps each user's settings. Production's hub on the managing
cluster now takes the managing cluster itself and nonprod, as well as orama-production-cluster-02 and -03 (Steve, 2026-10-31,
replacing #456's one hub per purpose). The welcome tour no longer comes back on every page load: the bootstrap now writes the
per-user rows Pixie's settings are saved in.

Measured Value
Resource kinds 54 CRDs in chart/crds (unchanged)
CRD changes ObservabilityAgent: spec.clusterRef optional (empty is the managing cluster), new optional spec.alerting
Operator code the Pixie bootstrap (per-user rows), the observability purpose rule, agent alert routing
Lake schema nothing vendored
Tests see Verification

What it includes

PR Change
#1124 Settings (#1121). The bootstrap gives each Pixie user its user_attributes and user_settings rows, which Pixie's own user creation writes and its settings updates need; existing users get them on the next reconcile. The bootstrap role gains SELECT, INSERT on the two tables. The managing cluster's agent (#1122). An ObservabilityAgent with no clusterRef runs on the managing cluster, beside production's hub without a link, registered under the hub's name. Nonprod on production's hub (#1122). The managing cluster's hub accepts nonprod agents; production still never reports to a nonprod hub. An agent may name its own alert receivers (spec.alerting); a nonprod agent on production's hub pages only through those, so production's receiver Secrets never reach nonprod. config/clusters: the managing cluster's agent in observability-hub.yaml; nonprod's agent reports to production's hub and keeps #monitoring-stg; nonprod's own hub is retired from the file

What it does not include

  • Any change to the platform charts, Lake or Cloud v1.
  • Applying the new agent files or deleting nonprod's hub: each needs Steve's consent after the roll.
  • The fix for the hub Prometheus not reaching the production leaves
    (#1111).

Upgrading

  • Apply the CRDs first: ObservabilityAgent drops clusterRef from required and gains alerting. Additive; existing agents
    are unchanged.
  • No pod restarts from the operator code itself; the hub's bootstrap writes two rows per user on its next pass.
  • After the roll, with consent: apply config/clusters/observability-hub.yaml (the managing cluster's agent: Pixie's node agents
    on the managing cluster) and config/clusters/orama-nonprod-cluster-01/observability.yaml (nonprod's agent moves to production's
    hub: its Vizier re-registers there and its in-memory trace history restarts), then delete ObservabilityHub/nonprod on the
    managing cluster, which removes nonprod's Pixie Cloud and its databases (sign-ins only).
  • What one hub costs: nonprod gets a PeerLink to production's hub like each production cluster: vzconn only, its own deploy
    key, the hub's CA; the hub reads back its read-only metrics proxy. Nonprod's "cluster lost" alerts come from the hub's rules and go
    to #monitoring.

Known issues

  • #1111: ObservabilityClusterUnreachable fires falsely for the
    production leaves.
  • #1104: kuberesolver 401 log noise on the hub's API server.
  • Pixie shows clusters as degraded ("non-operator version of Pixie"): cosmetic, because this operator installs Vizier instead of
    Pixie's own operator.
  • Licensing: Pixie's eBPF programs are GPL-2.0 (orama-bridge#268).

Verification

Check Result
go build ./..., go vet ./... pass
go test ./... with envtest (Kubernetes 1.30.3) not run for this release, on Steve Sperandeo's instruction ("we can skip the test this time"); stopped while running. On #1124's branch, go test ./... -count=1 passed (7 packages, no failures; run without envtest assets, so envtest suites may have skipped)
go test ./cmd/ passes on the release branch, TestReleaseVersionIsDocumented included
helm lint chart 1 chart(s) linted, 0 chart(s) failed
Pull request #1124 approved by Steve Sperandeo on 2026-10-31
Live rolled after the release; recorded in techops-docs

Release v0.11.11

Choose a tag to compare

@homer6 homer6 released this 31 Oct 23:15
4df5988

Released: 2026-10-31 · Branch: main · Previous: v0.11.10 (2026-10-31).

A patch release: the Pixie UI can run scripts. Signed in and with scripts loaded, running px/cluster failed with
"Server error, http status code: 0". Envoy serves gRPC-web on the proxy's port 4444, and its CORS allowed only upstream's
dev.withpixie.dev, so the browser's preflight for ExecuteScript got 403. The release also makes a configuration change roll
the pods that read it
: v0.11.10's script bundle URL reached the API server but not the proxy, which kept serving the old one.

Measured Value
Resource kinds 54 CRDs in chart/crds (unchanged)
CRD changes none
Operator code the PixieCloud rendering (proxy-envoy-config), and a ConfigMap fingerprint on every Pixie pod template (Cloud and Vizier)
Lake schema nothing vendored
Tests see Verification

What it includes

PR Change
#1117 Envoy's CORS origin becomes the hub's domain (suffix, so work.<domain> is covered); rendering fails if the vendored origin ever disappears. orama.cloud/pixie-config on every Pixie pod template is a hash of the rendered ConfigMaps it reads, so a change rolls exactly those pods. TestPixieCloudRendersNoUpstreamDomain (no rendered object names withpixie.dev, withpixie.ai, px.dev or gcr.io/pixie-oss) and TestPixieConfigChangeRollsThePodsThatReadIt

What it does not include

  • Any change to the platform charts, Lake or Cloud v1.
  • The fix for the hub Prometheus not reaching the production leaves
    (#1111).

Upgrading

  • No CRD changes.
  • Every Pixie pod restarts once: the hub on the managing cluster and on nonprod, and the agent on all three leaves, because each
    pod template gains the fingerprint. Pixie keeps its traces in memory, so each node's recent trace data is lost. No Lake or Cloud v1
    workload changes.
  • People reaching the UI forward the proxy's port 4444 too (script runs), as well as 443.

Known issues

  • #1111: ObservabilityClusterUnreachable fires falsely for the
    production leaves.
  • #1104: kuberesolver 401 log noise.
  • Licensing: Pixie's eBPF programs are GPL-2.0 (orama-bridge#268).

Verification

Check Result
go build ./..., go vet ./... pass
go test ./... with envtest (Kubernetes 1.30.3) 3,198 passed, 0 failed, 22 skipped with subtests (1,221 top-level tests). Skipped: the opt-in suites that need a local Postgres, a live zetta-tunnel, a live chart download, and a Prometheus binary
helm lint chart 1 chart(s) linted, 0 chart(s) failed
Not run before the release a script run through a live hub; checked after the roll
Pull request #1117 approved by Steve Sperandeo on 2026-10-31
Live rolled after the release; recorded in techops-docs

Release v0.11.10

Choose a tag to compare

@homer6 homer6 released this 31 Oct 22:05
d5ac107

Released: 2026-10-31 · Branch: main · Previous: v0.11.9 (2026-10-31).

A patch release: the Pixie UI and the hub's API server load the PxL script bundle. Once people could sign in (v0.11.9), the UI
showed "Failed to Load Script Bundles (Network Error (SCRIPT_BUNDLES_OVERRIDDEN))":

  • The UI's fetch was cross-origin. It runs on work.<domain>, but it was given https://<domain>/pxl_scripts/bundle.json,
    another origin that sends no CORS headers, so the browser refused it.
  • The API server couldn't resolve the address. It was given the same URL, which nothing in the cluster resolves (#1112).
Measured Value
Resource kinds 54 CRDs in chart/crds (unchanged)
CRD changes none
Operator code the PixieCloud rendering: pl-script-bundles-config and the API server's environment
Lake schema nothing vendored
Tests see Verification

What it includes

PR Change
#1113 The UI gets SCRIPT_BUNDLE_URLS=["/pxl_scripts/bundle.json"], a path on its own origin that the proxy already serves. The API server gets its own SCRIPT_BUNDLE_URLS, the bundle server's Service (http://pixie-scripts.plc.svc:8080/pixie-scripts/bundle.json). A render test pins both. No Pixie image changes

What it does not include

  • Any change to the platform charts, Lake or Cloud v1.
  • The fix for the hub Prometheus not reaching the production leaves
    (#1111). Until then ObservabilityClusterUnreachable fires for
    -02 and -03 although both are healthy.

Upgrading

  • No CRD changes.
  • What rolls on each cluster: the operator and the zetta-tunnel mesh pods. On the hubs, the proxy and the API server roll
    with their new configuration. No Lake or Cloud v1 workload changes.
  • A browser that set px-custom-script-bundle-paths to work around the error can remove it:
    localStorage.removeItem('px-custom-script-bundle-paths').

Known issues

  • #1111: the hub cannot federate the production leaves, so
    ObservabilityClusterUnreachable fires for them.
  • #1104: kuberesolver 401 log noise.
  • Otherwise unchanged from v0.11.9.

Verification

Check Result
go build ./..., go vet ./... pass
go test ./... with envtest (Kubernetes 1.30.3) not run for this release, on Steve Sperandeo's instruction ("skip the test suite... deploy everywhere"); stopped while running. #1113's own tests (go test ./internal/controller/ -run Pixie, go test ./cmd/) passed on its branch
helm lint chart 1 chart(s) linted, 0 chart(s) failed
Not run before the release the UI listing scripts on a live hub; checked after the roll
Pull request #1113 approved by Steve Sperandeo on 2026-10-31
Live rolled after the release; recorded in techops-docs

Release v0.11.9

Choose a tag to compare

@homer6 homer6 released this 31 Oct 21:31
3a7cac6

Released: 2026-10-31 · Branch: main · Previous: v0.11.8 (2026-10-31).

A patch release: the hub Prometheus writes its configuration, so it federates every cluster and evaluates the hub's alert
rules. After v0.11.7 both hubs and all three agents were Ready, but the hub Prometheus had no scrape targets: prometheus-operator
v0.94 writes no configuration for a Prometheus that sets none of the monitor selectors, so additionalScrapeConfigs and the rule
selector were ignored (Reconciled=True ConfigurationUnmanaged), and every agent reported metricsConnected=false.

v0.11.8 was released but never rolled. This release carries it too: the clusters go from v0.11.7 straight to v0.11.9, with
both fixes.

Measured Value
Resource kinds 54 CRDs in chart/crds (unchanged from v0.11.8)
CRD changes none since v0.11.8; from v0.11.7, ObservabilityHub.spec.users and PixieCloud.spec.users
Operator code the hub Prometheus's selectors
Lake schema nothing vendored
Tests see Verification

What it includes

PR Change
#1102 The hub Prometheus sets a scrapeConfigSelector (and its namespace selector) for the hub's own ScrapeConfigs, of which there are none. It is there so prometheus-operator writes the configuration: the federation jobs and the hub's rules then load. TestHubPrometheusIsPinnedAndScoped fails without it
#1103, from v0.11.8 People can sign in to the Pixie hubs: the organisation's domain, and spec.users (v0.11.8)

What it does not include

  • Any change to the platform charts, Lake or Cloud v1.
  • Live traces through a PxL script are still unproven (observability.md).

Upgrading

  • From v0.11.7, follow v0.11.8's upgrade notes: its two optional CRD fields are widened by the CRD
    rollout (crd-rollout.md), and each hub's file is applied after the roll to add its users.
  • What rolls on each cluster with the operator: the operator, and the zetta-tunnel mesh pods, which carry a container from
    the operator image. No Lake or Cloud v1 workload changes. On the hubs, the hub Prometheus reloads its configuration.

Known issues

Verification

Check Result
go build ./..., go vet ./... pass
go test ./... with envtest (Kubernetes 1.30.3) 3,196 passed, 0 failed, 22 skipped with subtests (1,219 top-level tests). Skipped: the opt-in suites that need a local Postgres, a live zetta-tunnel, a live chart download, and a Prometheus binary
helm lint chart 1 chart(s) linted, 0 chart(s) failed
Not run before the release the hub Prometheus's targets on a live hub; checked after the roll
Pull requests #1102 and #1103 approved by Steve Sperandeo on 2026-10-31
Live rolled after the release; recorded in techops-docs

Release v0.11.8

Choose a tag to compare

@homer6 homer6 released this 31 Oct 21:19
2656a48

Released: 2026-10-31 · Branch: main · Previous: v0.11.7 (2026-10-31).

A patch release: people can sign in to the Pixie hubs. After v0.11.7 both hubs ran, but Pixie turned their
administrator away with "Our system found an issue with your account". Pixie's sign-in refuses a member of an
organisation that has no domain, unless the organisation is named after that member's email
(src/cloud/auth/controllers/login.go, loginSimple). The operator had created the organisation orama without a
domain.

Measured Value
Resource kinds 54 CRDs in chart/crds (unchanged)
CRD changes ObservabilityHub.spec.users and PixieCloud.spec.users, optional lists of emails
Operator code the PixieCloud bootstrap: the organisation's domain, and one sign-in per user
Lake schema nothing vendored
Tests see Verification

What it includes

PR Change
#1103 The organisation gets the administrator's email domain, and a hub bootstrapped without one has it set: the bootstrap role may update orgs.domain_name, and nothing else of an existing row. spec.users: each email gets a generated password in plc/pixie-user-<email>, an identity in Kratos and a user in the organisation; an email removed loses its identity, then its Secret. Both hubs' files list steve@orama.com. Docs: the bootstrap, the Secrets, the CRD reference, and why the UI must be reached on port 443

What it does not include

  • Any change to the platform charts, Lake or Cloud v1.
  • Live traces through a PxL script are still unproven (observability.md).

Upgrading

  • CRDs: two optional fields are added. The CRD rollout (crd-rollout.md) widens them before
    the operator upgrade, as for any release.
  • What rolls on each cluster with the operator: the operator, and the zetta-tunnel mesh pods, which carry a container from
    the operator image. No Lake or Cloud v1 workload changes.
  • On each hub, on the next pass after the roll: the organisation gets its domain, and sign-in works for the administrator.
  • Then apply each hub's file (config/clusters/observability-hub.yaml, config/clusters/orama-nonprod-cluster-01/observability.yaml)
    to add its users. The operator writes each user's Secret and sign-in within five minutes.

Known issues

  • Unchanged from v0.11.7.
  • #1104: the hub's auth-server, profile-server and api-server
    log a kuberesolver 401 from the Kubernetes API every second. It does not stop sign-in or registration.
  • The proxy redirects <domain> to https://work.<domain>/ without the port, so the UI must be reached on port 443 (techops-docs'
    observability runbook gives two ways).

Verification

Check Result
go build ./..., go vet ./... pass
go test ./... with envtest (Kubernetes 1.30.3) 3,195 passed, 1 failed, 22 skipped with subtests (1,218 top-level tests). The one failure was TestReleaseVersionIsDocumented: the suite started before this page existed. go test ./cmd/ rerun after it was written: pass. Skipped: the opt-in suites that need a local Postgres, a live zetta-tunnel, a live chart download, and a Prometheus binary
The PR's own tests TestPixieCloudBootstrapLetsTheAdministratorAndEveryUserSignIn (new), TestPixieCloudBootstrapIssuesAndRevokesPerClusterKeys and TestPixieBootstrapMatchesVendoredSchema (now covers orgs.domain_name)
helm lint chart 1 chart(s) linted, 0 chart(s) failed
Not run before the release the domain update and the Kratos delete against a live hub; checked after the roll by signing in
Pull request #1103 approved by Steve Sperandeo on 2026-10-31
Live rolled after the release; recorded in techops-docs

Release v0.11.7

Choose a tag to compare

@homer6 homer6 released this 31 Oct 18:48
2066016

Released: 2026-10-31 · Branch: main · Previous: v0.11.6 (2026-10-31).

A patch release: Pixie's metadata index is created as soon as OpenSearch is ready, so the indexer starts and the
observability hubs can become Ready. After v0.11.6 both hubs were stuck at 19/20: the indexer exits when its index
md_entities_9 is missing, the hub is not Ready without the indexer, and the operator created the index only once the hub was
Ready.

Measured Value
Resource kinds 54 CRDs in chart/crds (unchanged)
CRD changes none
Operator code the PixieCloud bootstrap's order
Lake schema nothing vendored
Tests see Verification

What it includes

PR Change
#1098 The bootstrap takes OpenSearch's readiness apart from the hub's: it creates the metadata index once pixie-search is Ready, and does the rest (the administrator, the organisation, the deploy keys) once the hub is Ready, as before. TestPixieCloudBootstrapIssuesAndRevokesPerClusterKeys now expects the index and no keys with only OpenSearch ready
#1072 Docs: how people reach Perses, Prometheus, Alertmanager and the Pixie UI (kubectl port-forward, nothing public), and Verify records that the platform charts run on all four clusters

What it does not include

  • Any change to the platform charts, Lake or Cloud v1.
  • The Viziers connecting to their hub and live traces are still unproven
    (observability.md).

Upgrading

  • No CRD changes. The CRD rollout (crd-rollout.md) runs as for any release and changes nothing.
  • What rolls on each cluster with the operator: the operator, and the zetta-tunnel mesh pods, which carry a container from
    the operator image. No Lake or Cloud v1 workload changes. On the hubs, the operator then creates the index and the indexer
    starts.

Known issues

  • Unchanged from v0.11.6.
  • #1096: a CloudflareR2Bucket already gone keeps its finalizer;
    v1-dev-uploads gets 403 on permission groups.

Verification

Check Result
Full suite (go test ./... with envtest) not run for this release, on Steve Sperandeo's instruction ("skip testing again for this one... let's get it deployed asap")
The PR's own tests go test ./cmd/ ./internal/controller/ passed on #1098's branch (per the PR); the changed test fails without the fix
go test ./cmd/ -run TestReleaseVersionIsDocumented pass, on the release branch
helm lint chart 1 chart(s) linted, 0 chart(s) failed
Pull requests #1098 and #1072 approved by Steve Sperandeo on 2026-10-31
Live rolled to every cluster after the release; recorded in techops-docs

Release v0.11.6

Choose a tag to compare

@homer6 homer6 released this 31 Oct 17:46
f73026b

Released: 2026-10-31 · Branch: main · Previous: v0.11.5 (2026-10-31).

A patch release: the faults that stopped both observability hubs after Pixie's release was applied. With v0.11.5 and the
provider applied, each hub's Prometheus crash-looped and Pixie Cloud never started. Three operator faults are fixed here; the
fourth, the Postgres image's missing tag, was fixed in the committed provider on dev (1e912c0f) and applied the same day.

Measured Value
Resource kinds 54 CRDs in chart/crds (unchanged)
CRD changes none
Operator code the hub Prometheus, the Pixie release check, the Pixie prune, Pixie Cloud's dependency ConfigMaps
Lake schema nothing vendored
Tests see Verification

What it includes

PR Change
#1091 The hub Prometheus owns its volume. It set no pod securityContext, so its Azure disk stayed root's and Prometheus (non-root) failed with permission denied on /prometheus. It now runs as uid 1000 with fsGroup 2000, like the platform's kube-prometheus-stack. A Pixie Postgres image must carry its tag: CloudNativePG reads the PostgreSQL version from the tag and refuses an image given by digest alone, so the provider is not Ready until postgres is repo:<version>@sha256:…. config/clusters/observability-provider.yaml already names pixie-postgres:16.15@sha256:34cd4159… (1e912c0f)
#1092 The Pixie prune spares objects another controller owns. CloudNativePG copies the pixie-db Cluster's inherited labels, orama.cloud/pixie: cloud included, onto the ServiceAccount, Services and budget it creates; the prune listed them by that label and deleted them on every pass, so the database pod's token stopped working (Unauthorized). It also tried to delete CNPG's Role and RoleBinding (refused). It now skips an object whose controller is not an orama.cloud kind. pl-sentry-dsn-config is rendered, its three keys empty (Sentry off), as upstream ships it in cloud_deps; without it config-manager stopped at CreateContainerConfigError. Two tests: every ConfigMap key a rendered Cloud pod needs is rendered, and the prune leaves a CNPG-owned object alone

What it does not include

  • Any change to the platform charts, Lake or Cloud v1.
  • Pixie Cloud has not yet started on a cluster: past the database, Kratos and Hydra, the bootstrap, the deploy keys and the
    Viziers are still unproven (observability.md).

Upgrading

  • No CRD changes. The CRD rollout (crd-rollout.md) runs as for any release and changes nothing.
  • What rolls on each cluster with the operator: the operator, and the zetta-tunnel mesh pods, which carry a container from
    the operator image. On the two hubs (the managing cluster and nonprod), the hub Prometheus is re-created with its new
    securityContext. No Lake or Cloud v1 workload changes.
  • A pixie-db-1 pod started before the upgrade may still hold a token for a ServiceAccount the old prune deleted. If it keeps
    failing with Unauthorized once CNPG has recreated the account, the pod must be replaced.

Known issues

  • Unchanged from v0.11.5.
  • Nonprod's lake controller errors on workspace namespaces that no longer exist
    (#1090).

Verification

Check Result
Full suite (go test ./... with envtest) not run for this release, on Steve Sperandeo's instruction ("please skip the tests for this one and cut and deploy")
The PRs' own tests go test ./cmd/ ./internal/controller/ passed on each PR's own branch (#1091 per its description, #1092 locally); each new test fails without its fix. The merged tree of both was not tested
go test ./cmd/ -run TestReleaseVersionIsDocumented pass, on the release branch
helm lint chart 1 chart linted, 0 failed
Pull requests #1091 and #1092 approved by Steve Sperandeo on 2026-10-31 ("release and deploy these")
Live rolled to every cluster after the release; recorded in techops-docs

Release v0.11.5

Choose a tag to compare

@homer6 homer6 released this 31 Oct 15:22
b3d0351

Released: 2026-10-31 · Branch: main · Previous: v0.11.4 (2026-10-30).

A patch release: Pixie's release is committed, so the observability hubs and agents can install. Until now every
ObservabilityHub and ObservabilityAgent stayed Pending on ObservabilityProvider pixie does not exist, which is why
ObservabilityAgentNotReady and ObservabilityHubUnreachable kept firing for orama-production-cluster-02 and -03.

Measured Value
Resource kinds 54 CRDs in chart/crds (unchanged)
CRD changes none
Operator code none (one new test)
Lake schema nothing vendored
Tests see Verification

What it includes

PR Change
#1085 config/clusters/observability-provider.yaml: the ObservabilityProvider pixie, built from oramasearch/pixie at the vendored commit 4b445e0b7907 plus hack/pixie-patches/0001–0004: Vizier 0.14.16-pre-orama.1, Cloud 0.1.10-pre-orama.1, CLI 0.8.9-pre-orama.1, and 27 images, each a private oramasearch/pixie-* repository pinned by digest (the 8 dependency images are mirrors with their upstream's digest). The build patches 0002 (every image named pixie-*), 0003 (143 Go licences recorded, so the build does not depend on pkg.go.dev) and 0004 (the UI bundle's stamp values quoted for Bazel 7), and the build recipe that made the release. TestCommittedPixieReleaseIsInstallable fails when the operator's Pixie commit or image keys move away from the committed release. The sample drops cloud-plugin-load-db, a key the operator refuses

(#455 tracked the build.)

What it does not include

  • Any change to the operator's code, the hubs, the agents or the platform charts.
  • Pixie has not run on a cluster before this release (observability.md).

Upgrading

  • No CRD changes. The CRD rollout (crd-rollout.md) runs as for any release and changes nothing.
  • The provider is applied to the managing cluster from config/clusters/observability-provider.yaml. Once it exists,
    every hub and every agent installs at once: there is no setting that holds production back while nonprod goes first.
  • What rolls on each cluster with the operator: the operator, and the zetta-tunnel mesh pods, which carry a container
    from the operator image. No Lake or Cloud v1 workload changes.

Known issues

  • Unchanged from v0.11.1.
  • The managing cluster has a single node; the production hub requests about 1.8 CPU and 7 GiB of the about 2.3 CPU and 18 GiB
    free when the provider was applied.

Verification

Check Result
go build ./..., go vet ./... pass
go test ./... with envtest (Kubernetes 1.30.3) 3,193 passed, 0 failed, 22 skipped with subtests (1,216 top-level tests)
Skipped the opt-in suites that need a local Postgres, a live zetta-tunnel, a live chart download, and a Prometheus binary
helm lint chart 1 chart linted, 0 failed
Pull request reviewed by Steve Sperandeo on 2026-10-31; not re-tested in review, so this run is the first of the merged tree
Live rolled to every cluster after the release, and the provider applied; recorded in techops-docs

Release v0.11.4

Choose a tag to compare

@homer6 homer6 released this 31 Oct 04:17
a3c956c

Released: 2026-10-30 · Branch: main · Previous: v0.11.3 (2026-10-30).

A patch release: kube-prometheus-stack no longer raises KubeCPUQuotaOvercommit and KubeMemoryQuotaOvercommit on any
cluster. They fire when the namespaces' quotas add up to more than 1.5 times the nodes. Each Lake workspace gets a
workspace-defaults quota of 4 CPU and 8 GiB, a ceiling and not a reservation, so the quotas are meant to add up to more than
the cluster: on orama-nonprod-cluster-01, 19 workspaces come to 4.9 times its CPU and 2.9 times its memory.

Measured Value
Resource kinds 54 CRDs in chart/crds (unchanged)
CRD changes none
Operator code none
Lake schema nothing vendored
Tests see Verification

What it includes

PR Change
#1081 Two more entries in the defaultRules.disabled list from v0.11.3, set the same way in helmfile.yaml and every leaf's config/clusters/*/platform.yaml, with the reason beside them and in observability.md. A workspace that reaches its own quota still alerts (KubeQuotaExceeded, warning; KubeQuotaFullyUsed and KubeQuotaAlmostFull, info), and a pod that cannot be placed pages as KubePodNotReady after 15 minutes

What it does not include

  • Any change to quotas, node pools, requests or other rules.
  • Pixie itself (#455).

Upgrading

  • No CRD changes. The CRD rollout (crd-rollout.md) runs as for any release and changes nothing.
  • The platform values: the leaves take them when their ClusterHelmReleases are applied from
    config/clusters/*/platform.yaml; the managing cluster with helmfile sync -l name=kube-prometheus-stack. Only the
    PrometheusRule holding the rules changes; Prometheus reloads its rules without restarting.
  • What rolls on each cluster with the operator: the operator, and the zetta-tunnel mesh pods, which carry a container
    from the operator image. No Lake or Cloud v1 workload changes.

Known issues

Verification

Check Result
go build ./..., go vet ./... pass
go test ./... with envtest (Kubernetes 1.30.3) 3,192 passed, 0 failed, 22 skipped with subtests (1,215 top-level tests)
Skipped the opt-in suites that need a local Postgres, a live zetta-tunnel, a live chart download, and a Prometheus binary
helm lint chart 1 chart linted, 0 failed
Pull request reviewed by Steve Sperandeo on 2026-10-30; not re-tested in review, so this run is the first of the merged tree
Live rolled to every cluster after the release; recorded in techops-docs