Release 0.16.0

XMC Release 0.16.0

Release Version Determination: Minor bump (0.15.0 -> 0.16.0).
Rationale: Introduces interactive and headless per-user CLI OAuth2 login (xmanager login) backed by ESS schema migration 016_cli_login_codes, adds /v1/ Nginx proxying in the Web UI, hardens gRPC deployment admission and OAuth2 session verification, delivers dashboard ordering and xdash IAM fallback improvements, and advances behind-the-flag capabilities for Components (cluster ownership fencing via migration 015_component_ownership_fence, cold-standby takeover, per-CR UID isolation, and least-privilege RBAC) and PostgreSQL storage parity.

---

🌟 1. Public Changes (Available Out-of-the-Box)

🚀 Major Highlights & Database Migrations

• [Action Required / Database Migrations] Spanner & PostgreSQL ESS Schema Migrations (015 & 016): This release adds ESS schema migrations 015_component_ownership_fence (ComponentOwnershipFence table for cluster epoch fencing, #1234) and 016_cli_login_codes (CLILoginCodes table for one-time CLI OAuth2 login code exchange, #1314). When upgrading an existing deployment, ensure Spanner migrations are applied (executed automatically by default during dev_env.py gke-deploy and gke-upgrade).

✨ New Features & Improvements

• Interactive & Headless CLI OAuth2 Login (xmanager login): Added end-to-end per-user CLI OAuth2 authentication flows in the Experiment State Server (pkg/auth/oauth2), including sealed cli_login ticket cookies (#1348), loopback callback completion on /api/callback persisted via CLILoginCodes (016_cli_login_codes, #1314), one-time authorization code exchange for per-user XMC session tokens (#1315), and a headless manual confirmation page for remote/SSH sessions (#1329). Migrated all example workloads and CLI guides to xmanager login (#1324) and standardized xmanager CLI installation on pip across examples, docs, and nightly E2E (#1377).
  • Note: Requires xmanager >= 0.8.5.post1 on the client (pip install -U "xmanager>=0.8.5.post1" or re-run python3 deploy/cli/install_xmanager.py).
• Web UI /v1/ Reverse Proxy to Experiment State Server: Configured the Web UI Nginx server (cmd/ui/nginx.conf) to proxy /v1/ API requests directly to the Experiment State Server (#1369).

🐛 User-Facing Bug Fixes & Stability

• gRPC Deployment Admission, Session Token Hardening & Production Auth Guards: Enforced deployment-level admission checks across gRPC servers in ESS, XID Service, Dashboard Service, and Health Service (#1312), required verified email claims and normalized user identities at OAuth2 sign-in and session token parsing (#1311, #1313), and blocked disableAuthentication and mockAuth in Helm templates outside development environments (#1321).
• Pre-XID Creator ACL Check & Default Experiment Owner: Fixed CreateExperiment in ESS to validate the caller's creator ACL before provisioning an XID from the XID Service and to default the experiment owner to the authenticated creator when omitted (#1323).
• Long Token Wrapping in Web UI Status Messages: Fixed long unbroken tokens in experiment and work unit status messages overflowing table cells in the Web UI (#1373).

---

🚀 2. Production-Ready Features Behind Feature Flags (Safe to Enable)

The following capabilities are fully developed, verified, and safe to be enabled. Their default enablement is temporarily postponed solely to align with managed solution milestones or external BYOD prerequisites.

🔹 [Flag: enable_metrics & dashboard_service_deployment_enabled] Metrics Pipeline & Dashboard Service

• How to enable:
  • Metrics Ingestion: Set enable_metrics: true in deploy/ctk_v2/xmc-deployment-template.yaml (or metrics.enabled: true in deploy/helm/values.yaml).
  • Dashboard Service: Set dashboard_service_deployment_enabled: true in deploy/ctk_v2/xmc-deployment-template.yaml (or dashboardServiceDeployment.enabled: true in deploy/helm/values.yaml). (Note: Dashboard Service requires BYOD storage devices—see the deployment README—and also requires enable_metrics / metrics.enabled to be enabled for workload metrics data to be generated).
• What's New:
  • Newest-First Default Ordering in ListDashboards: Fixed ListDashboards in Spanner dashboard storage to order dashboards newest-first (create_time DESC) by default when no explicit order_by clause is provided (#1298).
  • Dashboards List Table Row Border Alignment & Screenshot Test: Fixed row border alignment in the Web UI Dashboards list page (/dashboards) and added golden Playwright screenshot verification (#1306).
  • xdash Python SDK IAM Token Fallback & Metrics Addon Setup Script: Added Google Cloud IAM identity token fallback to the xdash Python SDK client and added deploy/addons/metrics/setup_metrics_addons.sh to automate ClickHouse metrics addon setup (#1305).

---

🧪 3. In-Development / Experimental Features Behind Flags

Warning

The following capabilities are under active ongoing development, experimental, and may not yet be functional. They are available for early preview and evaluation by enabling their deployment flags in deploy/ctk_v2/xmc-deployment-template.yaml (or deploy/ctk_v2/xmanager-cloud.yaml) and Helm values.yaml.

🔹 [Flag: enable_components] Components

• How to enable: Set enable_components: true in deploy/ctk_v2/xmc-deployment-template.yaml (or componentsDeployment.enabled: true in deploy/helm/values.yaml).
• What's New:
  • Cluster Ownership Fence, Epoch Declaration & Cold-Standby Takeover: Added the ComponentOwnershipFence schema (015_component_ownership_fence) and DeclareClusterEpoch RPC in CMS (#1234), declared xmc-bridge cluster identity and epoch to CMS on startup (#1273), enforced cluster ownership fencing and source-object ownership across sourced intents, observations, deactivations, and CR deletion cleanup (#1233, #1235, #1238, #1240), held fenced bridges in cold standby (#1236), and added store-backed CMS ownership Prometheus gauges and DR takeover documentation (#1243, #1275).
  • Per-CR UID Isolation, Same-Cluster Recreated CR Adoption & Status Tracking: Isolated per-CR Helm actuation state by Kubernetes UID (#1239), enabled seamless adoption of recreated Component CRs on the same cluster via expected_owner_uid (#1274), recorded status.componentId and status.componentGeneration on Component CRs (#1237, #1241), reduced steady-state bridge log noise (#1244), and added disjunctive or dependency rules to component-linter (#1341).
  • Least-Privilege xmc-bridge RBAC & Modular Bridge/Store Subpackages: Scoped xmc-bridge RBAC to least-privilege workload and wrapper resources (removing componentsDeployment.rbac.clusterScope, #1242), and decomposed pkg/components/bridge (helm, statusdetector, healthprobe) and pkg/components/store (memory, spanner, ownership, dependency, lifecycle) into dedicated subpackages (#1362, #1363, #1364, #1365).

🔹 [Flag: enable_managed_mldiagnostics] Managed ML Diagnostics

• How to enable: Set enable_managed_mldiagnostics: true in deploy/ctk_v2/xmc-deployment-template.yaml.
• What's New:
  • Configurable MLRun Project, Location & Service Account Impersonation: Added reconciler configuration flags to override the target GCP project, location, and service account impersonation for MLRun synchronization and managed XProf sessions (#1370).

---

⚙️ 4. Operational, Infrastructure & Engineering Velocity

Changes relevant to cluster operators, CI/CD, and internal architecture that do not alter the end-user interface.

• PostgreSQL Storage Engine, Query Dialect & Schema Parity: Added the PostgreSQL connection pool, DBTX transaction runner, SQLSTATE error translator, and migration runner CLI (internal/storage/postgres, #1284, #1287, #1308), implemented the PostgreSQL pausestate repository with row locking (#1357), wired pgx pool and DBTX across all PostgreSQL repository packages (#1310, #1326, #1335, #1351), added PostgreSQL table views and dialect support to QueryGenerator and liststatement (#1301, #1336, #1338, #1339), and added a PostgreSQL testcontainer harness and Spanner-vs-PostgreSQL schema parity checks (#1291, #1356, #1371, #1372).
• KinD Profile Migration, 3P ClickHouse & WorkUnit Lifecycle E2E Suites: Added KinD E2E test suites verifying 1P gateway rolling upgrades, 3P ClickHouse continuity, and snapshot drift across profile migrations (#1383, #1384, #1389), expanded E2E coverage for WorkUnit lifecycle, batch mutations, and live priority updates (#1292), and hardened the Nightly ROD UI E2E paginator selector against aria-disabled buttons (#1295).
• UI Codebase Flattening, Backend Cleanups & Dependency Security: Flattened legacy utils/ directories and _helper files across the Angular Web UI (#1392), extracted shared WorkUnit status message and history helpers (#1255), dropped the unused Spanner client from the Actuator K8s client (#1382), remediated npm Dependabot alerts (@angular/router, follow-redirects, vite, launch-editor, #1340, #1347, #1391), and refined CI incremental approval policies and engineering documentation (#1299, #1300, #1304).