Repository navigation
XMC Release 1.0.0
Release Version Determination: Major bump (
0.18.0->1.0.0).
Rationale: Marks the XMC 1.0.0 milestone — promoting the Components control plane to mandatory/always-on across all Go services, Helm charts, Cluster Toolkit v2 blueprints, and developer tooling, removing legacy non-componentenable_metrics/enable_componentsflags, scoping thestable-2026aprofile to its threeSTABLEcomponents (redpanda,clickhouse,ingestion-gateway), adding ESS schema migrations018and019(Experiment.experiment_typeand study configuration persistence), and introducing per-user resource allocation views plus server-side sorting and filtering in the Resource Observability Dashboard (/resources).
🎯 What XMC 1.0.0 Offers
XManager on Cloud (XMC) is an open-source control plane for orchestrating, observing, and managing machine learning experiments on Google Kubernetes Engine (GKE). Beyond its core experiment and JobSet orchestration, XMC 1.0.0 brings together a full suite of platform capabilities built since the original launch:
- Experiment & JobSet Orchestration: Manages the
Experiment->WorkUnit->WorkUnitExecution->JobSetlifecycle over gRPC and HTTP APIs, scheduling workloads through Kueue (including MultiKueue remote execution, pausing/resuming, and non-restarting execution guarantees) with durable state in Cloud Spanner (or experimental PostgreSQL) and Redis. - Resource Observability Dashboard (ROD,
/resources): Provides real-time visibility into Kueue cluster queue utilization, per-user dominant resource allocations, and live running/pending jobs with server-side sorting and filtering. - Dashboards (
Dashboard Service&xdash): Delivers interactive metric charts in the Web UI and a programmatic Python dashboard SDK (xdash), backed by ClickHouse and BigQuery data engines. - Admin Panel (
/admin) & Diagnostics: Surfaces live cluster health checks, database configuration and schema version status, component stability states, and automated Cloud Logging root-cause diagnostics via Log Monitor. - Pluggable Components (
stable-2026a): Runs an always-on Components control plane (xmc-bridge+ComponentCRs) with stability contracts and aSTABLEtelemetry pipeline (Redpanda, ClickHouse, and Ingestion Gateway). - Clients, Auth & Deployment: Includes the Angular Web UI, XManager CLI (
xmanagerwith interactivexmanager loginand service-account auth), Python SDKs (xmc,xdash), experiment ACL sharing, Cluster Toolkit v2 blueprints (GKE + Gateway API + OAuth2), and a KinD (dev_env) local/hybrid dev workflow.
🌟 1. Public Changes (Available Out-of-the-Box)
🚀 Major Highlights & Breaking Changes
- [Breaking / Promoted to Always-On] Mandatory Components Control Plane & Removal of Legacy Flags:
- Always-On Components Runtime: The Components control plane (
xmc-bridgeandComponentCRs) is now unconditionally enabled across all Go services, the umbrella Helm chart (deploy/helm), Cluster Toolkit v2 (deploy/ctk_v2), KinD local environments, andverify_xmc.py(#1631, #1632). - Removed Legacy Configuration Flags: Removed
enable_components,componentsDeployment.enabled,enable_metrics,metrics.enabled,ingestionGatewayDeployment.*, staticclickhouse_{host,port,database,user}connection variables (retainingclickhouse_password/metrics.clickhouse.{secretName,password}for credentials), and thedev_env--enable-components/--enable-metricsCLI flags (#1632, #1637). Umbrella Helm upgrades fail closed ifcomponentsDeployment.enabled=false,ingestionGatewayDeployment.enabled=true, or an unmigrated umbrella-ownedDeployment/ingestion-gateway-deploymentremains. - [Action Required for Upgrades from
0.18.xor Legacy Clusters] — followdocs/migrations/components.md(helm >= 3.14.0):- Legacy umbrella
ingestion-gatewayclusters (enable_components: false): Runtools/migrations/components/migrate.py(prepare->handoff) before upgrading the umbrella release to1.0.0(migrate.pysupports direct cutovers to1.x), and verify the runbook's pre-upgrade checks (secrets.csi.clickhousePasswordSecretNamenow defaults to"", clearingctk-out/<deployment_name>/.ghpc/staged, and settingxmc_bridge_image/xmc_bridge_custom_imagefor custom registries/tags) (#1632, #1633). 0.18.xComponents clusters (xmc-profile):stable-2026a(minXmcVersion: "1.0.0",allowExperimental: false) now ships exclusively the threeSTABLEcomponents (redpanda,clickhouse,ingestion-gateway), removing experimentalhpoandvizier(#1687). Before upgradingxmc-profiledirectly viahelm(dev_env.pyautomates this), followAfter Migration: Upgrading XMCto unprotect disabledhpo/vizierCRs, pass--reset-valuesif stored values recordedhpo/vizier, and ensurecomponents.clickhouseis enabled if relying on ClickHouse (#1633, #1687).
- Legacy umbrella
- Aligned 1.0.0 Documentation: Synchronized
README.md,INSTALL.md,docs/architecture/,docs/migrations/components.md,deploy/ctk_v2/README.md, and agent skills for1.0.0(#1633, #1638).
- Always-On Components Runtime: The Components control plane (
- [Action Required] ESS Database Schema Migrations
018&019(Experiment.experiment_type& Study Config Support): AddedExperiment.experiment_type(STANDARD,STUDY) via migration018for Spanner and PostgreSQL (#1608), along with component requirement validation/pinning (#1609),STUDYconfig persistence via migration019(#1617), pre-XID study validation (#1627), and trial budget clamping (#1618). Ensure schema migrations (tov19) are applied during upgrade (automatic viagcluster deploy/dev_env.py gke-upgrade).
✨ New Features & Improvements
- Resource Observability Dashboard (
/resources) Enhancements:- Per-User Dominant Resource Allocation Chips & Author Filtering: Added
user_allocationstoGetRunningAndPendingJobsand the Spanner/PostgreSQL storage snapshot (#1595, #1597, #1598), rendered per-user allocation chips in/resources(#1599), and made chips clickable to filter queue jobs by author (#1647). - Server-Side Queue Job Sorting & Filtering: Moved queue job sorting (#1645) and filtering (#1646) to the backend before pagination so results remain consistent across pages.
- Per-User Dominant Resource Allocation Chips & Author Filtering: Added
🐛 User-Facing Bug Fixes & Stability
- Actuator GCS Volume CSI Driver Fix: Updated Actuator to mount GCS volumes using the
gcsfuse.csi.storage.gke.ioCSI driver (#1586) (existing clusters using GCS volumes ormanaged_xprof_gcs_bucketmust also apply the updated CTKcomputegroup or enable the GKE Cloud Storage FUSE CSI add-on). - Helm
allowed_emailsComma Escaping: Escaped commas in Cluster Toolkit's Helm--setvalue forallowed_emailsso multi-user allowlists deploy cleanly (#1610). - Dependency Security Updates: Resolved reported Dependabot vulnerabilities across Go, npm, and Python dependencies (#1651).
🚀 2. Production-Ready Features Behind Feature Flags (Safe to Enable)
The following capabilities are fully developed, verified, and safe to be enabled. Their default enablement is temporarily postponed solely to align with managed solution milestones or external BYOD prerequisites.
🔹 [Flag: dashboard_service_deployment_enabled] Dashboard Service
- How to enable: Set
dashboard_service_deployment_enabled: trueindeploy/ctk_v2/xmc-deployment-template.yaml(ordashboardServiceDeployment.enabled: trueindeploy/helm/values.yaml). (Requires BYOD storage configuration and thestable-2026ametrics components). - What's New:
- BigQuery Component RBAC & Kubeconfig-Free Fallback: Granted RBAC permissions for BigQuery component discovery in Helm and added a graceful fallback when kubeconfig is absent (#1630).
- Shared
pkg/metricsqueryExtraction: Extracted Dashboard Service'sDataEnginequery backends into the reusablepkg/metricsquerypackage (#1616).
🧪 3. In-Development / Experimental Features Behind Flags
Warning
The following capabilities are under active ongoing development, experimental, and may not yet be functional. They are available for early preview and evaluation by enabling their deployment flags in deploy/ctk_v2/xmc-deployment-template.yaml (or deploy/ctk_v2/xmanager-cloud.yaml) and Helm values.yaml.
(No new experimental feature additions in this release; existing experimental flags database.experimentalPostgres and enable_managed_mldiagnostics remain available as documented in INSTALL.md.)
⚙️ 4. Operational, Infrastructure & Engineering Velocity
Changes relevant to cluster operators, CI/CD, and internal architecture that do not alter the end-user interface.
- Go Static Analysis (
nogo), Complexity Linters & Code Decomposition: Wired Bazelnogoanalyzers and enabledx/toolsanalyzers, fast Go hygiene/boundary/revive linters,funlen, andgocognitchecks (#1653, #1654, #1660, #1661), alongside refactoring and decomposingfilter,querytools, Spanner row decoding, and ESS handlers (#1655, #1656, #1657, #1658, #1659). - CI Runners, Nightly UI Contracts & Release/Review Tooling: Migrated GitHub Actions workflows from
arc-runner-settoubuntu-latest(#1680), migrated nightly UI verifier selectors todata-testidwith a selector contract lint (#1626, #1643), restored nightly matrix work-unit wiring andWorkUnitStateresolution (#1624, #1652), restricted release base tag discovery to semver tags (#1620), and improvedtools/pr_review/,tools/benchmark/,ABSENCE.mddate ranges, and agent rules (#1612, #1621, #1622, #1623, #1639, #1641).