Skip to content

Release 1.0.0

Latest

Choose a tag to compare

@janzak-dev janzak-dev released this 13 Nov 14:52
a223a6a

XMC Release 1.0.0

Release Version Determination: Major bump (0.18.0 -> 1.0.0).
Rationale: Marks the XMC 1.0.0 milestone — promoting the Components control plane to mandatory/always-on across all Go services, Helm charts, Cluster Toolkit v2 blueprints, and developer tooling, removing legacy non-component enable_metrics / enable_components flags, scoping the stable-2026a profile to its three STABLE components (redpanda, clickhouse, ingestion-gateway), adding ESS schema migrations 018 and 019 (Experiment.experiment_type and study configuration persistence), and introducing per-user resource allocation views plus server-side sorting and filtering in the Resource Observability Dashboard (/resources).


🎯 What XMC 1.0.0 Offers

XManager on Cloud (XMC) is an open-source control plane for orchestrating, observing, and managing machine learning experiments on Google Kubernetes Engine (GKE). Beyond its core experiment and JobSet orchestration, XMC 1.0.0 brings together a full suite of platform capabilities built since the original launch:

  • Experiment & JobSet Orchestration: Manages the Experiment -> WorkUnit -> WorkUnitExecution -> JobSet lifecycle over gRPC and HTTP APIs, scheduling workloads through Kueue (including MultiKueue remote execution, pausing/resuming, and non-restarting execution guarantees) with durable state in Cloud Spanner (or experimental PostgreSQL) and Redis.
  • Resource Observability Dashboard (ROD, /resources): Provides real-time visibility into Kueue cluster queue utilization, per-user dominant resource allocations, and live running/pending jobs with server-side sorting and filtering.
  • Dashboards (Dashboard Service & xdash): Delivers interactive metric charts in the Web UI and a programmatic Python dashboard SDK (xdash), backed by ClickHouse and BigQuery data engines.
  • Admin Panel (/admin) & Diagnostics: Surfaces live cluster health checks, database configuration and schema version status, component stability states, and automated Cloud Logging root-cause diagnostics via Log Monitor.
  • Pluggable Components (stable-2026a): Runs an always-on Components control plane (xmc-bridge + Component CRs) with stability contracts and a STABLE telemetry pipeline (Redpanda, ClickHouse, and Ingestion Gateway).
  • Clients, Auth & Deployment: Includes the Angular Web UI, XManager CLI (xmanager with interactive xmanager login and service-account auth), Python SDKs (xmc, xdash), experiment ACL sharing, Cluster Toolkit v2 blueprints (GKE + Gateway API + OAuth2), and a KinD (dev_env) local/hybrid dev workflow.

🌟 1. Public Changes (Available Out-of-the-Box)

🚀 Major Highlights & Breaking Changes

  • [Breaking / Promoted to Always-On] Mandatory Components Control Plane & Removal of Legacy Flags:
    • Always-On Components Runtime: The Components control plane (xmc-bridge and Component CRs) is now unconditionally enabled across all Go services, the umbrella Helm chart (deploy/helm), Cluster Toolkit v2 (deploy/ctk_v2), KinD local environments, and verify_xmc.py (#1631, #1632).
    • Removed Legacy Configuration Flags: Removed enable_components, componentsDeployment.enabled, enable_metrics, metrics.enabled, ingestionGatewayDeployment.*, static clickhouse_{host,port,database,user} connection variables (retaining clickhouse_password / metrics.clickhouse.{secretName,password} for credentials), and the dev_env --enable-components / --enable-metrics CLI flags (#1632, #1637). Umbrella Helm upgrades fail closed if componentsDeployment.enabled=false, ingestionGatewayDeployment.enabled=true, or an unmigrated umbrella-owned Deployment/ingestion-gateway-deployment remains.
    • [Action Required for Upgrades from 0.18.x or Legacy Clusters] — follow docs/migrations/components.md (helm >= 3.14.0):
      • Legacy umbrella ingestion-gateway clusters (enable_components: false): Run tools/migrations/components/migrate.py (prepare -> handoff) before upgrading the umbrella release to 1.0.0 (migrate.py supports direct cutovers to 1.x), and verify the runbook's pre-upgrade checks (secrets.csi.clickhousePasswordSecretName now defaults to "", clearing ctk-out/<deployment_name>/.ghpc/staged, and setting xmc_bridge_image / xmc_bridge_custom_image for custom registries/tags) (#1632, #1633).
      • 0.18.x Components clusters (xmc-profile): stable-2026a (minXmcVersion: "1.0.0", allowExperimental: false) now ships exclusively the three STABLE components (redpanda, clickhouse, ingestion-gateway), removing experimental hpo and vizier (#1687). Before upgrading xmc-profile directly via helm (dev_env.py automates this), follow After Migration: Upgrading XMC to unprotect disabled hpo/vizier CRs, pass --reset-values if stored values recorded hpo/vizier, and ensure components.clickhouse is enabled if relying on ClickHouse (#1633, #1687).
    • Aligned 1.0.0 Documentation: Synchronized README.md, INSTALL.md, docs/architecture/, docs/migrations/components.md, deploy/ctk_v2/README.md, and agent skills for 1.0.0 (#1633, #1638).
  • [Action Required] ESS Database Schema Migrations 018 & 019 (Experiment.experiment_type & Study Config Support): Added Experiment.experiment_type (STANDARD, STUDY) via migration 018 for Spanner and PostgreSQL (#1608), along with component requirement validation/pinning (#1609), STUDY config persistence via migration 019 (#1617), pre-XID study validation (#1627), and trial budget clamping (#1618). Ensure schema migrations (to v19) are applied during upgrade (automatic via gcluster deploy / dev_env.py gke-upgrade).

✨ New Features & Improvements

  • Resource Observability Dashboard (/resources) Enhancements:
    • Per-User Dominant Resource Allocation Chips & Author Filtering: Added user_allocations to GetRunningAndPendingJobs and the Spanner/PostgreSQL storage snapshot (#1595, #1597, #1598), rendered per-user allocation chips in /resources (#1599), and made chips clickable to filter queue jobs by author (#1647).
    • Server-Side Queue Job Sorting & Filtering: Moved queue job sorting (#1645) and filtering (#1646) to the backend before pagination so results remain consistent across pages.

🐛 User-Facing Bug Fixes & Stability

  • Actuator GCS Volume CSI Driver Fix: Updated Actuator to mount GCS volumes using the gcsfuse.csi.storage.gke.io CSI driver (#1586) (existing clusters using GCS volumes or managed_xprof_gcs_bucket must also apply the updated CTK compute group or enable the GKE Cloud Storage FUSE CSI add-on).
  • Helm allowed_emails Comma Escaping: Escaped commas in Cluster Toolkit's Helm --set value for allowed_emails so multi-user allowlists deploy cleanly (#1610).
  • Dependency Security Updates: Resolved reported Dependabot vulnerabilities across Go, npm, and Python dependencies (#1651).

🚀 2. Production-Ready Features Behind Feature Flags (Safe to Enable)

The following capabilities are fully developed, verified, and safe to be enabled. Their default enablement is temporarily postponed solely to align with managed solution milestones or external BYOD prerequisites.

🔹 [Flag: dashboard_service_deployment_enabled] Dashboard Service

  • How to enable: Set dashboard_service_deployment_enabled: true in deploy/ctk_v2/xmc-deployment-template.yaml (or dashboardServiceDeployment.enabled: true in deploy/helm/values.yaml). (Requires BYOD storage configuration and the stable-2026a metrics components).
  • What's New:
    • BigQuery Component RBAC & Kubeconfig-Free Fallback: Granted RBAC permissions for BigQuery component discovery in Helm and added a graceful fallback when kubeconfig is absent (#1630).
    • Shared pkg/metricsquery Extraction: Extracted Dashboard Service's DataEngine query backends into the reusable pkg/metricsquery package (#1616).

🧪 3. In-Development / Experimental Features Behind Flags

Warning

The following capabilities are under active ongoing development, experimental, and may not yet be functional. They are available for early preview and evaluation by enabling their deployment flags in deploy/ctk_v2/xmc-deployment-template.yaml (or deploy/ctk_v2/xmanager-cloud.yaml) and Helm values.yaml.

(No new experimental feature additions in this release; existing experimental flags database.experimentalPostgres and enable_managed_mldiagnostics remain available as documented in INSTALL.md.)


⚙️ 4. Operational, Infrastructure & Engineering Velocity

Changes relevant to cluster operators, CI/CD, and internal architecture that do not alter the end-user interface.

  • Go Static Analysis (nogo), Complexity Linters & Code Decomposition: Wired Bazel nogo analyzers and enabled x/tools analyzers, fast Go hygiene/boundary/revive linters, funlen, and gocognit checks (#1653, #1654, #1660, #1661), alongside refactoring and decomposing filter, querytools, Spanner row decoding, and ESS handlers (#1655, #1656, #1657, #1658, #1659).
  • CI Runners, Nightly UI Contracts & Release/Review Tooling: Migrated GitHub Actions workflows from arc-runner-set to ubuntu-latest (#1680), migrated nightly UI verifier selectors to data-testid with a selector contract lint (#1626, #1643), restored nightly matrix work-unit wiring and WorkUnitState resolution (#1624, #1652), restricted release base tag discovery to semver tags (#1620), and improved tools/pr_review/, tools/benchmark/, ABSENCE.md date ranges, and agent rules (#1612, #1621, #1622, #1623, #1639, #1641).