Release 0.14.0

XMC Release 0.14.0

Release Version Determination: Minor bump (0.13.0 -> 0.14.0).
Rationale: Adds out-of-the-box support for NVIDIA H100 Mega 80GB (a3-megagpu-8g) GPU accelerators on GKE and an XMC Lite batch job example for existing GKE clusters, resolves JobSet finalizer patching and reconciler error reporting bugs, brings dynamic ClickHouse Component CR resolution and hot-reloading DataEngine to the production-ready Metrics & Dashboard Service pipeline, and advances behind-the-flag capabilities across Components (stable-2026a meta-chart, drift self-healing), Pause Scheduling, and GKE-hosted Log Monitor deployments. (Note: No .sql or .ddl database schema migrations were introduced in this release; migrations/ changes were internal Go compatibility refactors).

---

🌟 1. Public Changes (Available Out-of-the-Box)

✨ New Features & Improvements

• NVIDIA H100 Mega 80GB (a3-megagpu-8g) GPU Accelerator Support: Researchers and cluster users can now schedule workloads targeting GKE a3-megagpu-8g node pools using H100_MEGA (nvidia-h100-mega-80gb). The Actuator automatically injects cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb into the JobSet PodSpec nodeSelector, and Kueue and Resource Overview Dashboard (ROD) resource classifiers recognize and track the accelerator (#1115).
• XMC Lite Batch Job Example on Existing GKE Clusters: Added examples/xmc_lite_job/, providing a ready-to-run workflow for launching batch experiments directly onto an existing GKE cluster via xm_local.create_experiment and xm_local.Kubernetes—complete with local SQLite experiment state inspection (inspect_sqlite_state.py)—without deploying the full XMC control plane (#1047).

🐛 User-Facing Bug Fixes & Stability

• Atomic JobSet Finalizer Removal & Reconciler Error Visibility: Replaced full-list finalizer merge patches in the JobSet reconciler with an atomic RFC 6902 JSON Patch (test + remove) so concurrent finalizer updates on terminating workloads are never overwritten, enabled immediate reconciliation when malformed XMC labels are corrected on update, and surfaced non-retryable label/conversion failures as reconcile.TerminalError in controller-runtime metrics (#1025).
• Corrected Kueue LocalQueue Targeting in simple_job Example: Updated examples/simple_job documentation and CLI usage to pass Kueue LocalQueue names (cpu-queue, gpu-queue) instead of ClusterQueue names—preventing example workloads from stalling as Inadmissible—and documented required client environment variables (XMANAGER_ENDPOINT, IAP_CLIENT_ID, XMC_CLIENT_SA) (#1020).

---

🚀 2. Production-Ready Features Behind Feature Flags (Safe to Enable)

The following capabilities are fully developed, verified, and safe to be enabled. Their default enablement is temporarily postponed solely to align with managed solution milestones or external BYOD prerequisites.

🔹 [Flag: enable_metrics & dashboard_service_deployment_enabled] Metrics Pipeline & Dashboard Service

• How to enable:
  • Metrics Ingestion: Set enable_metrics: true in deploy/ctk_v2/xmc-deployment-template.yaml (or metrics.enabled: true in deploy/helm/values.yaml).
  • Dashboard Service: Set dashboard_service_deployment_enabled: true in deploy/ctk_v2/xmc-deployment-template.yaml (or dashboardServiceDeployment.enabled: true in deploy/helm/values.yaml). (Note: Dashboard Service requires BYOD storage devices—see the deployment README—and also requires enable_metrics / metrics.enabled to be enabled for workload metrics data to be generated).
• What's New:
  • Dynamic ClickHouse Connection Resolution & Hot-Reloading DataEngine: When running with components, dashboard-service dynamically resolves ClickHouse connection coordinates and credentials from the clickhouse Component CR (falling back to static CLICKHOUSE_* environment variables when absent), and automatically rebuilds and atomically swaps the cached ClickHouse DataEngine without requiring pod restarts whenever configuration or credentials change (#1037, #1038, #1039).
  • Expanded Nightly E2E Verification for Metrics & ClickHouse Workloads: Added dedicated automated E2E test verification for clickhouse_metrics_job, metrics_job, metrics_dashboard_job, and xdash_dashboard, continuously verifying that workloads successfully stream metrics through the Ingestion Gateway into ClickHouse and that charts hydrate correctly (#1057, #1064, #1099).
  • Hermetic KinD E2E 3P ClickHouse & Redpanda Integration Suite: Added a dedicated KinD E2E test suite running real OSS ClickHouse and Redpanda containers to validate metrics pipeline integration locally and in CI (#1105).

---

🧪 3. In-Development / Experimental Features Behind Flags

Warning

The following capabilities are under active ongoing development, experimental, and may not yet be functional. They are available for early preview and evaluation by enabling their deployment flags in deploy/ctk_v2/xmc-deployment-template.yaml (or deploy/ctk_v2/xmanager-cloud-deployment.yaml) and Helm values.yaml.

🔹 [Flag: enable_components] Components

• How to enable: Set enable_components: true in deploy/ctk_v2/xmc-deployment-template.yaml (or componentsDeployment.enabled: true in deploy/helm/values.yaml).
• What's New:
  • Single-Release stable-2026a Profile Meta-Chart: Packaged deploy/profiles/stable-2026a as a unified Helm meta-chart (xmc-profile-stable-2026a) that renders and validates every catalog Component Custom Resource (clickhouse, redpanda, ingestion-gateway) from a single values.yaml file with strict JSON Schema validation, extracted reusable profile chart contract tests, and updated component-linter to deduplicate nested chart Component definitions (#1016, #1034, #1059).
  • Self-Healing Drifted 1P Component Releases & Serialized Teardowns: xmc-bridge now verifies that all Kubernetes objects in a deployed 1P component's Helm manifest exist on cluster, reports missing resources as HelmReleaseDrifted, automatically repairs drifted or failed releases with rate-limited exponential backoff (MaxHistory=10), and serializes async Helm upgrades with uninstalls/deactivations so removed releases are never resurrected (#1030, #1061, #1072).
  • Dynamic ClickHouse Component CR Integration: Integrated pkg/components/crreader with dashboard-service so ClickHouse connection coordinates and credentials dynamically resolve from the clickhouse Component CR with hot-reloaded DataEngine updates (see Section 2 for details) (#1037, #1038, #1039).
  • Server-Side Apply (SSA) Status Applier, 3P Contract Breach Handling & Envtest Suite: Extracted ApplyComponentStatusSSA (fieldManager: "xmc-bridge") and conflict-retried finalizer synchronization, separated ContractBroken from InvalidWrapperSpec for 3P component status probes, modularized K8sHelmManager into focused sub-packages, and added a hermetic controller-runtime envtest integration suite covering CEL schema rules and ValidatingAdmissionPolicy deletion protection (#1067, #1068, #1069, #1070, #1071, #1077, #1104, #1118).
  • Dev Environment Deployment, CTK Verification & xmc-diag Component Diagnostics: Added --enable-components support to tools/dev_env (automatically installing the stable-2026a profile and verifying Component CRD, xmc-bridge, and Component CR readiness in verify_xmc.py) and extended tools/xmc-diag with --check-components to report Kubernetes Component CR health in text and JSON diagnostic reports (#1081, #1087).

🔹 [Flag: pause_scheduling_enabled] Pause Scheduling

• How to enable: Set pause_scheduling_enabled: true in deploy/ctk_v2/xmc-deployment-template.yaml (or featureFlags.pause_scheduling_enabled: true in deploy/helm/values.yaml).
• What's New:
  • Immediate STOPPED Transition for Paused Work Units: Stopping a work unit in the SCHEDULING_PAUSED state via StopWorkUnit or BatchStopWorkUnits now transitions its execution state directly to STOPPED in the Experiment State Server instead of waiting for a Kubernetes JobSet deletion watch event that never fires for unscheduled paused workloads (#1080).
  • Fast-Path Actuator Cache Advancement on Task Resumption: Replayed tasks now carry resumption_run_id and resumption_time metadata so an Actuator whose local pause cache has not yet refreshed immediately transitions to unpaused and dispatches replayed workloads to GKE instead of re-buffering them (#1089).
  • Graceful Pause State Error Handling in Health Service & Admin UI: GetHealthStatus now propagates scheduling_pause_error instead of silently failing open as unpaused when Spanner pause state cannot be read, while the Admin Panel renders an error banner and disables the pause scheduling toggle button to prevent conflicting mutations (#1093).
  • Pause & Resumption Prometheus Telemetry & Storage Decoupling: Instrumented 6 Prometheus metrics covering pause state, buffered task counts, and resumption replay across Health Service, Actuator, and ESS (/metrics), and decoupled pause scheduling domain types from the Spanner storage package (#1101, #1103).

🔹 [Flag: log_monitor_deployment_enabled] Log Monitor

• How to enable: Set log_monitor_deployment_enabled: true in deploy/ctk_v2/xmc-deployment-template.yaml (or logMonitorDeployment.enabled: true in deploy/helm/values.yaml).
• What's New:
  • In-Cluster GKE Deployment Option for Log Monitor: Added support for running the Log Monitor Pub/Sub streaming-pull subscriber as a Workload Identity-bound GKE Deployment (log_monitor_deployment_enabled: true) instead of a pinned Cloud Run service (false by default for staged migration), separated /healthz liveness from /readyz Spanner/PubSub readiness probes so transient Spanner outages do not restart-loop pods, and updated Health Service and verify_xmc.py to probe the configured deployment mode (#1027, #1082).

---

⚙️ 4. Operational, Infrastructure & Engineering Velocity

Changes relevant to cluster operators, CI/CD, and internal architecture that do not alter the end-user interface.

• GKE Deployment & Upgrade Hardening (tools/dev_env & verify_xmc.py): Added an automatic migrate step to gke-deploy (with --skip-migrations opt-out) so in-place GKE deployments apply pending Spanner migrations and roll pods before verification, updated verify_xmc.py and step_verify to wait for kubectl rollout status and fail closed on unhealthy pods or broken connectivity (with --skip-verification), purged stale .ghpc/staged/ charts before gcluster create, added deterministic SHA-256 Helm chart fingerprinting (helm_chart_content_hash) to trigger umbrella Helm upgrades on template-only changes, and added off-PATH CLI diagnosis (~/.terraform/bin, ~/go/bin, ~/.local/bin) (#1020, #1031, #1032, #1079, #1082).
• Google3 Compatibility Shim Elimination & Pure OSS Build: Removed internal google3 compatibility shims across Go and TypeScript (internal/compat/*, compat.bzl Load()/LoadConfig() flag injection, google3 runtime flags in Helm/Terraform, //go:embed path rewrites, and UI Closure/Jasmine stubs), replaced safesql and Tink shims with OSS packages, built grpc-gateway directly from OSS go_deps, and enforced a pre-commit/CI legacy import ratchet (check_legacy_imports.py) (#1074, #1075, #1078, #1095, #1123, #1124, #1125, #1126).
• Protobuf Breaking Change Detection (rules_buf): Added //api:protos_breaking_test using rules_buf (buf v1.73.0) to enforce FILE-level wire and source compatibility across all 11 first-party proto_library targets against a committed baseline (api/testdata/protos_baseline.binpb), and protected api/buf.yaml, api/BUILD.bazel, and the baseline image in the incremental PR approval policy (#1154).
• Nightly Examples E2E Expansion & Verification Gate: Expanded daily Nightly Examples E2E coverage across all 13 SDK examples (simple_job, clickhouse_metrics_job, workunit_artifact_job, chart_artifact_job, init_container_job, metrics_dashboard_job, metrics_job, multipod_job, sidecar, managed_xprof_artifact_job, per_job_node_selector, pathways_job, and xdash_dashboard), modularized tools/nightly_tests/common.py into tools/nightly_tests/verification/, and enforced a pre-commit/CI gate (check_examples_coverage.py) requiring every example under examples/ to wire a nightly E2E verifier (#1057, #1064, #1098, #1099, #1132, #1147).
• KinD E2E Harness Resilience, 3P Integration Suite & CI Automation: Hardened the KinD E2E runner with bounded Spanner retry budgets and redacted project-number stderr logging, phase-classified failure reporting with single-retry on transient infrastructure errors, signal/deadline-aware teardown, GitHub Actions job summary outcome tables, and a new 3P ClickHouse + Redpanda KinD E2E integration suite; also enforced mypy static type checking in pre-commit/CI, ran GitHub Actions script tests in lint.yml, and added a Dependabot auto-merge workflow (#1094, #1105, #1134, #1136, #1141, #1142, #1143, #1144, #1145).
• Reconciler, ESS & Health Service Internal Modularization: Decomposed the JobSet reconciler and hypercomputeclient into focused modules (predicates.go, startup_diagnostics.go, managed_xprof.go, mlrun_sync.go, mlrun_metrics.go, transport/operations/mlrun), simplified restartWorkUnitExecutions while preserving error chains in ESS, lazily initialized Spanner clients and providers in Health Service, and updated engineering rules and skills (#1006, #1051, #1060, #1084, #1109, #1110, #1111, #1112, #1113, #1133).
• Dependency & Security Maintenance: Remediated Dependabot security alerts in cmd/ui for compression (CVE-2026-87776), webpack-dev-middleware (CVE-2026-76844), and piscina (CVE-2026-102992), and bumped @modelcontextprotocol/sdk, shell-quote, and npm_and_yarn dependencies (#1091, #1097, #1102, #1120, #1148).