Release 0.15.0

XMC Release 0.15.0

Release Version Determination: Minor bump (0.14.0 -> 0.15.0).
Rationale: Promotes Pause Scheduling (pause_scheduling_enabled: true) to enabled out-of-the-box by default across Cluster Toolkit and Helm deployments (backed by Spanner schema migration 013_pause_scheduling), adds Work Units table sorting and status filtering in the Web UI, fixes JobSet reconciler STOPPED status persistence and ROD gateway routing, expands Nightly Playwright UI E2E coverage across Metrics Dashboards, ROD, and the Admin Panel, and advances behind-the-flag capabilities for Components (CMS authentication/authorization, Workload Identity, kstatus conditions, and 014_component_intent_source migration).

---

🌟 1. Public Changes (Available Out-of-the-Box)

🚀 Major Highlights & Promoted Capabilities

• [Promoted to Default] Pause Scheduling (pause_scheduling_enabled: true): Cluster-wide workload scheduling pause and resume is now enabled out-of-the-box by default in deploy/ctk_v2/xmanager-cloud.yaml (deploy/ctk_v2/xmc-deployment-template.yaml) and deploy/helm/values.yaml (featureFlags.pause_scheduling_enabled: true) (#1283).
  • Operator & User Experience: Administrators can pause and resume workload scheduling directly from the Admin Panel (/admin) or Health Service API during cluster maintenance while researchers continue submitting experiments without disruption. While paused, the Experiments List page (/experiments) displays a global scheduling-paused banner (#1217) and the Actuator records a clear SCHEDULING_PAUSED system status message on buffered work units (#1258).
  • Spanner Persistence & Broadcast Resilience: Added the Spanner PauseState and PausedTasks schema (013_pause_scheduling) and storage repositories (#1046, #1085, #1116), hardened Health Service and the Actuator broadcast listener against Redis pub/sub outages and transient Spanner read errors (#1205, #1289), and documented the end-to-end architecture and operational flows (#1264).
• [Action Required / Database Migrations] Spanner ESS Schema Migrations (013 & 014): This release adds Spanner ESS migrations 013_pause_scheduling (PauseState and PausedTasks tables, #1085) and 014_component_intent_source (request_source column on ComponentIntents, #1232). When upgrading an existing deployment, ensure Spanner migrations are applied (executed automatically by default during dev_env.py gke-deploy and gke-upgrade).

✨ New Features & Improvements

• Work Units Table Sorting & Status Filtering in Web UI: Users can now sort the Work Units table on the Experiment Details page (/experiments/:id) by Work Unit ID, Status, Created time, and Completed time (#1251), and filter work units by status directly from the Work Units filters menu (#1252).

🐛 User-Facing Bug Fixes & Stability

• Work Units Table Reload on Experiment Navigation: Fixed an issue in the Web UI where navigating directly between different experiment detail routes did not reload the Work Units table for the newly selected experiment ID (#1260).
• ROD batchGetSecondaryData Gateway Route Matching: Fixed the Envoy Gateway HTTPRoute for the Resource Overview Dashboard (rod) to match the AIP-136 custom verb /v1/clusterqueues:batchGetSecondaryData, ensuring queue velocity and predictability metrics resolve reliably through the gateway (#1212).
• JobSet Reconciler STOPPED State Ordering & Batcher Fault Isolation: Fixed the JobSet reconciler to flush and persist the STOPPED status in Spanner before releasing the JobSet protection finalizer (#1189), added bounded backoff retries for failed batcher flushes (#1200), isolated permanent per-work-unit batcher errors via typed WorkUnitNotFoundError so missing work units never block status updates for sibling work units (#1201, #1202), and mapped Spanner NotFound / AlreadyExists errors to canonical grpcerr sentinels (#1250).

---

🚀 2. Production-Ready Features Behind Feature Flags (Safe to Enable)

The following capabilities are fully developed, verified, and safe to be enabled. Their default enablement is temporarily postponed solely to align with managed solution milestones or external BYOD prerequisites.

🔹 [Flag: enable_metrics & dashboard_service_deployment_enabled] Metrics Pipeline & Dashboard Service

• How to enable:
  • Metrics Ingestion: Set enable_metrics: true in deploy/ctk_v2/xmc-deployment-template.yaml (or metrics.enabled: true in deploy/helm/values.yaml).
  • Dashboard Service: Set dashboard_service_deployment_enabled: true in deploy/ctk_v2/xmc-deployment-template.yaml (or dashboardServiceDeployment.enabled: true in deploy/helm/values.yaml). (Note: Dashboard Service requires BYOD storage devices—see the deployment README—and also requires enable_metrics / metrics.enabled to be enabled for workload metrics data to be generated).
• What's New:
  • Nightly Playwright UI E2E Verification for Metrics Dashboards: Added automated Playwright browser verification (metrics_dashboard_ui_verify.py) to the Nightly E2E suite, asserting that dashboards created by metrics_dashboard_job and xdash_dashboard render cleanly on /dashboards and /dashboards/:id with hydrated chart cards and zero runtime or HTTP errors (#1256).
  • Backend-Neutral Dashboard Storage Logic & Sentinel Error Mapping: Extracted dashboard chart plot construction and validation into backend-neutral storage packages (internal/storage/data) (#1196) and mapped Spanner NotFound / AlreadyExists errors in GetChart and DeleteChart to canonical grpcerr status codes (#1250).

---

🧪 3. In-Development / Experimental Features Behind Flags

Warning

The following capabilities are under active ongoing development, experimental, and may not yet be functional. They are available for early preview and evaluation by enabling their deployment flags in deploy/ctk_v2/xmc-deployment-template.yaml (or deploy/ctk_v2/xmanager-cloud.yaml) and Helm values.yaml.

🔹 [Flag: enable_components] Components

• How to enable: Set enable_components: true in deploy/ctk_v2/xmc-deployment-template.yaml (or componentsDeployment.enabled: true in deploy/helm/values.yaml).
• What's New:
  • CMS Authentication, Mutating RPC Authorization & Request-Source Ownership: Enforced gRPC authentication and mutating RPC authorization on the Component Management Service (CMS) in ESS (#1219), configured per-RPC bearer token credentials and GKE Workload Identity bindings for xmc-bridge (#1247), and recorded the authenticated request source as the owner of sourced component intents (014_component_intent_source, #1232).
  • controller-runtime Bridge Architecture, Leader Election & Configurable Concurrency: Migrated xmc-bridge to a controller-runtime controller running as a single leader-elected writer (#1224, #1225), made reconcile timeout and concurrency configurable via CLI flags and Helm values (#1272), and tracked CMS sync and deactivation rechecks in memory (#1229).
  • kstatus Conditions, Finalizer Ordering & CRD Lifecycle Retention: Emitted standard kstatus Reconciling and Stalled conditions alongside decoupled Synced and Ready conditions on Component Custom Resources (#1227, #1228), attached the CMS cleanup finalizer before executing any side effects (#1226), preserved the Component CRD across Helm uninstalls and feature-flag flips (#1223), and added KinD E2E lifecycle tests covering blocked deactivation and finalizer cleanup (#1151, #1276).
  • Unified Idempotent Intent Planning & Structured ErrorInfo Rejections: Consolidated component intent writes into a single shared plan with idempotent application and a Spanner no-op fast path (#1152, #1155, #1231), and attached structured google.rpc.ErrorInfo reasons to missing-dependency, cycle, and active-dependents rejections (#1230).

---

⚙️ 4. Operational, Infrastructure & Engineering Velocity

Changes relevant to cluster operators, CI/CD, and internal architecture that do not alter the end-user interface.

• Real Upstream JobSet & Kueue Controllers in KinD E2E: Installed real upstream Kubernetes JobSet and Kueue CRDs and controllers in the KinD E2E harness, adding end-to-end verification for JobSet schema actuation, COMPLETED and FAILED pod terminal state transitions, and Day-2 work-unit stop, restart, and Kueue preemption flows (#1207, #1208, #1268, #1269, #1270).
• Nightly MultiKueue Addon & Playwright UI E2E Suites (ROD, Admin Panel, Dashboards): Added a MultiKueue worker addon (deploy/addons/multikueue) and nightly E2E verification (#1182), added Playwright UI E2E verifiers for the Resource Observability Dashboard (/resources, #1204), Admin Panel (/admin, #1280), and Metrics Dashboards (/dashboards, #1256), consolidated the nightly workflows, and split tools/nightly_tests into verification/ and examples_verification/ (#1206, #1253, #1261, #1266, #1286).
• PostgreSQL Schema Foundation & Backend-Neutral Domain Extraction: Added canonical PostgreSQL schema files (ess-schema.sql, xid-schema.sql) and baseline migrations (013_baseline, 014_component_intent_source) (#1259, #1279), and extracted backend-neutral domain orchestration out of Spanner storage packages into internal/experiment, internal/workunit, internal/storage/data, and cmd/rod_data_provider (#1187, #1195, #1196, #1218, #1222, #1249, #1281, #1282, #1285).
• Completion of Pure OSS Bazel & Import Path Normalization: Removed compat.bzl, internal/compat/status, third_party/xmanager_cloud proto prefixes, custom protoc genrules, and go_default_library target suffixes across all Go packages, protobuf definitions, Python SDKs, and Angular UI imports (app/* and app/protos) (#1166, #1167, #1168, #1169, #1175, #1176, #1190, #1197, #1198, #1199, #1209, #1210, #1213, #1214, #1215, #1221, #1262).
• Actuator Builder Decomposition, Affected-Test Runner & CI Governance: Decomposed Actuator jobset.go into dedicated container, network, coordinator, policy, and CRUD lifecycle builders (#1128, #1129, #1130), added tools/run_affected_tests.py with Bazel rdeps test-scope resolution for local pre-commit and CI backend test execution (#1161, #1162, #1163, #1164), prepared E2E checks for merge-queue gating and refined incremental PR re-approval for code owners (#1192, #1265, #1290), and updated the CTK Spanner schema applier to locate the workspace root via MODULE.bazel (#1271).