Skip to content

XMC Release Notes 0.10.0

Choose a tag to compare

View all tags
@janzak-dev janzak-dev released this 11 Sep 12:35
· 505 commits to main since this release
0.10.0
ef8c353
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

XMC Release Notes (Week of September 7 – September 11, 2026)

📌 Release Summary & SemVer Determination

  • Analysis Timeframe: 2026-09-07T00:00:00Z to 2026-09-11T12:18:09Z
  • Total Merged Pull Requests: 46 PRs (100% accounted for & verified)
  • Recommended Version Bump: Minor Bump (0.Y.0 / X.Y.0)
    • Reasoning: Introduces backwards-compatible public features (ChartArtifact full-width plot rendering and automated plot specs, permanent promotion of core platform status/error tracking, resilient task retry classification), new production-ready blueprint and Helm toggles (Managed ML Diagnostics on GKE, federated JWKS/OIDC authentication, hybrid remote Kubernetes cluster actuation), and major architectural additions across preview subsystems (Custom Components).

Section 1: 🌟 Public Changes (Available Out-of-the-Box)

🚀 Major Highlights & Promoted Capabilities

  • Core Platform Guarantees Promoted to Permanent Invariants: Status history tracking, inadmissible workload detection, and startup pod error inspection have been stabilized and promoted to permanent, out-of-the-box platform guarantees with the permanent removal of their legacy deployment flags (#801).
  • Full-Width Interactive Chart & Metric Visualization: Researchers can now view wide-aspect ChartArtifact time-series graphs spanning the entire Work Unit details page, dramatically improving metric readability and visual analysis across desktop displays (#811).
  • Automated Plot Specification Rendering for Artifacts: Users no longer need to write custom frontend charting code to visualize experiment results; declaring a ChartArtifact automatically provisions and renders complete interactive plot specifications in the dashboard (#810).

✨ New Features & User-Facing Improvements

  • Live Metric Streaming Example Workload: Developers and researchers can reference an updated example job demonstrating how to stream live multi-metric training series (such as loss, accuracy, learning rate, and step time) directly into XMC dashboards (#676).
  • Multi-Worker Job Template with Automated Queue Submission: Researchers can launch multi-workunit batch jobs across Kueue resource queues using a standardized example script that automatically manages dynamic project domains and worker replicas (#847).

🐛 User-Facing Bug Fixes & Stability

  • Resilient Workload State Handling on Network Glitches: Workloads no longer erroneously fail during transient Kubernetes API timeouts or network hiccups, as retryable network errors are automatically distinguished from terminal container failures in task workers (#814).
  • Intuitive UI Action Button Tooltip: Improved interface clarity by renaming the Work Units table action button tooltip from "Run Profile" to "Profile" (#851).

Section 2: 🚀 Production-Ready Features Behind Feature Flags (Safe to Enable)

1. Google Cloud Managed ML Diagnostics on GKE

  • Configuration Toggle:
    • Cluster Toolkit Blueprint: Set enable_managed_mldiagnostics: true and managed_xprof_gcs_bucket: "gs://<bucket-name>" in deploy/ctk_v2/xmanager-cloud.yaml (#762).
    • Helm Chart: Set managedMLDiagnostics.enabled: true in deploy/helm/values.yaml.
  • Functional Capability & Operator Enablement:
    • Delivers zero-configuration performance profiling and diagnostic trace capture for JAX, PyTorch, and TensorFlow workloads running on GKE TPU/GPU clusters.
    • Automatically injects profiling sidecars into training pods and provisions required Cloud IAM bindings and GKE cluster integrations (#762).
    • Single-Click Profiling Navigation: Reconciler automatically appends tab=profiles to MLRun Diagon console URLs, landing users directly on active profiling dashboards (#791).
    • Scheduling Latency Visibility: Exports JobSet creation timestamps to MLRun workload details, allowing operators to measure queuing and scheduling delay (#822).
    • Reliable Session Discovery: Eliminates profiling session discovery race conditions on active training runs (#839).
    • Safe Metadata Updates: Preserves workload ownership labels and orchestrator details during experiment updates (#845).
    • Standardized Label Formats: Migrated system labels to snake_case for consistent search and query filtering (#819).
    • Streamlined Configuration: Cleaned up redundant environment variables and CLI overrides in favor of per-work-unit bucket paths (#840, #842).
  • Associated PRs: #762, #791, #819, #822, #839, #840, #842, #845.

2. Hybrid Remote Cluster Management & Out-of-Cluster Kubernetes

  • Configuration Toggle:
    • Helm Chart: Set remoteK8s.enabled: true and mount cluster credentials via remoteK8s.secretName: "remote-k8s-config" in deploy/helm/values.yaml (#798).
    • Dev Env CLI: Run dev_env.py kind-up --remote-k8s --remote-k8s-context=<context> (#799).
  • Functional Capability & Operator Enablement:
    • Enables hybrid deployment topologies where a lightweight XMC control plane (running in local KinD or a central management cluster) schedules, monitors, and manages training workloads across remote compute clusters without deploying redundant control planes (#798, #799).
  • Associated PRs: #798, #799.

Section 3: 🧪 In-Development / Experimental Features Behind Flags

Warning

The following capabilities are in active preview and default to disabled (false). Schemas and interfaces are subject to change prior to general availability.

Pluggable Custom Components Framework

  • Configuration Flag:
    • Cluster Toolkit: enable_components: false in deploy/ctk_v2/xmanager-cloud.yaml
    • Helm Chart: componentsDeployment.enabled: false in deploy/helm/values.yaml
    • CLI / Env: --enable_components=false / ENABLE_COMPONENTS=false
  • Functional Capability & Operator Enablement (Preview):
    • Proactive Experiment Drift Detection: Experiment State Server checks Spanner consistency views to detect divergences between live component catalogs and immutable experiment snapshots, surfacing warning annotations in the UI (#805).
    • Automated Lifecycle Management for 3rd-Party Addons: Enables declarative installation, upgrading, and uninstallation of auxiliary Helm packages directly alongside experiment workloads via an in-process Go Helm SDK runner (#806).
    • Custom Component Parameterization: Allows users to customize addon behavior by declaring and passing configurable parameters to managed components (#827).
    • Live Addon Health & Status Probing: Automatically discovers Kubernetes Custom Resource Definitions (CRDs) via JSONPath evaluation (#581, #708), executes TCP socket connectivity checks (#582), and reports aggregated health status (#792).
    • Dependency-Aware Workload Placement: Workloads requiring specific cluster services can automatically wait for dependent components to become healthy before starting (#802).
    • Dynamic Value Persistence: Persists resolved component values inside cm-<release-name> ConfigMaps for traceable release state (#735).
    • Unified Component Schema & Test Doubles: Standardized component references and mock management tools to accelerate ecosystem extension development (#766).
  • Associated PRs: #766, #792, #802, #805, #806, #827, #735, #581, #582, #708.

Section 4: ⚙️ Operational, Infrastructure & Engineering Velocity

⚙️ Operational & Infrastructure Updates

  • Automated Nightly GKE Staging & Health Validation: Continuous nightly deployment pipelines now build container images, compile Cluster Toolkit blueprints, provision GKE environments, and execute end-to-end health verification automatically every day (#844).
  • Cloud Quota & Capacity Resiliency: Switched default system node pools in Cluster Toolkit blueprints to n2-standard-4, avoiding deployment failures caused by regional VM stockouts in us-central1 (#844).
  • Portable Database Schema Provisioning: Standardized the database schema migration runner on portable shell execution across heterogeneous deployment runners (#844).
  • Security & Microservice Dependency Upgrades: Kept the platform secure and up to date by upgrading gRPC communication libraries (#781, #825) and web service dependencies (#829).

🛠️ Developer Velocity & Engineering Quality

  • Accelerated & Reliable CI Validation Pipelines: Consolidated post-merge CI into a structured, dependency-aware workflow (#830), resolved workflow permission inheritance (#838, #846), and fortified pull request incremental review automation against git reference errors and header corruption (#789, #820, #821).
  • Resilient Test Execution: Unit test runners now automatically fall back to full test suites when selective build target resolution encounters diff ambiguities, preventing false negative CI runs (#824).
  • Faster Pre-Commit & CI Lint Checks: Decoupled component linting tools from heavy backend storage libraries, allowing linter checks to run substantially faster as standalone compiled binaries (#778).
  • Automated Release Metadata Extraction Tooling: Introduced dedicated CLI tooling to reliably inspect git history and extract pull request metadata for release documentation (#703).
  • Automated Visual UI Regression Suite: Added comprehensive end-to-end Playwright screenshot tests covering Experiment Lists, Details, and Work Unit views to prevent visual regressions across releases (#788).
  • Codebase Hygiene & Clear Architecture Reference:
    • Published comprehensive end-to-end architecture documentation covering all 12 platform microservices and 22 interactive workflow diagrams (#668).
    • Codified declarative unit testing assertion standards in Go guidelines (#833).
    • Cleaned up obsolete Python client files (#831) and deprecated review checklists (#795).
    • Updated reviewer availability tracking to ensure smooth and uninterrupted pull request triage (#790, #843).
Assets 2
Loading