Skip to content

Release 0.18.0

Latest
Latest

Choose a tag to compare

View all tags
@janzak-dev janzak-dev released this 06 Nov 15:07
0.18.0
b103091
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

XMC Release 0.18.0

Release Version Determination: Minor bump (0.17.0 -> 0.18.0).
Rationale: Marks XMC's launch as a public open-source repository (pre-1.0 milestone), opens the 0.18.x one-time transition window for migrating legacy umbrella-managed ingestion-gateway metrics workloads to the stable-2026a Component profile (enable_components: true), promotes Components (enable_components / componentsDeployment.enabled) to Section 2 with an end-to-end ComponentStability contract across stable-2026a (redpanda, clickhouse, and ingestion-gateway as STABLE), introduces experimental Vizier (3P) and HPO (xmc-study-controller, 1P) components (EXPERIMENTAL with acknowledgeExperimental), adds ESS database schema migration 017 (Xid queue ordering tie-breaker), and delivers Resource Observability Dashboard (/resources) enhancements alongside BYO PostgreSQL support in Cluster Toolkit.

Important

Running ingestion-gateway without Components (enable_components: false)? 0.18.x is the only release for migrating to Components. Complete the cutover in docs/migrations/components.md before upgrading to 0.19.0. See the first [Action Required] item below.

🌍 Public Open-Source Repository Launch (Pre-1.0 Milestone)

XMC is now available as a public open-source repository! While XMC remains pre-1.0 as we continue stabilizing core APIs, database backends, and component profiles toward a future 1.0 release, 0.18.0 marks our transition to open-source development with public onboarding documentation, contribution workflows, and security reporting guidelines in place.

🌟 1. Public Changes (Available Out-of-the-Box)

🚀 Major Highlights & Promoted Capabilities

  • [Action Required] One-Time Components Transition Window (0.18.x Only) & Post-Migration xmc-profile Upgrades:
    • Supported Migration Window (0.18.x Only): 0.18.x is the only supported window for migrating an existing installation from legacy umbrella-managed metrics workloads (enable_components: false, umbrella ingestion-gateway) to the stable-2026a Component profile (enable_components: true). Starting with 0.19.0, the legacy non-component metrics templates and the one-time migration tool (tools/migrations/components/migrate.py) are scheduled for removal. Operators running ingestion-gateway from the umbrella chart must complete the cutover on 0.18.x before upgrading to 0.19.0 (clusters without a live umbrella ingestion-gateway do not need to run the handoff, as handoff is a no-op for them).
    • Cutover Runbook: Follow docs/migrations/components.md (prepare -> handoff -> upgrade with components -> install xmc-profile -> finalize) (#1461, #1606). The cutover keeps the live ingestion-gateway serving throughout: Helm ownership is handed over without deleting pods, and xmc-bridge then adopts the release with a rolling update.
    • Upgrading XMC After Migration: Because xmc-profile is a separate Helm release that gcluster deploy and umbrella helm upgrade do not upgrade, on every subsequent XMC upgrade operators must bump the pinned gateway image tag in profile-values.yaml (components.ingestion-gateway.imageTag, and components.ingestion-gateway.parameters.image if set) and re-run the xmc-profile upgrade (docs/migrations/components.md section "After Migration: Upgrading XMC", #1606).
    • New Installations: Fresh installations do not need the migration tool—start directly by explicitly setting enable_components: true (componentsDeployment.enabled: true; the default is still false) and install the stable-2026a profile (deploy/profiles/stable-2026a/README.md).
  • [Action Required] Cluster Toolkit Upgrades Must Re-Compile the Blueprint: INSTALL.md upgrade Step 4 (Option A) now runs ./cluster-toolkit/gcluster create deploy/ctk_v2/xmanager-cloud.yaml -d <deployment_config_path> -o ctk-out -l WARNING -w --force before ./cluster-toolkit/gcluster deploy ctk-out/<deployment_name> --only xmc_app --auto-approve. Running gcluster deploy alone re-applies a stale ctk-out/<deployment_name> without your updated image_tag, enable_components setting, or chart templates. This applies to every Cluster Toolkit upgrade, including the Components cutover (#1606).
  • [Action Required] ESS Database Schema Migration 017 (017_work_units_queue_index_xid_tiebreaker): Added migration 017_work_units_queue_index_xid_tiebreaker for both Spanner and PostgreSQL to append Xid as a deterministic tie-breaker to the active work unit queue ordering index (#1491). Ensure schema migrations are applied before upgrading workloads: they are not run automatically by gcluster deploy. Follow INSTALL.md → Step 3: Run Database Schema Migrations (or pass --run-migrations to dev_env.py gke-upgrade in development environments).

✨ New Features & Improvements

  • Resource Observability Dashboard (/resources) Enhancements:
    • Experiment Title & Live Running Duration in Queue Jobs Table: Added the parent experiment title (#1522) and live running duration for active jobs (#1588) to the Queue Jobs side panel table.
    • Draggable Pane Resize Divider & Persisted Full-Row Expandable Jobs Toggle: Added a draggable resize divider between the queues list and side panel (#1562) and a full-row expandable jobs layout toggle backed by localStorage persistence (#1570).
    • Generic Accelerator Parent Categories in Resource Type Picker: Updated the resource type filter picker to offer generic accelerator parent categories (e.g. grouping accelerator variants) (#1526).
  • Configurable Web UI Documentation Link (XMC_DOCS_URL): Made the documentation link in the Web UI navigation header configurable via the XMC_DOCS_URL environment variable and Helm chart values (#1584).

🐛 User-Facing Bug Fixes & Stability

  • Executing-Cluster Resolution for Work Unit Logging & Dashboard Links: Updated the Reconciler and Web UI URL processor so work unit Cloud Logging and GKE dashboard links point to the actual executing cluster (properly supporting MultiKueue delegated workloads on remote clusters) (#1551).
  • Deterministic Active Work Unit Queue Ordering: Added Xid as a secondary sort tie-breaker when fetching active work units by queue across Spanner and PostgreSQL (#1491).
  • Opaque Search Syntax Help Popover: Fixed the experiment list search syntax help popover background so it renders opaquely over table content (#1513).
  • Repo-Checkout SDK Verification in install_xmanager.py: Updated deploy/cli/install_xmanager.py to assert that local Python SDKs (xmc and xdash) resolve from the repository checkout after installation (#1485).

🚀 2. Production-Ready Features Behind Feature Flags (Safe to Enable)

The following capabilities are fully developed, verified, and safe to be enabled. Their default enablement is temporarily postponed solely to align with managed solution milestones or external BYOD prerequisites.

🔹 [Flag: enable_components] Components

  • How to enable: Set enable_components: true in deploy/ctk_v2/xmc-deployment-template.yaml (or componentsDeployment.enabled: true in deploy/helm/values.yaml), then install the stable-2026a Component profile (deploy/profiles/stable-2026a/README.md) or migrate an existing installation using docs/migrations/components.md.
  • stable-2026a Profile Catalog & Declared Stability:
    • redpanda: STABLE
    • clickhouse: STABLE
    • ingestion-gateway: STABLE
    • vizier: EXPERIMENTAL (requires acknowledgeExperimental: true — see Section 3)
    • hpo: EXPERIMENTAL (requires acknowledgeExperimental: true — see Section 3)
  • What's New:
    • ComponentStability Contract & CRD Admission Enforcement: Added the ComponentStability contract (STABLE, EXPERIMENTAL) to components.proto and the Component CRD admission rules, declaring explicit stability tiers across the stable-2026a catalog (#1538). Note: spec.stability is now a required Component field. If you author or commit Component manifests yourself (e.g. GitOps), re-render them from the 0.18.0 stable-2026a profile so they are not rejected on apply (#1564).
    • Stability Gating & Dependency Inversion Rules in CMS, xmc-bridge, Profile Chart & Linter: Enforced acknowledgeExperimental gating and dependency stability inversion rules (preventing STABLE components from depending on EXPERIMENTAL components) at runtime in CMS and xmc-bridge (#1539) as well as statically in the stable-2026a profile chart and cmd/component-linter (#1540).
    • End-to-End Component Stability Visibility (ESS, Health Service, Admin Panel UI & xmc-diag): Surfaced component stability tiers across ESS drift detection, Health Service responses, the Admin Panel UI (/admin), and tools/xmc-diag output (#1541).
    • One-Time Legacy Metrics Migration Tool (migrate.py) & Runbook: Added tools/migrations/components/migrate.py (prepare, handoff, finalize) and the step-by-step operator runbook in docs/migrations/components.md (linked from INSTALL.md upgrade Step 6) for migrating umbrella-managed metrics workloads to stable-2026a while keeping the live gateway serving (#1461, #1606).

🧪 3. In-Development / Experimental Features Behind Flags

Warning

The following capabilities are under active ongoing development, experimental, and may not yet be functional. They are available for early preview and evaluation by enabling their deployment flags in deploy/ctk_v2/xmc-deployment-template.yaml (or deploy/ctk_v2/xmanager-cloud.yaml) and Helm values.yaml.

🔹 [Flag: enable_components + acknowledgeExperimental: true] Experimental Vizier (3P) & HPO (xmc-study-controller, 1P) Components

  • How to enable: With enable_components: true (componentsDeployment.enabled: true), enable components.vizier and/or components.hpo in your stable-2026a profile values and set acknowledgeExperimental: true on each enabled experimental component.
  • What's New:
    • vizier (3P) & hpo (1P) Component Registration in stable-2026a: Registered the vizier (3P OSS Vizier service) and hpo (1P hyperparameter optimization controller) components in the stable-2026a profile catalog with EXPERIMENTAL stability (#1566).
    • xmc-study-controller Daemon Binary, Image & hpo Component Workload: Added the cmd/study_controller (xmc-study-controller) daemon binary and container image with health probes and configuration loading (#1575), and wired the hpo component chart and xmc-bridge RBAC/Helm builder to deploy xmc-study-controller as the hpo workload (#1580).
    • Dev/Test-Only Vizier Server Fixture for KinD 3P E2E Verification: Added a hermetic dev/test-only Vizier server fixture (test/e2e/testdata/vizier) and KinD E2E harness test for the vizier 3P component (#1583).

🔹 [Flag: database.experimentalPostgres] Experimental PostgreSQL Control-Plane Backend

  • How to enable: Set database.backend: postgres and database.experimentalPostgres: true in deploy/helm/values.yaml (or configure BYO PostgreSQL in Cluster Toolkit via deploy/ctk_v2/xmc-deployment-template.yaml).
  • What's New:
    • BYO PostgreSQL Support in Cluster Toolkit & dev_env.py: Added Bring-Your-Own (BYO) PostgreSQL support to the Cluster Toolkit (deploy/ctk_v2) blueprint modules and upgrade_dev_env.py (#1498).
    • Component Manager Service (cms) & PostgreSQL dbstore Parity: Implemented PostgreSQL component writes, ran the full dbstore concurrency and fencing suite against PostgreSQL (#1418), and wired ESS to serve the Component Manager Service from dbstore on PostgreSQL (#1505).
    • ESS PostgreSQL Retention Sweeper & Deadline-Aware Serializable Retries: Added an ESS background retention sweeper to delete expired rows on PostgreSQL (matching Spanner TTL behavior) (#1496), retried PostgreSQL serializable transactions until the context deadline (#1507), and removed deprecated per-service database config blocks in favor of the unified db-config (#1506).

🔹 [Flag: enable_managed_mldiagnostics] Managed ML Diagnostics

  • How to enable: Set enable_managed_mldiagnostics: true in deploy/ctk_v2/xmc-deployment-template.yaml (or reconciler.enableManagedMlDiagnostics: true in deploy/helm/values.yaml).
  • What's New:
    • mlrun URL Caching & 409 Conflict Log Downgrade in Reconciler: Cached resolved mlrun URLs across the batch flush window in the JobSet reconciler and downgraded expected 409 conflict responses during mlrun sync from error to info/debug logs (#1500).

⚙️ 4. Operational, Infrastructure & Engineering Velocity

Changes relevant to cluster operators, CI/CD, and internal architecture that do not alter the end-user interface.

  • Public Repository Launch Readiness, Documentation & Sanitization: Rewrote README.md for public onboarding and architecture discovery (#1527, #1601), added SECURITY.md vulnerability reporting guidelines (#1528), documented the fork-based pull request workflow in CONTRIBUTING.md (#1529), and sanitized hardcoded environment domains, project IDs, and internal references across the codebase (#1519).
  • Nightly E2E Verification, Triage Tooling & Per-PR Examples Smoke Checks: Added per-PR and nightly examples-smoke import verification across all example entrypoints and verification modules (#1486, #1523, #1524, #1530), fixed nightly-visual-snapshot.yml to install xmanager and local xdash (#1504), added the triage_nightly_run.py CLI, /XMC Fix Nightly skill, and branch dispatch support in dispatch_and_watch.py (#1549), hardened ROD UI nightly verification and pure parsing unit tests (#1545, #1547, #1550), and updated KinD E2E Component CR fixtures to set required spec.stability (#1564).
  • Repo-Wide Lint, Type-Checking & Build Toolchain Hygiene: Enabled whole-repo mypy type checking and cross-file pre-commit gates on file deletions (#1565, #1571), enforced repository file-naming conventions via check_file_naming.py (#1573), aligned UI dependencies with the Angular build toolchain and run_ui_tests.py fast-fail hints (#1512, #1525, #1531), included //cmd/ui Python tests in backend CI test scope resolution (#1556), and hardened tools/common/tool_utils.py, tools/pr_review/, and agent workflow documentation (#1518, #1520, #1532, #1533, #1534, #1535, #1536, #1552, #1553, #1554, #1555, #1557, #1558, #1559, #1560, #1574).
Assets 2
Loading