Skip to content

OME 1.2.2

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 05 Aug 17:48
· 489 commits to main since this release
v1.2.2
5f1c409
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

OME 1.2.2

1.2.2 is about shedding weight and making the model agent trustworthy under churn. Two subsystems leave the tree — the Knative-backed Serverless deployment mode and the in-repo web console, while model-agent status reconciliation gains UID fencing so a deleted model can no longer be resurrected by an in-flight write. Three new model families land alongside.

✨ Highlights

Serverless (Knative) Deployment Mode Is Removed (#681)

Serverless is gone as a deployment mode, and with it the entire Knative integration — roughly 5.9k lines and 10 files:

  • Knative Serving is no longer a dependency — it drops out of the optional-components list entirely, and Istio is now needed only for Virtual Service ingress
  • minReplicas: 0 no longer selects Serverless — it falls through to RawDeployment, so scale-to-zero is no longer available; RawDeployment explicitly does not support scale to/from zero
  • Removed components — the Knative Service reconciler, the serverless ingress strategy, the VirtualService builder, the qpext queue-proxy extension (binary, tests, and Dockerfile — the image is no longer built or published), and the queue-proxy metrics-aggregation pod mutator along with its EnableMetricAggregation / SetPrometheusAnnotation annotations
  • Ingress config keys removed — localGateway, localGatewayServiceName, and knativeLocalGatewayService no longer exist in the ingress config

Behavior on a stale annotation: Serverless is no longer a valid ome.io/deploymentMode value, so the annotation is silently ignored rather than rejected — the InferenceService falls back to defaultDeploymentMode from the controller config. There is no admission error to alert you, so audit for the annotation before upgrading.

The In-Repo Web Console Is Gone (#686)

The web console no longer lives in this repository; deployment moves to ome-console. This is the single largest change in the release at ~27.3k deletions across 155 files — the entire web-console/ tree, frontend and backend.

  • Repo plumbing is cleaned up with it: the web-console / frontend / backend issue labels and labeler rules are removed, and the Pages workflow no longer builds the console
  • The npm-lockfile pre-commit hook added in 1.2.0 (#667) is retargeted rather than dropped — it now guards site/package-lock.json instead of web-console/frontend/package-lock.json

Race-Safe Model-Agent Reconciliation (#662, #680)

Two independent hardening passes on the model agent, continuing the deletion-race work from 1.2.0:

  • A deleted model generation can never be restored — per-model ConfigMap mutations are now fenced by model UID. A write that was already in flight when deletion began can no longer repopulate the status cache with a stale generation, and a fenced UID is never restored or updated again (#680)
  • Eviction and fencing are now distinct — node-local cleanup evicts the cache entry, because the same CR generation may legitimately become eligible for that node again; deletion fences the UID permanently. Conflating the two is what allowed resurrection (#680)
  • Centralized optimistic-concurrency retry — ConfigMap mutations funnel through one helper that re-reads the latest object and retries on both Conflict and AlreadyExists, replacing ad-hoc RetryOnConflict blocks (#680)
  • Real downloads no longer queue behind startup revalidation — the normal-priority queue splits into separate download and revalidation lanes, and startup revalidation replays are classified into the revalidation lane. A node restarting with many Ready models no longer makes genuinely new downloads wait behind the replay backlog (#662)

New Model Families and Capabilities (#666, #687, #689)

  • Kimi K3 — vllm-kimi-k3, a vLLM runtime for KimiK3ForConditionalGeneration safetensors checkpoints in the 1400B–1600B range, auto-selected at priority 1, 8 GPUs per replica, with MLA prefill via TRTLLM_RAGGED and prefix caching enabled (#687)
  • Inkling small NVFP4 — srt-inkling-small-nvfp4, an SGLang runtime for InklingForConditionalGeneration NVFP4 checkpoints in the 250B–300B range, auto-selected at priority 1, 8 GPUs per replica (#689)
  • VIDEO_TEXT_TO_TEXT capability — added to the ModelCapability enum, modelconfig, and the modelparser mapping, and the Nemotron H Nano rule is split so only nemotronh_nano_omni reports the omni capabilities (#666)

⚠️ Upgrading:

  • Migrate Serverless InferenceServices before upgrading. Set an explicit RawDeployment (or MultiNode) mode and replace any minReplicas: 0 with 1, since scale-to-zero no longer exists. A stale Serverless annotation is silently ignored, not rejected. The Knative Service reconciler is gone, so any ksvc OME previously created is no longer reconciled — clean those up by hand (#681).
  • Drop removed ingress config keys. localGateway, localGatewayServiceName, and knativeLocalGatewayService no longer exist; leaving them in a values file or ConfigMap has no effect (#681).
  • Nemotron H Nano capabilities change on re-parse. The rule no longer emits AUDIO_TO_TEXT for any Nemotron H Nano; non-omni models now report only IMAGE_TEXT_TO_TEXT + TEXT_TO_TEXT, and nemotronh_nano_omni reports AUDIO_TEXT_TO_TEXT + VIDEO_TEXT_TO_TEXT instead. Anything selecting these models by capability needs updating (#666).
  • VIDEO_TEXT_TO_TEXT needs the new CRDs. It extends a +kubebuilder:validation:Enum, so apply the updated CRDs before creating a BaseModel that carries it (#666).
  • Deploy the console from ome-console. The web-console/ tree is no longer in this repo and no console image is built from it (#686).

🚀 Features & Enhancements

  • Split the model-agent normal-priority queue into separate download and revalidation lanes so real downloads are not blocked behind startup revalidation replay (#662)
  • UID-fenced, retry-centralized model-agent ConfigMap reconciliation, with eviction and permanent fencing as distinct operations (#680)
  • VIDEO_TEXT_TO_TEXT model capability, plus a Nemotron H Nano rule split so only nemotronh_nano_omni reports omni capabilities (#666)
  • New vllm-kimi-k3 runtime for Kimi K3 (1400B–1600B, 8 GPUs, MLA prefill + prefix caching), auto-selected at priority 1 (#687)
  • New srt-inkling-small-nvfp4 SGLang runtime for Inkling small NVFP4 (250B–300B, 8 GPUs), auto-selected at priority 1 (#689)
  • Generated typed client, informer, and lister for AcceleratorClass via +genclient / +genclient:nonNamespaced, so it is reachable through the versioned clientset like every other API type (#688)
  • Refreshed Kimi K3 router and engine images to nightly-20260731-8fe4cdd and v1.9.0-vllm-kimi-k3 (#689)

🐛 Bug Fixes

  • Lock the Cargo registry and git caches (sharing=locked) in the model-agent Dockerfile, so the concurrent amd64 and arm64 build stages no longer corrupt a shared cache mount (#676)
  • Give each container image its own GitHub Actions cache scope, with ignore-error=true, so images no longer evict each other's layer cache in the dev-image and release workflows (#682)

♻️ Maintenance

  • Remove the in-repo web console and retarget the npm-lockfile pre-commit hook at site/ (#686)
  • Remove the Serverless (Knative) deployment mode and its documentation (#681)
  • Bump bundled image references to v1.2.1 across the Helm chart, the InferenceService ConfigMap, and the kustomize manager / model-agent overlays (#679)
  • Expand model-parser unit tests around quantization config parsing, covering the quant_method field and a realistic MoE exclude list (#684)

📦 Container Images

The following container images are available:

# OME Manager
docker pull ghcr.io/moirai-internal/ome-manager:v1.2.2

# Model Agent
docker pull ghcr.io/moirai-internal/model-agent:v1.2.2

# OME Agent
docker pull ghcr.io/moirai-internal/ome-agent:v1.2.2

# Multinode Prober
docker pull ghcr.io/moirai-internal/multinode-prober:v1.2.2

⎈ Helm Installation

Option 1: OCI Registry (Recommended)

# Install directly from OCI registry
helm install ome-crd oci://ghcr.io/moirai-internal/charts/ome-crd --version 1.2.2 --namespace ome --create-namespace
helm install ome oci://ghcr.io/moirai-internal/charts/ome-resources --version 1.2.2 --namespace ome

Option 2: GitHub Releases

# Add the OME Helm repository
helm repo add ome https://github.com/ome-projects/ome/releases/download/v1.2.2
helm repo update

# Install OME
helm install ome-crd ome/ome-crd --namespace ome --create-namespace
helm install ome ome/ome-resources --namespace ome

📋 Changelog

Full Changelog: v1.2.1...v1.2.2

📄 Software Bill of Materials

SBOMs are available in both SPDX and CycloneDX formats for:

  • Source code
  • All container images

🔐 Signatures

All container images are signed with cosign. Verify with:

cosign verify ghcr.io/moirai-internal/ome-manager:v1.2.2   --certificate-identity-regexp=https://github.com/ome-projects/ome/.github/workflows/release.yaml@refs/tags/.*   --certificate-oidc-issuer=https://token.actions.githubusercontent.com

What's Changed

New Contributors

Contributors

fredericsun, pallasathena92, and 2 other contributors
Assets 24
Loading