Skip to content

Kthena v1.0.0

Latest
Latest

Choose a tag to compare

View all tags
@LiZhenCheng9527 LiZhenCheng9527 released this 16 Jul 07:32
v1.0.0
2d67015
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Summary

We are excited to announce Kthena v1.0.0, a major milestone for Kubernetes-native LLM inference. This release focuses on production readiness across the serving stack: more accurate Gateway API routing, first-class role-level autoscaling for prefill/decode disaggregated workloads, safer role-level rolling updates, better router scheduling signals, session boost for multi-turn conversation workloads, richer cache-aware router observability with Prometheus metrics and example dashboards, and a more complete CLI experience.

Kthena v1.0.0 also includes an important autoscaling API consolidation. AutoscalingPolicyBinding has been removed, and target configuration now lives directly in AutoscalingPolicy through homogeneousTarget, heterogeneousTarget, and disaggregatedTarget.

Release Highlights

Key Features Overview

  • AutoscalingPolicy consolidation and P/D coordinated disaggregated autoscaling: AutoscalingPolicyBinding is removed, and autoscaling target configuration is consolidated into AutoscalingPolicy; disaggregatedTarget enables coordinated role-level autoscaling for prefill/decode workloads, allowing each role to scale from its own metrics while keeping P/D replica ratios within healthy bounds through optional ratio constraints.
  • Session boost for multi-turn conversations: The router can prioritize follow-up requests from recently completed sessions, improving the chance of reusing warm KV cache under concurrent agentic and chat workloads.
  • Router scheduling and observability: Per-pod in-flight request tracking, Redis-backed cross-router synchronization, configurable pod metrics scraping, and cache-aware Prometheus metrics improve scheduling accuracy and operational visibility.
  • Role-level rolling update availability control: Enhancements to the RoleRollingUpdate feature. Now each role can set maxUnavailable to control the pace of the upgrade.
  • Gateway API and HTTPRoute correctness: Kthena router now honors HTTPRoute hostnames, keeps matched route rules consistent for backend selection and URL rewrites, fixes PathPrefix semantics, and respects Gateway listener allowedRoutes.
  • CLI and OpenAI-compatible API improvements: The CLI now exposes richer status output and supports ModelRoute and ModelServer, while the router adds an OpenAI-compatible GET /v1/models endpoint.

AutoscalingPolicy Consolidation and P/D Coordinated Disaggregated Autoscaling

Kthena v1.0.0 introduces a simpler and more powerful autoscaling API. Users now configure what to scale, how to collect metrics, and scaling boundaries in a single AutoscalingPolicy resource.

The new disaggregatedTarget mode is designed for coordinated P/D autoscaling in role-based ModelServing deployments, especially prefill/decode disaggregated inference. Prefill and decode roles can make scaling decisions from their own metrics, while the autoscaler applies shared constraints so the two sides grow and shrink together instead of drifting independently. Each role can define its own replica range, metrics, and metric sources, and operators can configure a ratioConstraint to keep the P/D replica ratio within a healthy range.

Example disaggregatedTarget configuration:

spec:
  disaggregatedTarget:
    targetRef:
      apiVersion: workload.serving.volcano.sh/v1alpha1
      kind: ModelServing
      name: vllm-qwen-pd-ms
    roles:
      prefill:
        minReplicas: 1
        maxReplicas: 8
        metrics:
          - name: prefill_waiting_requests
            targetValue: "1"
        metricSources:
          prefill_waiting_requests:
            prometheus:
              serverURL: http://kube-prometheus-stack-prometheus.test.svc.cluster.local:9090
              query: sum(vllm:num_requests_waiting{namespace="autoscale-demo", service="vllm-prefill"})
      decode:
        minReplicas: 1
        maxReplicas: 16
        metrics:
          - name: decode_gpu_cache_usage
            targetValue: "0.75"
        metricSources:
          decode_gpu_cache_usage:
            prometheus:
              serverURL: http://kube-prometheus-stack-prometheus.test.svc.cluster.local:9090
              query: sum(vllm:gpu_cache_usage_perc{namespace="autoscale-demo", service="vllm-decode"})
    ratioConstraint:
      numeratorRole: prefill
      denominatorRole: decode
      minRatio: "0.25"
      maxRatio: "1"

Related changes:

Session Boost for Multi-Turn Conversation Workloads

Kthena v1.0.0 adds session boost support to improve multi-round chat, agentic workflows, and RAG chains where each request depends on previous responses. In these workloads, follow-up requests often reuse a large shared prefix. If they wait behind unrelated traffic for too long, corresponding backend KV cache entries may be evicted and time-to-first-token can increase.

Session boost lets the router track recently completed sessions and prioritize follow-up requests from those sessions in the waiting queue. The implementation keeps this behavior separate from user-fairness scheduling and uses dedicated session-boost configuration, including session header selection, bounded recent-session tracking, inflight admission limits, and an optional grace period for advanced cache-hit optimization.

The feature is designed to improve cache reuse opportunities under concurrent multi-turn traffic without claiming pod affinity by itself. For the strongest KV cache benefit, operators should also ensure that session-aware or cache-aware routing can place follow-up requests on backends that still hold the warm cache.

Example Helm configuration:

networking:
  kthenaRouter:
    sessionBoost:
      enabled: true
      header: X-Session-ID
      maxSessions: 4096
      inflightPerPod: 16
      gracePeriod: 0s

Related changes:

Smarter Router Scheduling and Cache-Aware Observability

The router now has better load signals for scheduling decisions. Kthena tracks per-pod in-flight requests and can synchronize these counters across router replicas through Redis, allowing the least-request plugin to make decisions based on real-time load instead of only local router state.

Cache-aware scheduling is also much more observable. The prefix-cache and kvcache-aware score plugins now export Prometheus metrics on the router's existing /metrics endpoint, replacing klog-only visibility with queryable, model-labelled time series for load testing and production tuning.

For cache effectiveness, Kthena records match-ratio histograms instead of simple hit/miss counters. kthena_router_prefix_cache_match_ratio and kthena_router_kvcache_aware_match_ratio report the fraction of a prompt's blocks already available on the best-matching candidate pod, with 0 representing a real miss. This makes hit rate derivable from the le="0.0" bucket while also showing how much of the prefix was reused.

Related changes:

Role RollingUpdate Availability Control

Kthena v0.4.0 introduced RoleRollingUpdate, but role updates could still delete all outdated Role replicas in a ServingGroup at once. When spec.replicas was 1, this could make the service temporarily unavailable during a role-level rollout.

Kthena v1.0.0 adds per-Role maxUnavailable support for RoleRollingUpdate. Operators can now control the step size of each Role update independently, using either an absolute number or a percentage. This gives role-level rolling updates the same availability-oriented control that ServingGroup-level updates already had, while keeping Role-level and ServingGroup-level rollout budgets separate.

Example Role-level rollout configuration:

spec:
  rolloutStrategy:
    type: RoleRollingUpdate
  template:
    roles:
    - name: prefill
      replicas: 2
      maxUnavailable: 1
      # entryTemplate and workerTemplate omitted
    - name: decode
      replicas: 4
      maxUnavailable: 25%
      # entryTemplate and workerTemplate omitted

Related changes:

Gateway API and HTTPRoute Correctness

Kthena-router now handles Gateway API traffic with stronger correctness guarantees. The router honors HTTPRoute.spec.hostnames, keeps the matched HTTPRoute rule associated with backend selection and URL rewrite filters, and chooses more specific path rules within a route. This avoids accidentally routing a request through a backend or filter from a different rule than the one that matched the request.

This release also fixes Gateway API PathPrefix matching semantics and makes the router respect Gateway listener allowedRoutes before accepting HTTPRoutes.

Related changes:

CLI and OpenAI-Compatible API Improvements

The kthena CLI now surfaces more useful status information and supports more resource types:

  • kthena get model-servings now shows READY and STATUS columns.
  • kthena get model-boosters now shows a STATUS column.
  • kthena get model-routes and kthena get model-servers are now supported.
  • kthena describe model-route and kthena describe model-server are now supported.

The router also adds an OpenAI-compatible GET /v1/models endpoint, returning available model names in the standard list response shape.

Related changes:

Additional Enhancements

  • Added KEDA/HPA compatibility for ModelServing by setting the scale subresource label selector. #839
  • Added a controller-manager debug port for dumping cached ServingGroup and Role configuration. #900
  • Added SGLang Dynamo mocker coverage and SGLang inference simulator integration. #920, #1231
  • Added router pprof endpoint support. #1057
  • Added debugPort Helm chart support for controller-manager. #1032
  • Improved ModelBooster GPU and offline environment support. #972, #1141, #1146, #945
  • Added GPU usage plugin E2E coverage. #1199
  • Refreshed the quick-start documentation and recommends starting with ModelServing. #1260
  • Added DeepSeek-v4 model-serving examples. #936, #937
  • Added KV-cache-aware scheduler plugin documentation. #910

Stability and Correctness Highlights

  • Router request handling correctness: Fixes in JWT auth, empty request validation, streaming content-type parsing, retry behavior, mid-stream error propagation, IPv6 backend URLs, and HTTPRoute matching make the router more predictable under production traffic.
  • Gateway API reconciliation correctness: The router now respects HTTPRoute hostnames, matched rule selection, path-prefix semantics, and Gateway listener namespace admission rules, reducing mismatches between Gateway API configuration and runtime routing behavior.
  • Autoscaler and metrics reliability: Autoscaler metric collection now uses the target-referenced namespace first, checks HTTP response status before parsing metrics, stabilizes scale-down with the maximum sliding window, and supports bounded concurrent pod metric scraping.
  • Controller concurrency safety: ModelBooster and ModelServing controller fixes address data races, concurrent map writes, stale pod bindings, duplicate role deletion, and unsafe PodInfo access.
  • Webhook and deployment stability: Webhook certificate generation, cert-manager CA injection, webhook names, validator messages, and quick-start example defaults were corrected to reduce installation and onboarding failures.
  • Test and CI hardening: Additional unit and E2E coverage was added for controllers, autoscaler, router debug modules, ModelBooster lifecycle operations, SGLang PD routing, and status-aware scale-down behavior.

Bug Fixes

Router

  • Fixed router auth panic when the JWKS cache is empty. #1219
  • Required the Bearer scheme for JWT auth. #1035
  • Rejected empty router model requests. #1036
  • Corrected streaming content-type detection when parameters are present. #1145
  • Fixed retries sending empty bodies to fallback pods in the aggregated proxy path. #1031
  • Returned mid-stream and copy errors from proxy requests. #1049
  • Added IPv6 pod backend URL support. #1071
  • Fixed stale KV cache ownership. #1224
  • Removed an unnecessary goroutine in the ModelPrefixStore LRU eviction callback. #1243

Autoscaler and Metrics

  • Used the target-referenced namespace first in autoscaler metric collection. #1068
  • Checked HTTP status before parsing metrics responses. #1142
  • Used the maximum sliding window for autoscaler scale-down stabilization. #946
  • Used gauge values for SGLang metrics. #976
  • Treated zero TTFT/TPOT as uninitialized in least-latency scoring. #1040
  • Used ModelServer's configured workload port for metrics instead of hardcoded defaults. #1205

ModelServing and ModelBooster Controllers

  • Used controller owner references for ModelBooster children. #1054
  • Fixed a ModelBoosterController data race and concurrent-map-write panic. #1085
  • Fixed stale ModelServer pod bindings. #1126
  • Fixed concurrent grace-map access in error-pod handling. #1157
  • Guarded PodInfo access against data races. #1167
  • Avoided duplicate role deletion while a role is already deleting. #1269
  • Fixed ModelServing webhook panic on missing replicas. #1055
  • Synced PodGroup NetworkTopology when ModelServing topology changes. #1088

Connectors and PD Paths

  • Rebuilt NIXL prefill/decode request bodies on every proxy call. #947
  • Rebuilt SGLang prefill/decode request bodies on every proxy call. #984
  • Added streaming error handling improvements. #1236

Helm, Webhooks, and Examples

  • Fixed cert-manager CA injection annotation for controller-manager webhooks. #1018
  • Used random serial numbers for webhook certificates. #1160
  • Corrected webhook names and validator messages. #1152, #1159
  • Fixed quick-start ModelBooster cacheURI and Hugging Face endpoint examples. #1246

API Changes and Upgrade Notes

Breaking Change: AutoscalingPolicyBinding Removed

AutoscalingPolicyBinding has been removed from CRDs, generated clients, informers, listers, apply configurations, and Helm chart embedded CRDs.

Users should migrate autoscaling target configuration into one of the following AutoscalingPolicy.spec fields:

  • homogeneousTarget
  • heterogeneousTarget
  • disaggregatedTarget

For further details, please refer to the CRD documentation and the Proposal.

New ModelBooster and Router Configuration

  • ModelBackend.runtimeClassName configures the Kubernetes RuntimeClass used by generated backend pods.
  • ModelWorker.tolerations configures worker pod tolerations.
  • KTHENA_SKIP_ENGINE_DEPENDENCY_INSTALL=true skips startup-time connector dependency installation for offline or prebuilt engine images.
  • METRICS_SCRAPE_INTERVAL controls the router pod metrics scrape interval.
  • Controller-manager Helm values now support debugPort.

Build Environment

The project toolchain, Dockerfiles, CI workflows, and development documentation have been upgraded to Go 1.26.4. See feat: comprehensive upgrade to Go 1.26.4 #1244.

Tests, Docs, and Infrastructure

Kthena v1.0.0 includes broad test, documentation, and infrastructure improvements:

  • Added unit tests for ModelRouteController and GatewayController. #992
  • Added E2E coverage for updating and deleting ModelBooster. #1000
  • Added E2E coverage for status-aware scale-down behavior. #982
  • Added SGLang PD router E2E coverage. #994
  • Added autoscaler, config, and router debug unit tests. #903
  • Fixed flaky controller and router tests. #950, #1102, #1162
  • Used the PodGroup informer cache when listing existing PodGroups. #1081

Upgrade Instructions

To upgrade to Kthena v1.0.0 after the release is published:

1. Review Breaking API Changes

Before upgrading, review any existing autoscaling resources in your cluster. AutoscalingPolicyBinding is removed in v1.0.0, so clusters using the old two-resource autoscaling model must migrate to the new single-resource AutoscalingPolicy model.

Check whether your cluster still has old binding resources:

kubectl get autoscalingpolicybindings.workload.serving.volcano.sh --all-namespaces

If any resources are returned, migrate their target and metric-source configuration into one of the following AutoscalingPolicy.spec fields before or during the upgrade:

  • homogeneousTarget
  • heterogeneousTarget
  • disaggregatedTarget

For prefill/decode disaggregated workloads, use disaggregatedTarget.roles and, when needed, disaggregatedTarget.ratioConstraint.

2. Back Up Existing Kthena Resources

Back up Kthena custom resources before applying the new CRDs and controllers:

kubectl get modelservings.workload.serving.volcano.sh --all-namespaces -o yaml > modelservings-backup.yaml
kubectl get modelboosters.workload.serving.volcano.sh --all-namespaces -o yaml > modelboosters-backup.yaml
kubectl get autoscalingpolicies.workload.serving.volcano.sh --all-namespaces -o yaml > autoscalingpolicies-backup.yaml
kubectl get modelroutes.networking.serving.volcano.sh --all-namespaces -o yaml > modelroutes-backup.yaml
kubectl get modelservers.networking.serving.volcano.sh --all-namespaces -o yaml > modelservers-backup.yaml

If you still have AutoscalingPolicyBinding resources, back them up before removing or migrating them:

kubectl get autoscalingpolicybindings.workload.serving.volcano.sh --all-namespaces -o yaml > autoscalingpolicybindings-backup.yaml

3. Upgrade Kthena

Using Helm

For OCI-based Helm installation from GHCR:

helm upgrade kthena oci://ghcr.io/volcano-sh/charts/kthena \
  --version v1.0.0 \
  --namespace kthena-system

If this is a fresh installation instead of an upgrade:

helm install kthena oci://ghcr.io/volcano-sh/charts/kthena \
  --version v1.0.0 \
  --namespace kthena-system \
  --create-namespace

If you install from the release chart package:

curl -L -o kthena.tgz https://github.com/volcano-sh/kthena/releases/download/v1.0.0/kthena.tgz
helm upgrade kthena kthena.tgz --namespace kthena-system

4. Verify the Upgrade

Check that all Kthena components are running:

kubectl get pods -n kthena-system
kubectl get svc -n kthena-system
kubectl get crd | grep serving.volcano.sh

Verify workload, networking, and autoscaling resources:

kubectl get modelservings.workload.serving.volcano.sh --all-namespaces
kubectl get modelboosters.workload.serving.volcano.sh --all-namespaces
kubectl get autoscalingpolicies.workload.serving.volcano.sh --all-namespaces
kubectl get modelroutes.networking.serving.volcano.sh --all-namespaces
kubectl get modelservers.networking.serving.volcano.sh --all-namespaces

Upgrade Notes

  • AutoscalingPolicyBinding is removed. Migrate to AutoscalingPolicy.spec.homogeneousTarget, spec.heterogeneousTarget, or spec.disaggregatedTarget before relying on v1.0.0 autoscaling behavior.
  • disaggregatedTarget supports role-level autoscaling and optional role ratio constraints. Fixed roles can use minReplicas == maxReplicas.
  • Existing ModelBooster behavior remains backward compatible by default. Set KTHENA_SKIP_ENGINE_DEPENDENCY_INSTALL=true only when using offline or prebuilt engine images that already include connector dependencies.
  • GPU clusters that require custom runtimes can set ModelBackend.runtimeClassName; tainted GPU nodes can be targeted with ModelWorker.tolerations.
  • Router pod metrics scraping can be tuned with METRICS_SCRAPE_INTERVAL. Keep the default unless you need fresher scheduling signals or lower scrape overhead.
  • Development and build prerequisites now use Go 1.26.4; this affects contributors and image builders, not normal Helm or manifest-based runtime upgrades.

Thank You, Contributors

Thank you to everyone who contributed to Kthena v1.0.0 across controllers, router, autoscaler, CLI, Helm charts, examples, docs, CI, generated clients, and tests.

Special thanks to contributors including @Abirdcfly, @Alivestars04, @anirudh240, @avinxshKD, @blenbot, @FAUST-BENCHOU, @hzxuzhonghu, @JagjeevanAK, @katara-Jayprakash, @kube-gopher, @LiZhenCheng9527, @madmecodes, @nabrahma, @nXtCyberNet, @rajnish-jais, @Sanchit2662, @verma-garv, @WHOIM1205, @xrwang8, @zhy76, and @YaoZengzeng.

We warmly invite developers, operators, and AI infrastructure teams to try Kthena v1.0.0 and help shape the next generation of cloud native LLM serving.

What's Changed

Exciting New Features 🎉

Bug fixes 🐛

  • Replace the kubeclient.Update method in autoScaler with kubeclient.Patch by @LiZhenCheng9527 in #915
  • test: wait for router webhook readiness by @acsoto in #930
  • fix(autoscaler): use maximum sliding window for scale-down stabilization by @kube-gopher in #946
  • fix(nixl): rebuild prefill/decode request bodies on every Proxy call by @Sanchit2662 in #947
  • fix: replace testify/assert/yaml with sigs.k8s.io/yaml in scheduler p… by @anirudh240 in #967
  • fix: use GetGauge() instead of GetCounter() for sglang metrics by @anirudh240 in #976
  • fix(router): retry sends empty body to fallback pods in aggregated proxy path by @Sanchit2662 in #1031
  • fix(autoscaler/metric_collector): exit decode loop on non-EOF error by @Kernel-9 in #952
  • fix: require bearer scheme for JWT auth by @pm-ju in #1035
  • fix: reject empty router model requests by @pm-ju in #1036
  • test/e2e: fix flaky TestKthenaRouterValidatingWebhook caused by webhook race after pod restart by @nXtCyberNet in #1065
  • model-booster: fix data race and concurrent-map-write panic in ModelBoosterController by @kube-gopher in #1085
  • fix: use controller owner refs for modelbooster children by @pm-ju in #1054
  • fix(autoscaler/metric_collector): Use the target-referenced namespace first by @Kernel-9 in #1068
  • remove unused GetModelRoutesByGateway by @anirudh240 in #1078
  • fix: support ipv6 pod backend URLs by @pm-ju in #1071
  • Fix HTTPRoute PathPrefix matching by @Monti-27 in #1119
  • fix(router): return mid-stream and copy errors from proxyRequest by @Sanchit2662 in #1049
  • metrics: check HTTP status before parsing response by @Sri-Varshith in #1142
  • test(e2e): add self-healing coverage for ModelBooster by @katara-Jayprakash in #1143
  • fix(model-booster): wire ModelWorker.Affinity to pod spec templates by @Abirdcfly in #1146
  • Fix concurrent graceMap check/store race in handleErrorPod by @nXtCyberNet in #1157
  • fix: guard PodInfo access against data races by @zhy76 in #1167
  • Fix router auth panic when JWKS cache is empty by @avinxshKD in #1219
  • Validate ModelRoute rule fields by @avinxshKD in #1201
  • fix: remove unnecessary goroutine in ModelPrefixStore LRU eviction callback by @nabrahma in #1243
  • fix(scheduler): treat zero TTFT/TPOT as uninitialized in least-latency plugin by @Sanchit2662 in #1040
  • fix: aggregate labeled autoscaler metrics by @pm-ju in #1064
  • Fix stale KV cache ownership by @avinxshKD in #1224
  • fix(router): respect Gateway allowedRoutes by @avinxshKD in #1263
  • fix: parallelize pod metrics scraping loop with bounded concurrency by @nabrahma in #1255
  • Fixed an issue where a PG was created by mistake when scaling up a Se… by @LiZhenCheng9527 in #1268

Documentation Updates 📚

Other Changes 🔄

New Contributors

Full Changelog: v0.4.0...v1.0.0

Contributors

Abirdcfly, lx1036, and 34 other contributors
Assets 9
Loading