Skip to content

v0.8.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 13 Oct 00:06
· 30 commits to main since this release
v0.8.0
453ab36
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

AIBrix v0.8.0

We’re excited to announce AIBrix v0.8.0! 🚀

This release includes 443 commits from 100+ contributors, with major improvements across model lifecycle management, inference routing, multi-node serving, autoscaling, LoRA, batch inference, and production reliability.

For a deeper look at the release:

Note: ModelClaim and ModelWarmup are experimental features in v0.8.0.

Highlights

Model Lifecycle & Orchestration

  • ModelClaim (Experimental): Decouple model lifecycle from Pod lifecycle and enable multiple models to share GPU runtimes.
  • ModelWarmup (Experimental): Prepare nodes ahead of workload placement by preloading container images and running custom preparation actions.
  • StormService: Improved multi-node inference lifecycle management, rollout, scale-in, failure handling, and Volcano integration.

Inference Routing

  • Least-Request-Top-K: Better request distribution across replicas in distributed Gateway deployments.
  • Per-Replica Capacity: RPS and in-flight capacity-aware routing with aggregate capacity that adjusts as deployments scale.
  • Session Affinity: Redis-backed session pinning across Gateway replicas.
  • External Routing (Experimental): Support for integrating custom routing and scheduling logic.

Autoscaling & Serving

  • Scheduled replica bounds and predictive autoscaling APIs.
  • Improved autoscaling correctness and metric handling.
  • Improved LoRA and ModelAdapter management.
  • More elastic and resilient batch inference.
  • Continued improvements to heterogeneous GPU load balancing.

Reliability & Developer Experience

  • Additional Gateway and P/D failure handling.
  • Improved distributed routing state and model discovery.
  • Runtime and model artifact reliability improvements.
  • Expanded integration, E2E, and installation testing.
  • Improved documentation, Helm charts, samples, and installation workflows.

Thank you to everyone in the AIBrix community who contributed code, reviews, testing, documentation, and feedback to this release!

What's Changed ## What's Changed * fix: correct batch status transitions in planner poll loop by @Jeffwan in https://github.com//pull/2374 * chore(docs): Update prefix cache doc by @xiedeyantu in https://github.com//pull/2372 * Add batch replica resource request by @Jeffwan in https://github.com//pull/2375 * Add job-level batch client config controls by @Jeffwan in https://github.com//pull/2378 * feat(batch): support configurable completion windows by @Jeffwan in https://github.com//pull/2376 * Storage small file multi-upload optimization for large batch job finalization by @zhangjyr in https://github.com//pull/2379 * Expose batch client settings in Console create-job UI by @Jeffwan in https://github.com//pull/2380 * feat: restrict routing to least-loaded pods under load imbalance in preble algorithm by @xiedeyantu in https://github.com//pull/2382 * fix(kvcache): decode lora_id/medium/lora_name/group_idx in BlockStored (#2285) by @pjdurden in https://github.com//pull/2384 * fix(console): filter out extra_body from UI metadata by @DwyaneShi in https://github.com//pull/2386 * [Bug] Propagate Redis password to KubernetesJob batch worker by @V-3604 in https://github.com//pull/2366 * [Bug] add v1/responses to reserved-router by @jan-stanek in https://github.com//pull/2390 * [Misc] Refactor simplify client.MatchingLabelsSelector as client.MatchingLabels by @zedongh in https://github.com//pull/2381 * feat(batch): Add new GPU package for model templates by @scarlet25151 in https://github.com//pull/2388 * feat(autoscaler): support Kubernetes external metrics by @googs1025 in https://github.com//pull/2367 * fix(gateway): report async SGLang prefill outcomes by @googs1025 in https://github.com//pull/2391 * fix(pd): tracker registration ordering, prefill executor cleanup, decode scorer comment fix by @varungup90 in https://github.com//pull/2394 * fix(gateway): move LLM engine validation before router selection for PD, fix httproute error check by @varungup90 in https://github.com//pull/2397 * [Bug]Modify model of output payload to keep consistent with input by @zhangjyr in https://github.com//pull/2398 * feat(console): add metrics component by @DwyaneShi in https://github.com//pull/2399 * feat(console): add key metrics by @DwyaneShi in https://github.com//pull/2400 * fix(mock): support Flask-HTTPAuth 4.8.1 auth flow by @googs1025 in https://github.com//pull/2393 * feat(gateway): add conductor algorithm as prefill/decode score policy for PD router by @CarolWinddd in https://github.com//pull/2387 * [Misc] Strengthen batch execution with recovery-aware runtimes and atomic state updates by @zhangjyr in https://github.com//pull/2360 * Fix StormService scale-in ordering for not-ready RoleSets by @googs1025 in https://github.com//pull/2405 * feat(batch): batch grafana dashboard by @DwyaneShi in https://github.com//pull/2403 * fix(helm): start built-in Redis master with --requirepass when password is set (#2396) by @Anai-Guo in https://github.com//pull/2407 * feat(batch): add sglang support by @scarlet25151 in https://github.com//pull/2410 * fix(batch): run jobs concurrently regardless of endpoint count by @Jeffwan in https://github.com//pull/2408 * fix(batch): align output/error file id exposure with OpenAI by @Jeffwan in https://github.com//pull/2409 * fix(kvevent): purge pod prefixes on AllBlocksCleared (#2287) by @pjdurden in https://github.com//pull/2385 * [Bug]Fixes batch job recovery issues around metadata-service restart. by @zhangjyr in https://github.com//pull/2411 * [Docs] Update LoRA dynamic loading sample by @Suyi32 in https://github.com//pull/2414 * fix(batch): probe request-locking support once to avoid per-request warning spam (#2332) by @Anai-Guo in https://github.com//pull/2412 * [Bug] Reflect KubernetesJob worker counts and usage on finalize by @V-3604 in https://github.com//pull/2404 * [Bug]Reset local worker counters when persisting runtime refs after restart by @zhangjyr in https://github.com//pull/2416 * [Misc]Harden batch status-copy key encoding across storages by @zhangjyr in https://github.com//pull/2415 * fix(autoscaler): reject HPA role subtargets by @googs1025 in https://github.com//pull/2418 * fix(gateway): keep request stats bound to original pod by @googs1025 in https://github.com//pull/2424 * [Bug]Batch: harden runtime lifecycle recovery, deregister stale runtime refs, and improve interruption control by @zhangjyr in https://github.com//pull/2427 * feat(batch): support job deployment detail by @DwyaneShi in https://github.com//pull/2429 * [Bug]Batch: Fix recovery cleanup for recovered active batch jobs staged in pending by @zhangjyr in https://github.com//pull/2430 * fix(batch): pass context to deployment detail provider by @DwyaneShi in https://github.com//pull/2432 * fix(batch): add infrastructure_context to app state by @DwyaneShi in https://github.com//pull/2433 * feat(gateway-plugin): support bearer token auth and chart config by @NelZyhh in https://github.com//pull/2428 * Fix RayClusterReplicaSet scale-down ordering by @googs1025 in https://github.com//pull/2426 * feat(console): add matching order timeline and provision detail by @DwyaneShi in https://github.com//pull/2437 * feat(pd): roleset eligibility extraction, load-imbalance fixes, and tracker registration ordering by @varungup90 in https://github.com//pull/2395 * Support in-place role image rollout by @googs1025 in https://github.com//pull/2402 * docs: pod inplace image docs updates by @googs1025 in https://github.com//pull/2438 * fix(gateway): support multimodal content-parts input for embeddings requests by @varungup90 in https://github.com//pull/2443 * experiment api: add ModelClaim CRD by @Jeffwan in https://github.com//pull/2439 * [Bug]Batch: persist runtime ownership to recover safely during MDS rolling updates by @zhangjyr in https://github.com//pull/2436 * feat: route ModelClaim models by @Jeffwan in https://github.com//pull/2446 * feat: add ModelClaim runtime foundation by @Jeffwan in https://github.com//pull/2447 * fix(modelclaim): validate fixed GPU pool topology by @Jeffwan in https://github.com//pull/2448 * [Misc]Batch: skip expiry checks for jobs already in FINALIZING by @zhangjyr in https://github.com//pull/2445 * fix(types): reset traceAdded when recycling RoutingContext from the pool by @Anai-Guo in https://github.com//pull/2442 * Fix console batch status sync by @Jeffwan in https://github.com//pull/2383 * feat(console): add create model page by @DwyaneShi in https://github.com//pull/2454 * feat(modelclaim): harden pooled runtimes and add KV policy loop by @Jeffwan in https://github.com//pull/2453 * fix(batch): stabilize adaptive concurrency under error bursts by @scarlet25151 in https://github.com//pull/2452 * fix(modelclaim): enforce single replica contract by @Jeffwan in https://github.com//pull/2458 * [Misc]feat(modelclaim): surface invalid pool policy configuration to operators by @FAUST-BENCHOU in https://github.com//pull/2462 * feat(modelclaim): add Grafana dashboard for high-density runtime pools by @Shaurya2k06 in https://github.com//pull/2461 * feat(brixbench): add workload PodMonitor setup by @JinKim48 in https://github.com//pull/2431 * fix(cache): decrement pendingRequests exactly once per request (#2440) by @Anai-Guo in https://github.com//pull/2463 * feat(modelclaim): add automatic residency lifecycle policy by @Jeffwan in https://github.com//pull/2460 * [Docs] Add ModelClaims guidance by @Jeffwan in https://github.com//pull/2466 * docs: update AI Gateway integration samples by @googs1025 in https://github.com//pull/2467 * [Misc] feat(observability): chart pool policy validity on ModelClaim dashboard by @ecgang in https://github.com//pull/2465 * [API][Misc] Add provider-backed Console deployment creation with Kubernetes implementation by @scarlet25151 in https://github.com//pull/2202 * docs: fix broken license links in python READMEs by @io-wy in https://github.com//pull/2470 * docs: fix typo in distributed tutorial README by @io-wy in https://github.com//pull/2471 * feat(batch): enhance error logging by @DwyaneShi in https://github.com//pull/2472 * feat(modelclaim): add operational metrics for pooled runtimes by @FAUST-BENCHOU in https://github.com//pull/2464 * feat(batch): add OpenAI frontend and ViPE engine impl by @DwyaneShi in https://github.com//pull/2473 * fix(batch): vipe engine uses explicit storage type by @DwyaneShi in https://github.com//pull/2480 * fix(docs): update outdated master branch reference in CONTRIBUTING.md by @arijitroy003 in https://github.com//pull/2486 * Remove deprecated exportloopref linter and pin CI action by @arijitroy003 in https://github.com//pull/2488 * [Bug]: clean requests buffer when request exit by @vie-serendipity in https://github.com//pull/2487 * fix(gateway): measure streaming TTFT/KV-transfer from first chunk (fixes #2477) by @Anai-Guo in https://github.com//pull/2481 * feat(gateway): support opaque session affinity keys by @catyans in https://github.com//pull/2485 * docs(aws): Neuron (Trainium) DRA device allocation for disaggregated inference by @yahavb in https://github.com//pull/2490 * feat(gateway): add engine-agnostic observability metrics and overhaul Grafana dashboard by @varungup90 in https://github.com//pull/2392 * ci: raise test coverage threshold-total to 45 by @varungup90 in https://github.com//pull/2491 * [Bug]preserve byte-level field order in SGLang PD prefill/decode bodies by @qidaye in https://github.com//pull/2476 * [Batch] increase replica limitation at batch to 1024 by @scarlet25151 in https://github.com//pull/2478 * feat(batch): add vllm schema extension to openai frontend by @DwyaneShi in https://github.com//pull/2479 * docs: fill TODO placeholders in CONTRIBUTING.md with community links by @arijitroy003 in https://github.com//pull/2489 * [Bug] Reserve space for service suffix so generated resource names stay within 63-char limit by @addielaruee in https://github.com//pull/2496 * fix(test): remove sync.Map iteration order dependency in TestUpdateModelReplicaMetrics by @varungup90 in https://github.com//pull/2499 * [test] Fix flaky gateway e2e tests by @Jeffwan in https://github.com//pull/2500 * [Batch] fix large size validation by @scarlet25151 in https://github.com//pull/2468 * [Batch] Fix runtime timeout setting by @scarlet25151 in https://github.com//pull/2492 * fix(console): fix provision timeline by @DwyaneShi in https://github.com//pull/2503 * fix(webhook): improve artifactURL unsupported schema error message by @io-wy in https://github.com//pull/2474 * [Feat] Add Dynamo provider support to brixbench by @whalepark in https://github.com//pull/2469 * feat(batch): enhance vipe engine by @DwyaneShi in https://github.com//pull/2509 * chore: more service discovery logging by @DwyaneShi in https://github.com//pull/2511 * [Bug] Refresh batch job progress by @Shubham-Rasal in https://github.com//pull/2508 * fix(benchmarks/client): honor per-request output_length from the workload (#2505) by @Anai-Guo in https://github.com//pull/2507 * [Bug] Complete Console deployment routing and Playground integration by @Jeffwan in https://github.com//pull/2501 * [Feat] Add llm-d provider support to brixbench by @JinKim48 in https://github.com//pull/2510 * [CLI] Add Agent session workload generator by @catyans in https://github.com//pull/2483 * [Docs] Fix incorrect script names, flags, and module path in gpu_optimizer README by @latent-9 in https://github.com//pull/2517 * [API]: feat: add Topology Policy for StormService by @googs1025 in https://github.com//pull/1846 * [Docs][CI] Enable EnvoyPatchPolicy via helm --set by @henrichter in https://github.com//pull/2521 * perf(gateway): Reduce streaming response parsing overhead by @rayne-Li in https://github.com//pull/2515 * [API] Add Kubernetes ModelAdapter management to Console by @scarlet25151 in https://github.com//pull/2514 * docs: add StormService topology policy guide by @googs1025 in https://github.com//pull/2523 * fix(test): use dynamic port in pd tests to avoid cross-package collisions by @Jonglee999 in https://github.com//pull/2525 * feat: support configurable PodAutoscaler metric windows by @googs1025 in https://github.com//pull/2522 * [Feat] Publish brixbench artifacts to TOS with aggregate CSV and prebuilt gateway option by @JinKim48 in https://github.com//pull/2524 * improve tracing logic and span name, add span attr by @rayne-Li in https://github.com//pull/2506 * feat: add Valkey as OSS metadata store option with external managed endpoint support by @atao2004 in https://github.com//pull/2434 * fix(test): make SimpleQueue concurrency tests deterministic by @Jonglee999 in https://github.com//pull/2526 * fix: sync requestID if it was overridden by traceparent header by @rayne-Li in https://github.com//pull/2527 * docs: document PodAutoscaler metric windows by @googs1025 in https://github.com//pull/2528 * [Bug] Drop hardcoded namespace from envoy sidecar ConfigMap by @henrichter in https://github.com//pull/2532 * bugfix: add path /v1/responses for aibrix-reserved-router by @rayne-Li in https://github.com//pull/2539 * fix(zed): scale down should not get stuck when not-ready pod exists by @zedongh in https://github.com//pull/2538 * [Bug] Fix helm unit-tests by @linolayani in https://github.com//pull/2542 * [Misc] avoid logging full request payloads in debug logs by @Jeffwan in https://github.com//pull/2545 * fix(batch): reduce inference max retries from 120 to 5 by @Jeffwan in https://github.com//pull/2547 * fix(zed): scale down should not get stuck when not-ready pod exists by @zedongh in https://github.com//pull/2549 * fix(console): pin the DB driver's timestamp zone to UTC by @Jeffwan in https://github.com//pull/2556 * fix(batch): keep smart client endpoints alive under transport errors by @Jeffwan in https://github.com//pull/2555 * fix(console): use runtime batch deadline by @Jeffwan in https://github.com//pull/2558 * fix(console): progressively load batch jobs by @Jeffwan in https://github.com//pull/2559 * feat(gateway-plugin): add lockedRoutingStrategy to pin per-model routing strategy by @Yang1032 in https://github.com//pull/2544 * fix: stormservie roleset stateful pod name shorten if exceed by @zedongh in https://github.com//pull/2563 * [Batch] fix: use dedicated file upload timeout by @scarlet25151 in https://github.com//pull/2504 * Fix batch jobs losing prepared output file IDs when refresh replays an older snapshot by @Jeffwan in https://github.com//pull/2564 * [Bug] Honor per-request force_download override on S3/TOS downloads by @AmirF194 in https://github.com//pull/2566 * feat(batch): align jobs with provision windows by @DwyaneShi in https://github.com//pull/2575 * Close the mkstemp descriptor before unlinking in the batch driver test by @ErenAta16 in https://github.com//pull/2572 * [Bug] Mark interrupted benchmark streams as errors by @daixinwang in https://github.com//pull/2567 * Support scheduled replica bounds for PodAutoscaler by @googs1025 in https://github.com//pull/2529 * [Misc] Batch: Improve batch error context by @zhangjyr in https://github.com//pull/2554 * [Misc]Batch: Fix batch cancel conflict handling, batch error context, and finalized dataset downloads by @zhangjyr in https://github.com//pull/2553 * [Misc]Batch: Add metrics support for mds. by @zhangjyr in https://github.com//pull/2543 * docs(samples): add health and readiness probes to AI gateway examples by @gegemeimingzi in https://github.com//pull/2582 * docs(samples): add health and readiness probes to volcano-engine examples by @gegemeimingzi in https://github.com//pull/2581 * [Docs] Add multi-LoRA inference example by @Suyi32 in https://github.com//pull/2580 * fix(console): prevent queued jobs from being skipped indefinitely by @DwyaneShi in https://github.com//pull/2588 * feat(chart): add configurable gateway TLS support by @NelZyhh in https://github.com//pull/2586 * feat(gateway): add auto config profile selection by @googs1025 in https://github.com//pull/2570 * fix(gateway): Finalize request lifecycle once by @rayne-Li in https://github.com//pull/2585 * fix(chart): correct HTTPS gateway listener patch path by @NelZyhh in https://github.com//pull/2589 * feat(console): add provider-specific job resource config by @DwyaneShi in https://github.com//pull/2595 * feat(brixbench): Add brixbench routing topology comparison matrix by @JinKim48 in https://github.com//pull/2576 * [CLI] Support session key header injection for session-affinity replay by @tomsen02 in https://github.com//pull/2512 * [Misc] Share session-affinity header constants by @henriquejsza in https://github.com//pull/2599 * [API] Batch: add adaptive AIMD ramp-up controls by @scarlet25151 in https://github.com//pull/2530 * fix(gateway): stop double-wrapping upstream engine error responses by @Yang1032 in https://github.com//pull/2594 * [API] Validate RoleSet role specifications by @original4422 in https://github.com//pull/2584 * [Bug] stop KPA/APA scaling down on unknown metrics by @wieghx in https://github.com//pull/2592 * docs: document gateway target headers by @ANONYMOUSZED-beep in https://github.com//pull/2598 * [Bug] Reject unparseable metric targetValue in controller and surface HPA generation failures in status by @lettycat in https://github.com//pull/2593 * [API] Add explicit StormService deployment mode field by @V-3604 in https://github.com//pull/2450 * Docs: add Overview & Concepts page and navigation aids by @yaojiejia in https://github.com//pull/2607 * fix: inject Volcano scheduler for gang pods by @mikemikimike in https://github.com//pull/2621 * [Bug] Fix external metric fetching for GPU optimizer autoscaling and replace dead scale-down annotation in samples by @yaojiejia in https://github.com//pull/2616 * feat(metrics): add lora_adapter label to gateway request metrics by @varungup90 in https://github.com//pull/2623 * [Bug] Reject invalid Volcano gang scheduling configuration in StormService webhook by @yaojiejia in https://github.com//pull/2629 * [Cache] Back off pod metrics fetch failures by @googs1025 in https://github.com//pull/2635 * Surface Volcano gang scheduling status by @googs1025 in https://github.com//pull/2632 * [API]: Add historical node replacement scheduling for RoleSet by @googs1025 in https://github.com//pull/2628 * feat(gateway): add load-balance router and centralize the load-imbalance gate by @varungup90 in https://github.com//pull/2365 * [Misc] Test controller registration error paths by @askarshabdanov1-collab in https://github.com//pull/2639 * docs: document both multi-node inference orchestration paths by @DsThakurRawat in https://github.com//pull/2643 * [Bug] Derive cache service identity from an explicit flag, not EnableKVSync by @hamacekh in https://github.com//pull/2618 * fix(modeladapter): surface missing backend scheduling reasons by @mikemikimike in https://github.com//pull/2625 * Feat/brixbench dynamo v1.3.1 v1.4.0 by @JinKim48 in https://github.com//pull/2644 * [Docs] Add architecture, configuration and usage content to feature pages by @yaojiejia in https://github.com//pull/2622 * feat: add StormService progress deadline by @zupengwang in https://github.com//pull/2604 * docs: document PodAutoscaler support for Kubernetes external metrics by @DsThakurRawat in https://github.com//pull/2645 * fix(console): bound planner background work by @scarlet25151 in https://github.com//pull/2634 * [Misc] Add KVCache controller integration tests by @weichenxu923 in https://github.com//pull/2648 * [Feat] Consume StormService spec.mode in the controller update path and PodAutoscaler by @rishabhsinha17 in https://github.com//pull/2617 * [Bug] Fix benchmark prompts silently exceeding target token length by @pujitha24 in https://github.com//pull/2655 * [Bug] Fix spurious malformed JSON errors on split SSE chunks by @pujitha24 in https://github.com//pull/2653 * [Bug] Fail fast when direct-path LoRA adapter uses s3/gcs/tos URL by @pujitha24 in https://github.com//pull/2654 * [Bug] Allows for resumable downloads after a failure when loading model adapters. by @ashsmith in https://github.com//pull/2029 * ci: improve workflow check display by @googs1025 in https://github.com//pull/2656 * fix(modeladapter): fix stale status when base model pods disappear by @varungup90 in https://github.com//pull/2657 * [Misc] Add ModelRouter integration tests by @SarnadAbhilash in https://github.com//pull/2640 * [API]: Add drain-aware RoleSet rollout and scale-in by @googs1025 in https://github.com//pull/2651 * [Misc] Add RayClusterFleet integration tests by @yigitcan-ozturk in https://github.com//pull/2652 * [CI] Upgrade golangci-lint to v2.13.1 by @dongjiang1989 in https://github.com//pull/2630 * [Bug] Batch: Improve batch dispatch error capture by @zhangjyr in https://github.com//pull/2660 * [Bug] Batch: disable prometheus sink by default by @zhangjyr in https://github.com//pull/2614 * [Bug] Fix /v1/embeddings input limits the gateway cannot measure by @tomhol in https://github.com//pull/2664 * [Misc] Add unit tests for scaling algorithm factory and constraints by @stywzn in https://github.com//pull/2666 * [CI] Stabilize KVCache pod-triggered reconciliation tests by @yigitcan-ozturk in https://github.com//pull/2669 * [Misc] Add ModelAdapter controller integration tests by @SarnadAbhilash in https://github.com//pull/2665 * [Misc] Add unit tests for KPA panic mode and HPA algorithm by @stywzn in https://github.com//pull/2668 * Add pending replica guard for pod autoscaler by @googs1025 in https://github.com//pull/2671 * [Bug] Fix TestSonicSerializationWithRawMessage failing outside sonic's supported range by @stywzn in https://github.com//pull/2673 * docs: add StormService gang scheduling guide by @googs1025 in https://github.com//pull/2672 * [UI] Add UI for aiconfigurator tool by @CarolWinddd in https://github.com//pull/2649 * [Gateway] Restructure E2E test suites by @googs1025 in https://github.com//pull/2676 * [Bug][API] Make kubectl get modeladapter show a truthful status by @yaojiejia in https://github.com//pull/2624 * [Bug] Register PD prefill/decode selections atomically with the selection by @qidaye in https://github.com//pull/2678 * [Bug] Wire deployment detail and Playground to real data by @SarnadAbhilash in https://github.com//pull/2679 * [feat]support custom imagepullpolicy by @my-git9 in https://github.com//pull/2683 * [Misc] Add unit tests for pod key helpers and random pod selection by @stywzn in https://github.com//pull/2685 * [Gateway] Add token_load prefill score policy to the PD router by @qidaye in https://github.com//pull/2681 * [Gateway] Add hybrid_cache_load prefill policy and new-tokens charge to the PD router by @qidaye in https://github.com//pull/2682 * [Gateway] Keep crypto randomness and log formatting out of the PD selection critical section by @qidaye in https://github.com//pull/2691 * [Misc] Batch: Improve batch output persistence resilience during execution and finalization by @zhangjyr in https://github.com//pull/2659 * [Misc] Batch: Extend batch session retry handling with capped backoff to support more retry times. by @zhangjyr in https://github.com//pull/2658 * [Misc] Batch: Add HTTPX telemetry and call-site instrumentation. by @zhangjyr in https://github.com//pull/2684 * fix(batch): scale concurrency with live capacity by @scarlet25151 in https://github.com//pull/2633 * [Bug] Batch: validate scheduled jobs concurrently by @scarlet25151 in https://github.com//pull/2646 * [Docs] Fix api-key sample path in the LoRA guide by @Hiro208 in https://github.com//pull/2693 * [Gateway] Apply AIBRIX_MIN_MATCH_PCT to the prefix_cache prefill policy by @qidaye in https://github.com//pull/2689 * feat: add vLLM-Omni video generation support by @varungup90 in https://github.com//pull/2686 * [Bug] Stop scaling AIBRIX_TOKENIZER_* durations by time.Second twice by @Hiro208 in https://github.com//pull/2694 * [Docs] Add repository agent guidance by @googs1025 in https://github.com//pull/2695 * fix(telemetry): propagate PD routing traces by @rayne-Li in https://github.com//pull/2697 * [Gateway] Add strict PD mock contracts by @googs1025 in https://github.com//pull/2688 * [Bug] Reject non-positive durations in LoadEnvDuration by @Hiro208 in https://github.com//pull/2699 * [Misc] Add local AIBrix workflow bot by @googs1025 in https://github.com//pull/2696 * [Misc] Make AIBrix Bot PR validation advisory by @googs1025 in https://github.com//pull/2707 * [Bug] Fix Helm chart drift: missing modelclaims CRD, stale RBAC, and broken vedeployment webhook by @sanmuny in https://github.com//pull/2701 * [Misc] Add path-based PR labeler by @googs1025 in https://github.com//pull/2711 * [Misc] Add successful PD protocol contract E2E coverage by @googs1025 in https://github.com//pull/2709 * [Misc] Add PD failure E2E coverage by @googs1025 in https://github.com//pull/2712 * [Docs] Fix chat tokenizer examples for JSON message content by @lllleolin-max in https://github.com//pull/2713 * [TEST] Add Gateway Plugin integration regression coverage by @googs1025 in https://github.com//pull/2716 * fix(modeladapter): drop max loading retries cap by @jiangxiaobin96 in https://github.com//pull/2710 * [Bug] Fix running-request count drift from concurrent pod delete/re-add races by @varungup90 in https://github.com//pull/2717 * [Bug][CI] Reduce HTTPRoute cache and installation E2E flakes by @googs1025 in https://github.com//pull/2721 * [Bug] Preserve byte-level request body on vLLM and TRT-LLM PD prefill/decode paths by @qidaye in https://github.com//pull/2725 * [Bug] Scope BlockRemoved prefix eviction to the pod that sent it by @rishabhsai in https://github.com//pull/2723 * [Bug] Guard SLOQueue.queueRank against zero throughput by @stywzn in https://github.com//pull/2718 * [Bug] Prevent adaptive concurrency backoff overflow by @lllleolin-max in https://github.com//pull/2720 * feat: [API] Add per-replica inflight and RPS limits to config profiles by @varungup90 in https://github.com//pull/2726 * [Misc] Fix SetOutputPreditor typo in RoutingContext API by @stywzn in https://github.com//pull/2731 * [CI] Add installation smoke pipeline by @googs1025 in https://github.com//pull/2730 * feat(bot): add explicit Area field to Issue Forms, use as primary source by @Bryandero98 in https://github.com//pull/2729 * [CI][Misc] Add StormService and Volcano gang E2E coverage by @googs1025 in https://github.com//pull/2733 * [Feat] Route vLLM /tokenize requests through the gateway by @hamacekh in https://github.com//pull/2732 * [CI] Enable scheduled bounds and RoleSet drain E2E by @googs1025 in https://github.com//pull/2736 * [Misc] Add unit tests for SLOQueue rank path and SLO branch precedence by @stywzn in https://github.com//pull/2734 * [Feat] Add Redis-backed session-key pinning for cross-replica agentic routing by @varungup90 in https://github.com//pull/2742 * [Bug] Fix data race and stale snapshots in utils.Registry by @phakeandy in https://github.com//pull/2728 * [Misc] Add unit tests for SLOQueue Dequeue guard and Route error propagation by @stywzn in https://github.com//pull/2738 * [Misc] Add unit tests for SLOQueue featuresKey and higherRank by @stywzn in https://github.com//pull/2748 * [CI] Add ModelRouter lifecycle E2E coverage by @googs1025 in https://github.com//pull/2743 * [CI] Add ModelClaim controller integration coverage by @googs1025 in https://github.com//pull/2739 * [API] Reject StormService /scale above 1 in Pooled mode by @roshpr in https://github.com//pull/2661 * [TEST] Add streaming chunk-edge coverage to Gateway integration suite by @bolubo in https://github.com//pull/2750 * [Bug] Skip profile cache updates when Redis client is nil in standalo… by @ijayer in https://github.com//pull/2755 * [Gateway] Move power-of-two routing onto the shared running-requests counter by @varungup90 in https://github.com//pull/2749 * [API] Add async job registry and Videos routing by @czczycz in https://github.com//pull/2740 * [CI] Add RayClusterFleet lifecycle E2E coverage by @googs1025 in https://github.com//pull/2753 * [Gateway] Fail the client request and abort the decode leg when the SGLang PD prefill leg fails by @qidaye in https://github.com//pull/2746 * [Bug] Guard ModelGPUProfile.GetSignature against short indexes by @bolubo in https://github.com//pull/2756 * [CI] Add PodSet lifecycle E2E coverage by @googs1025 in https://github.com//pull/2751 * [Bug][CI] Wait for inference readiness in ModelAdapter all-pods-removed E2E by @bolubo in https://github.com//pull/2761 * [Bug] Keep ranked candidates in SLOQueue when a rank lookup fails by @bolubo in https://github.com//pull/2760 * [Misc] Fix misleading SLOQueue fallback log message by @bolubo in https://github.com//pull/2763 * [Bug] Use SET NX for the initial session-affinity Redis claim by @bolubo in https://github.com//pull/2765 * [CI] Add Gateway backend request E2E coverage by @czczycz in https://github.com//pull/2764 * [Misc] Add unit tests for the RoundRobin dispatch policy by @stywzn in https://github.com//pull/2752 * [Bug] Stop LocalStorage rewriting newlines in stored text objects by @stywzn in https://github.com//pull/2757 * [API] Add an authoritative model routing policy by @abe-servescale in https://github.com//pull/2767 * [API] Allow disabling gateway rate limiting while keeping Redis enabled by @abe-servescale in https://github.com//pull/2758 * [Bug] Gate the session-affinity Redis refresh on the stored value by @bolubo in https://github.com//pull/2768 * [Bug] Strip query string before matching request paths by @varungup90 in https://github.com//pull/2775 * [CLI] Make gateway-plugin Kubernetes API client QPS and burst configurable by @Hansburg183 in https://github.com//pull/2762 * [Feat] Recover panics on the gateway plugin request path by @bolubo in https://github.com//pull/2776 * [Misc] Add queue observability metrics to the gateway SLO queue by @bolubo in https://github.com//pull/2774 * [Gateway] Score load-balance on output-token rate, committed work, and KV pressure by @varungup90 in https://github.com//pull/2771 * [Bug] Recover panics in the queue router serve loop by @bolubo in https://github.com//pull/2781 * [CI] Cancel superseded pull-request workflow runs by @bolubo in https://github.com//pull/2777 * [Misc] Harden _write_json_file against newline translation by @stywzn in https://github.com//pull/2789 * [Misc] Cover the growing half of the RoundRobin channel-set spec by @stywzn in https://github.com//pull/2790 * [Feat] Drive gateway routing thresholds from model config profiles by @bolubo in https://github.com//pull/2786 * [Misc] Accept [Feat] in the PR title prefix check by @bolubo in https://github.com//pull/2793 * [Feat] Map the request priority tier to the vLLM request priority by @bolubo in https://github.com//pull/2787 * [Feat] Scope the VTC token tracker per profile by @bolubo in https://github.com//pull/2796 * [Misc] Pin the SLOQueue ordering contract and cover the policy switch matrix by @bolubo in https://github.com//pull/2797 * [CI] Serialize the poetry install in the python image by @bolubo in https://github.com//pull/2801 * [Feat] Accept workers and prefill/decode worker keys in the static config by @bolubo in https://github.com//pull/2803 * [Bug] Infer StormService mode from spec.replicas when spec.mode is unset by @btxu-db in https://github.com//pull/2791 * [Bug] Key PD router per-pod load state by namespace/name by @qidaye in https://github.com//pull/2785 * [Bug] Include canonicalized tools in the prefix-match text by @qidaye in https://github.com//pull/2784 * [Bug] Enforce tokenizer pool capacity during concurrent creation by @lllleolin-max in https://github.com//pull/2719 * [Bug] Scope StormService RoleSet lookups to the owning StormService by @btxu-db in https://github.com//pull/2792 * [CI] Serialize the poetry install in the Python tests workflow by @bolubo in https://github.com//pull/2810 * [Misc] Re-enable prefix cache indexer unit tests by @DsThakurRawat in https://github.com//pull/2815 * [Bug] Purge a pod's prefix cache when the engine sleep-state metric shows it went to sleep by @ankit373 in https://github.com//pull/2735 * [Feat] Answer 503 with Retry-After for a model whose ModelClaim is not placed yet by @Zeyu-ZEYU in https://github.com//pull/2809 * [Bug] Harden tokenizer pool capacity test and unlock before logging by @lllleolin-max in https://github.com//pull/2811 * [Bug] Hold KPA/APA replicas within the scale-up/scale-down cooldown windows by @pujitha24 in https://github.com//pull/2822 * [Feat] Add the predictive autoscaling API by @bolubo in https://github.com//pull/2823 * [Bug] Fix gateway rate-limiter window boundary by @qi-wutu in https://github.com//pull/2816 * [Bug] Include prompt fields in prefix matching by @1fanwang in https://github.com//pull/2813 * [Bug] Repin the session-affinity Redis pin on a bypassed final target by @pujitha24 in https://github.com//pull/2773 * [CI] Retry RoleSet updates on conflict in the topology policy spec by @hengguangcui11-cyber in https://github.com//pull/2828 * [Bug] Preserve shared model HTTPRoute on workload deletion by @googs1025 in https://github.com//pull/2831 * [Bug] Wait for the stats update winner before AddRequestCount returns by @hengguangcui11-cyber in https://github.com//pull/2833 * [Feat] Support TensorRT-LLM generation-first (parallel) P/D routing by @mrsibe in https://github.com//pull/2794 * [Bug] Key gateway rate history by namespace/name by @qidaye in https://github.com//pull/2802 * chart: Add Envoy diagnostic access log configuration by @rayne-Li in https://github.com//pull/2834 * [Bug] Chain each block of a multi-block BlockStored event from its predecessor by @ankit373 in https://github.com//pull/2820 * [Bug] Restore the S3 storage factory parameter test by @RainieLLM in https://github.com//pull/2839 * [Feat] Add adaptive bucket serving to the pd prefill routing by @bolubo in https://github.com//pull/2798 * [Bug] Key the prefix hash index by namespace/name by @qidaye in https://github.com//pull/2835 * [Bug] Key single-port pods as pod/port in the least-request DP path by @AlSh007 in https://github.com//pull/2832 * [Feat] Add token_load decode score policy to the PD router by @hengguangcui11-cyber in https://github.com//pull/2842 * [Bug] Key per-pod ports by namespace/name by @qidaye in https://github.com//pull/2836 * [Bug] Key PD per-request maps by namespace/name by @qidaye in https://github.com//pull/2837 * [Bug] Key the Preble tree and throughput pick by namespace/name by @qidaye in https://github.com//pull/2838 * [Bug] Reschedule ModelClaim after terminal engine failure by @Shubham-Rasal in https://github.com//pull/2829 * [Feat] Add external replica routing to Gateway by @googs1025 in https://github.com//pull/2841 * [Bug] Enforce SyncPrefixHashTable's maxContexts cap and evict expired pods by @AlSh007 in https://github.com//pull/2846 * [Misc] Remove FilterPodByName and match LPRadixCache pods by pod key by @qidaye in https://github.com//pull/2847 * [Bug] Detect the PD role from the role-name label when deriving token throughput by @hengguangcui11-cyber in https://github.com//pull/2851 * [Bug] Scrape static-discovery endpoints on their serving port by @hengguangcui11-cyber in https://github.com//pull/2855 * [Misc] Expand external router test coverage by @googs1025 in https://github.com//pull/2853 * Fix race in bucket-serve metric capture by @shuaiyafei in https://github.com//pull/2857 * [Misc] Re-enable RayClusterFleet reconcile spec by @Re1lya in https://github.com//pull/2862 * [Misc] Re-enable the ModelAdapter controller specs by @hmdsefi in https://github.com//pull/2861 * [Misc] Re-enable the RoleSet and StormService controller specs by @hmdsefi in https://github.com//pull/2864 * [Misc] Run the Redis rate limiter tests on miniredis by @hmdsefi in https://github.com//pull/2865 * [Misc] Test that WorkerPool.Stop returns while Submit calls race it by @hmdsefi in https://github.com//pull/2866 * [Feat] Count generated output in the token_load decode score by @hengguangcui11-cyber in https://github.com//pull/2852 * [Feat] Add elastic EP scaling state endpoints to the mocked vLLM app by @bolubo in https://github.com//pull/2869 * [Bug] Propagate the reconcile context to the ModelAdapter model lookup by @PDGGK in https://github.com//pull/2871 * [Feat] Route vLLM /pooling requests through the gateway by @sanmuny in https://github.com//pull/2778 * [Misc] Run the Kubernetes catalog test against fake clusters by @hmdsefi in https://github.com//pull/2870 * [Bug] Drop pool policy activity records of pods that stopped reporting by @ankit373 in https://github.com//pull/2885 * [Bug] Scale KPA on the total load, not the per-pod mean by @ankit373 in https://github.com//pull/2884 * [API] Add ModelWarmup image preloading by @googs1025 in https://github.com//pull/2769 * [Docs] Expand ModelClaim sample validation guide by @googs1025 in https://github.com//pull/2886 * [Bug] Stop NewJSONPatch from prepending empty operations by @hmdsefi in https://github.com//pull/2878 * [CI] Run ZMQ-tagged tests in CI by @czczycz in https://github.com//pull/2887 * [Bug] Make the gateway plugin's pprof address configurable and its bind failure non-fatal by @hengguangcui11-cyber in https://github.com//pull/2889 * [Bug] Keep prefix cache blocks dirty when they change during a delta push by @hmdsefi in https://github.com//pull/2877 * [Bug] Reject mismatched RayClusterFleet selectors by @1fanwang in https://github.com//pull/2722 * [Misc] Re-enable completion streaming e2e test by @czczycz in https://github.com//pull/2897 * [CI] Run integration-tagged controller tests in CI by @hmdsefi in https://github.com//pull/2896 * [Bug] Recreate missing model HTTPRoutes in ModelRouter by @btxu-db in https://github.com//pull/2873 * [API][Docs] Place a ModelClaim only where its card has room, and hold each engine to its share by @Zeyu-ZEYU in https://github.com//pull/2647 * [Bug] Fix TOS V1 download when part_chunksize is not set by @hmdsefi in https://github.com//pull/2895 * [Docs] Propose @googs1025 to lead autoscaling by @Jeffwan in https://github.com//pull/2900 * [UI] Batch: expose adaptive AIMD ramp-up controls by @Jeffwan in https://github.com//pull/2602 * [Docs] Hand over autoscaling maintainership to @googs1025 by @Jeffwan in https://github.com//pull/2901 * [Bug] Use the full pod selector when listing ModelAdapter pods by @btxu-db in https://github.com//pull/2875 * [CI] Skip code workflows for documentation-only changes by @czczycz in https://github.com//pull/2908 * [Bug] Validate ModelAdapter spec.podSelector in the webhook by @btxu-db in https://github.com//pull/2899 * [Gateway] Fail PD requests whose decode pod never starts responding by @qidaye in https://github.com//pull/2859 * [API][Feat] Add custom actions to ModelWarmup by @googs1025 in https://github.com//pull/2890 * [Bug] Use the full selector and owner check in RayClusterReplicaSet by @hmdsefi in https://github.com//pull/2893 * [Bug] Respect cancellation in acquired remote text tokenizers by @PDGGK in https://github.com//pull/2891 * [Feat] Divide a card's KV again as its engines and their load change by @Zeyu-ZEYU in https://github.com//pull/2843 * [Feat] Back off from claims that cannot be placed, and say when one can never fit by @Zeyu-ZEYU in https://github.com//pull/2844 * [Feat] Route a new engine within seconds of being ready by @Zeyu-ZEYU in https://github.com//pull/2845 * [Bug] Keep a runtime that does not answer from holding the ModelClaim controller by @Zeyu-ZEYU in https://github.com//pull/2818 * [Feat] Wake a sleeping model through the controller, and move one that cannot wake by @Zeyu-ZEYU in https://github.com//pull/2903 * [Feat] Share the token_load decode ledger across gateway replicas by @hengguangcui11-cyber in https://github.com//pull/2868 * [Feat][Docs] Add Jev-compatible external router sample by @googs1025 in https://github.com//pull/2910 * [Feat] Let a sleeping model release its memory reservation by @Zeyu-ZEYU in https://github.com//pull/2904 * [Feat] Make room for a new model by putting idle ones to sleep by @Zeyu-ZEYU in https://github.com//pull/2905 * [Feat] Route SGLang /v1/decisions requests through the gateway by @varungup90 in https://github.com//pull/2911 * [Feat] Make gateway model discovery optional and add ready-Pod listing by @abe-servescale in https://github.com//pull/2913 * [Bug] Avoid repeated Redis image pulls by @googs1025 in https://github.com//pull/2916 * [Feat] Route SGLang /v1/systemone requests through the gateway by @varungup90 in https://github.com//pull/2919 * [Feat] Add least-request-top-k routing and use it as the auto-blend load scorer by @varungup90 in https://github.com//pull/2918 * [Feat] Verify gateway model-list discovery on demand by @abe-servescale in https://github.com//pull/2915 * [Bug] Stop the concurrent-admission test failing on an expired Redis read budget by @PerryLink in https://github.com//pull/2920 * [Bug] Fix the ModelClaim problems found by a stress test by @Zeyu-ZEYU in https://github.com//pull/2922 * [Bug] Make room for ModelClaims in a fairer order and on multi-GPU pods by @Zeyu-ZEYU in https://github.com//pull/2923 * docs: fix broken relative links in docs by @AlvaroBalbin in https://github.com//pull/2687 * [Misc] Fix KV event replay wiring in the semantic-router samples by @begcdn in https://github.com//pull/2927 * [Misc] Expand ModelClaim integration test coverage by @googs1025 in https://github.com//pull/2928 * [Bug] Wait for the wall clock in TestLatestPodReadySince instead of assuming it advanced by @PerryLink in https://github.com//pull/2931 * [Misc] Expand ModelClaim end-to-end test coverage by @googs1025 in https://github.com//pull/2929 * [CI] Restore KV cache tests hidden by module skips by @zilinfg in https://github.com//pull/2930 * [Feat] Log the auto-blended routing strategy as resolved_strategy by @varungup90 in https://github.com//pull/2932 * [Gateway] Time the PD decode first response to its first token and cut silent streams by @qidaye in https://github.com//pull/2860 * [Gateway] Arm the PD decode watchdog after a non-terminal prefill failure by @qidaye in https://github.com//pull/2933 * [Bug] Apply replayed KV event batches instead of leaving them unread by @sohom-cs in https://github.com//pull/2907 * [Bug] Keep a waiting deactivate off the runtime's event loop by @Jeffwan in https://github.com//pull/2934 * [Bug] Keep autoscaler schedule transitions at local clock times by @JimmyWang0417 in https://github.com//pull/2937 * [Docs] Add Simplified Chinese catalogs and a navbar language switcher by @FAUST-BENCHOU in https://github.com//pull/2700 * [Feat] Report how many prefix blocks are cached by @hmdsefi in https://github.com//pull/2945 * [CI][Docs] Publish the ModelClaim kvcached runtime image by @Zeyu-ZEYU in https://github.com//pull/2939 * fix(kvcache): decode vLLM map-format KV events; tier-aware prefix scoring by @wuyan-zs in https://github.com//pull/2587 * [API] Rename the ModelClaim keys and cap claim names at 63 characters by @Zeyu-ZEYU in https://github.com//pull/2946 * fix: separate GPU optimizer deps to reduce runtime image size (#2863) by @Paramveersingh-S in https://github.com//pull/2872 * [Docs] Publish English and Chinese docs from one RTD project by @googs1025 in https://github.com//pull/2950 * [Docs] Rewrite the ModelClaim guide, samples and validation runbook by @Zeyu-ZEYU in https://github.com//pull/2956 * [CI] Cut avoidable waits from the installation E2E suite by @Jeffwan in https://github.com//pull/2953 * [Misc] Refactor ModelWarmup reconciliation around one snapshot by @googs1025 in https://github.com//pull/2935 * [Bug] Skip LRU eviction when the interval is not positive by @hmdsefi in https://github.com//pull/2944 * [Bug] Keep a one-block prefix match from scoring zero by @hmdsefi in https://github.com//pull/2943 * [Bug] Initialize the global router manager once by @hengguangcui11-cyber in https://github.com//pull/2951 * [Bug] Track KV block presence per tier and group in the prefix index by @Jeffwan in https://github.com//pull/2949 * [Bug] Fix runtime download status for nested and Hugging Face models by @Jackmeson1 in https://github.com//pull/2961 * [Bug] Fix flaky test_job_restore_after_mds_crash_during_in_progress (crash re-arms on restart) by @saylee21 in https://github.com//pull/2959 * [Bug] Keep a claim on a warm pod that leaves the pool by @Zeyu-ZEYU in https://github.com//pull/2955 * cut release v0.8.0-rc.1 by @varungup90 in https://github.com//pull/2963 * [Bug] Start a download for a model directory with no tracked files by @Jackmeson1 in https://github.com//pull/2967 * [Bug] Treat an omitted role replicas as zero in RoleSet status by @Jackmeson1 in https://github.com//pull/2968 * [Bug] Count assistant tool_calls in the chat routing text by @ptimizeroracle in https://github.com//pull/2964 * [Bug] Admit user RPM on the counter's incremented value by @Jackmeson1 in https://github.com//pull/2972 * [Bug] Report a missing latency from PendingLoadProvider.GetConsumption by @AlSh007 in https://github.com//pull/2975 * [Bug] Group PodArray pods by namespace and name by @AlSh007 in https://github.com//pull/2976 * [Bug] Enforce Reader.size_limiter on the bytes actually read by @saylee21 in https://github.com//pull/2977 * [Bug] Encode non-ASCII filenames in Files API response headers by @saylee21 in https://github.com//pull/2979 * [Bug] Skip nested folder markers in S3 and GCS artifact downloads by @Jackmeson1 in https://github.com//pull/2982 * [Bug] Disable the HPA scaling direction whose max rate is 1 by @Jackmeson1 in https://github.com//pull/2986 * [Bug] Validate RoleSet update budgets before running role syncers by @Jackmeson1 in https://github.com//pull/2984 * [Bug] Fix timezone-dependent overdue-job expiry test by @saylee21 in https://github.com//pull/3000 * [Bug] Accept array prompts for completions by @ANONYMOUSZED-beep in https://github.com//pull/2969 * [Bug] Raise ModelNotFoundError when an S3/TOS model source has no objects by @Jackmeson1 in https://github.com//pull/2988 * [Gateway] Reject PD routing for unsupported inference endpoints by @askarshabdanov1-collab in https://github.com//pull/2995 * [Bug] Count the deprecated function_call in the chat routing text by @ptimizeroracle in https://github.com//pull/2996 * [Bug] Add the Messages and Audio paths to the shared reserved route by @MannXo in https://github.com//pull/2997 * [Bug] Keep request bodies and prompts out of normal gateway logs by @MannXo in https://github.com//pull/2998 * [Feat] Route SGLang /v1/score requests through the gateway by @MannXo in https://github.com//pull/2999 * [Bug] Use external and domain totals in APA by @Jackmeson1 in https://github.com//pull/3002 * [Bug] Use supported keys in the APA samples by @Jackmeson1 in https://github.com//pull/3004 * chore(kvcache): bump build image to manylinux2_28-builder:cuda13.0 by @DwyaneShi in https://github.com//pull/2970 * [Docs] Reorganize feature docs by topic and refresh architecture pages by @varungup90 in https://github.com//pull/3005 * [Docs] Add overview diagrams to ModelClaim and ModelWarmup pages by @varungup90 in https://github.com//pull/3006 * [CI] Raise the Go coverage threshold to 55% by @varungup90 in https://github.com//pull/3007 * [Bug] Match the PodAutoscaler owner kind when mapping HPA events by @AarnavNoble in https://github.com//pull/3010 * [Bug] Meter streamed /v1/messages requests when usage has no total_tokens by @theminjunchoi in https://github.com//pull/3009 * [Bug] Route /v1/messages/count_tokens through the gateway by @theminjunchoi in https://github.com//pull/3012 * [Feat] Support listener-aware Envoy Service NodePorts by @NelZyhh in https://github.com//pull/3013 * [Bug] Update existing model HTTPRoutes that drift from their workload by @theminjunchoi in https://github.com//pull/3017 * [Bug] Keep fractional values from the custom metrics API by @AarnavNoble in https://github.com//pull/3018 * [Bug] Reject unusable autoscaling annotation values by @AarnavNoble in https://github.com//pull/3019 * [Bug] Release per-PodAutoscaler state when it is deleted by @AarnavNoble in https://github.com//pull/3020 * [Bug] Route vLLM /classify requests through the gateway by @theminjunchoi in https://github.com//pull/3015 * cut release v0.8.0-rc.2 by @varungup90 in https://github.com//pull/3022 * fix(kvcache): bump main cuda version to 13.0 by @DwyaneShi in https://github.com//pull/3025 * cut release v0.8.0 by @varungup90 in https://github.com//pull/3028

New Contributors

Full Changelog: v0.7.0...v0.8.0

Contributors

zhangjyr, sanmuny, and 99 other contributors
Assets 5
Loading