v0.3.0

This alpha release adds immediate push-based cold activation, a current NVIDIA A10 profile, and
expanded real-accelerator evidence for Hearth alongside Volcano and Kthena. The API remains
v1alpha1, and Hearth is not production-ready for shared or customer-facing workloads.

Added

• An opt-in, co-located KEDA ExternalScaler that streams cold activation immediately while retaining
  queue metrics for scale-out, scale-down, observability, and rollback.
• An activation lease for reject-mode cold starts, including current-state replay across scaler
  reconnects and a no-GPU E2E mode for the external-push path.
• An NVIDIA A10 profile and reproducible K3s validation report for vLLM v0.25.1.
• A hardware-neutral, command-driven Hearth and Kthena demo showing a hot model alongside a
  long-tail model that activates from zero and returns to zero.

Changed

• Helm and the manager accept gateway.scalerMode / --scaler-mode; metrics-api remains the
  compatibility default and external-push requires one gateway replica.
• The A100 example now uses device-specific names, vLLM v0.25.1, the positional model argument,
  and a conservative one-replica default. Its historical hardware evidence remains tied to vLLM
  v0.22.0 pending focused revalidation of the new stack.
• The primary quickstart now uses the physically validated A10 profile.

Removed

• The redundant standalone A100 HostPath manifest. cache.strategy: HostPath remains available in
  the API; bundled profiles consistently use NodeLocalPVC and require a dynamic StorageClass.

Fixed

• Use vLLM's current vllm:kv_cache_usage_perc metric in all runtime profiles, the CPU stub, and
  the optional Grafana dashboard.
• Correct the recorded Ascend 910B3 identity and tighten the evidence boundaries in the 910B3 and
  310P validation reports.
• Keep prerelease image publication from moving the stable latest operator and gateway tags.

Verified

• The v0.3.0-rc.1 release build completed the full Hearth lifecycle on two physical NVIDIA A10 GPUs
  with vLLM v0.25.1, KEDA external-push, prewarming, 0→1→2→0, bounded admission, reject mode,
  graceful drain, component replacement, and host reboot recovery.
• Volcano v1.15.0 enforced queue placement and quota on Kind, then scheduled the real A10
  workloads through separate queues and placed two Hearth replicas on distinct whole GPUs.
• A Kthena-managed hot model and a Hearth-managed long-tail model served concurrently on the same
  host. Hearth recovered automatically after reboot; the documented Kthena device-admission race
  required manual Pod replacement and remains outside Hearth's controller boundary.

---

🙏 Thanks to everyone who contributed to this release — the merged PRs and first-time
contributors are credited under What's Changed and New Contributors below.

Full Changelog: v0.2.0...v0.3.0

v0.3.0-rc.1

This release candidate adds immediate push-based cold activation and a physically validated NVIDIA
A10 profile while aligning bundled vLLM metrics and NVIDIA examples with current runtime behavior.
Hearth remains alpha software with a v1alpha1 API and is not production-ready.

Added

• An opt-in, co-located KEDA ExternalScaler that streams cold activation immediately while retaining
  queue metrics for scale-out, scale-down, observability, and rollback.
• An activation lease for reject-mode cold starts, including current-state replay across scaler
  reconnects and a no-GPU E2E mode for the external-push path.
• An NVIDIA A10 profile and reproducible K3s validation report for vLLM v0.25.1.

Changed

• Helm and the manager accept gateway.scalerMode / --scaler-mode; metrics-api remains the
  compatibility default and external-push requires one gateway replica.
• The A100 example now uses device-specific vllm-nvidia-a100 / qwen3-8b-a100 names, vLLM
  v0.25.1, the positional model argument, and a conservative one-replica default. Its historical
  hardware evidence remains tied to vLLM v0.22.0 pending focused revalidation of the new stack.

Removed

• The redundant standalone A100 HostPath manifest. cache.strategy: HostPath remains available in
  the API; bundled profiles consistently use NodeLocalPVC and require a dynamic StorageClass.

Fixed

• Use vLLM's current vllm:kv_cache_usage_perc metric in all runtime profiles, the CPU stub, and
  the optional Grafana dashboard.
• Correct the recorded Ascend 910B3 identity and tighten the evidence boundaries in the 910B3 and
  310P validation reports.
• Keep prerelease image publication from moving the stable latest operator and gateway tags.

Verified

• Two physical NVIDIA A10 GPUs completed the whole-device 0→1→2→0 lifecycle with real vLLM
  tokens, prewarming, cache persistence, backpressure, draining, recovery, and scale-down to zero.
• Volcano v1.15.0 on a three-node Kind cluster completed queue placement, quota enforcement, and
  Hearth's 0→1→0 path with the CPU stub and a fake extended resource. Real-accelerator topology,
  HAMi sharing, and gang scheduling remain separate validation work.

---

🙏 Thanks to everyone who contributed to this release — the merged PRs and first-time
contributors are credited under What's Changed and New Contributors below.

Full Changelog: v0.2.0...v0.3.0-rc.1

v0.2.0

Alpha release completing Hearth's first integrated Ascend hardware-validation cycle and hardening
the scale-to-zero control plane based on those results. Atlas 300I Duo is verified through the
multi-replica 0→1→2→0 lifecycle, while Ascend 910B3 is verified through the complete
single-device 0→1→0 lifecycle. Hearth remains v1alpha1 and is not production-ready.

Added

• A shared Ascend hardware-validation guide covering required images, validation levels, evidence,
  and separate 910B, Atlas 300I Duo, and Atlas 300I Pro result sets.
• Full physical 910B3 evidence for device-plugin scheduling, single-device 0→1→0, cold and warm
  inference, reject mode, bounded-queue backpressure, metrics, drain, self-heal, Helm upgrade, and
  reboot recovery.
• Full Atlas 300I Duo hardware evidence for the integrated 0→1→2→0 lifecycle, inference,
  backpressure, reject mode, drain, cache persistence, self-heal, Helm upgrade, and reboot recovery.
• Validation bounds for accelerator counts, scaling values, runtime ports, and termination grace
  periods.

Changed

• Prometheus Operator resources and RBAC are decoupled from core reconciliation. Hearth now exposes
  stable metrics discovery labels while optional ServiceMonitor and Grafana assets live under
  examples/observability/.
• Runtime metric fields are optional metadata for external observability; backend adapters and
  autoscaling no longer carry an unused runtime-metrics dependency.
• InferenceRuntime family and vendor enums now match the built-in implementations: vLLM on
  NVIDIA or Ascend. Manager RBAC is reduced to the CRD operations used by the passive runtime and
  service reconcilers.
• Ascend device profiles use equal priority, so clusters with multiple installed profiles must pin
  the intended hardware runtime instead of silently preferring 910B.
• Runtime and service examples now live in independently deployable, vendor/device-specific
  examples/ directories, with concise filenames that leave the device model to the directory.
• Prewarm Jobs inherit the runtime's node selector, tolerations, scheduler, and Volcano queue so
  node-local model data is prepared where the backend can run.
• LLMService resources are reconciled when a matching InferenceRuntime changes.
• The Ascend 910B runtime now invokes vllm serve explicitly and uses MindCluster's standard
  accelerator=huawei-Ascend910 node label. A dedicated hardware-validation service example uses a
  60-second drain, the shortest tested value that completed a live 910B3 stream without aborting.
• Ascend 310P examples invoke vllm serve explicitly and pin FP16 for the validated 310P3 path.

Fixed

• Keep retrying unchanged reconciliation failures, reject ambiguous equal-priority runtime
  selections, and reject Volcano queue names unless the Volcano scheduler is selected.
• Apply scaleDownStabilization to both KEDA's zero cooldown and HPA scale-down behavior, with
  validation for the HPA's one-hour limit.
• Build multi-platform images on the build host architecture and fail the docker-buildx target
  when the image build fails.
• Embed the release tag in operator and gateway images built by the release workflow.
• Publish the release chart only after both release images build successfully.
• Align Helm manager pod security and termination settings with the Kustomize deployment.
• Pin Kind and KEDA in E2E workflows, install Helm explicitly, and fail CI on uncommitted module or
  generated-file changes.
• Make repeated Podman scale-to-zero E2E runs replace and clean their temporary image archives.
• Reject unsupported BakedImage cache requests instead of silently rendering them as uncached.
• Reject invalid PVC claim names and model paths that could escape the mounted model volume.
• Prevent accelerator-free prewarm Pods from autoloading PyTorch vendor backends and requiring host
  driver libraries.
• Report the committed SSE 200 for keepalive activation timeouts and avoid classifying canceled
  clients as activation timeouts in gateway metrics.

---

🙏 Thanks to everyone who contributed to this release — the merged PRs and first-time
contributors are credited under What's Changed and New Contributors below.

What's Changed

🔧 Other Changes

• docs(crd): flag unimplemented fields by @ded-furby in #40

New Contributors

• @ded-furby made their first contribution in #40

Full Changelog: v0.1.0...v0.2.0

v0.2.0-rc.1

Pre-release documenting the first real-hardware bring-up of the Ascend 910B backend. vLLM-Ascend
now serves on a physical 910B, the operator's rendered manifests are confirmed correct for the 910
family, and the gateway data-plane is verified against the live NPU. The remaining gap — the operator
scheduling a backend pod onto an NPU via the device plugin (full integrated scale-to-zero e2e) — needs
a schedulable NPU node and stays open. Ascend support is therefore experimental / technical preview,
not yet "supported." Still v1alpha1 and not production-ready.

Added

• Ascend 910B validation report + bring-up runbook (docs/ascend-910b-validation.md)
  capturing the verified environment (910B2C 64 GB, CANN 9.0.0 / driver 26.0.rc1), the smoke test,
  the operator render dry-run, and the gateway data-plane results.

Changed

• Ascend runtime image pinned to quay.io/ascend/vllm-ascend:v0.21.0rc1 (was v0.18.0) — the
  base Atlas-A2/910B tag, matching the vllm_ascend 0.21.0rc1 stack verified on real hardware.
• Ascend status updated from "scaffolded, not run on hardware" to experimental / technical
  preview across README, ROADMAP, and the adapter docs, reflecting what is now verified on a real
  910B (vLLM-Ascend serves; manifests render correctly; gateway data-plane works) versus what is not
  (device-plugin scheduling + full integrated e2e).

Verified (Ascend 910B, real hardware)

• vLLM-Ascend serves on the NPU — Qwen2.5 loaded onto a 910B2C and answered via the OpenAI API
  (CANN 9.0.0 / driver 26.0.rc1, vllm-ascend 0.21.0rc1).
• Operator renders a correct 910B backend (kind dry-run) — huawei.com/Ascend910 request, CANN
  driver host-mounts, ModelScope cache wiring, load-gated probes; vendor selector resolves to vllm-ascend.
• Gateway data-plane on real NPU — /healthz, /hearth/queue (incl. demand-linger), /metrics,
  OpenAI passthrough (streaming + non-streaming), and cold-start SSE keepalive → activation timeout → 503.

Not yet verified

• Operator → Ascend device plugin → backend pod scheduled and serving on the NPU, and the full
  integrated 0→1→N→0 loop on a real NPU node (the v1 "supported" milestone).

---

🙏 Thanks to everyone who contributed to this release — the merged PRs and first-time
contributors are credited under What's Changed and New Contributors below.

What's Changed

🔧 Other Changes

• docs(crd): flag unimplemented fields by @ded-furby in #40

New Contributors

• @ded-furby made their first contribution in #40

Full Changelog: v0.1.0...v0.2.0-rc.1

v0.1.0

First release (alpha). The core thesis — declarative, scale-to-zero serving of self-hosted
open-source LLMs on Kubernetes — is implemented and verified end-to-end on real NVIDIA GPUs, and
the full loop now runs in CI with no hardware.
Not production-ready (see ROADMAP.md).

Added

• CRDs — LLMService (namespaced) and InferenceRuntime (cluster-scoped, pluggable backend),
  API group serving.hearth.dev/v1alpha1.
• NVIDIA backend — renders the vLLM serving workload (image, templated args, accelerator
  resource, model-load-aware probes, metrics). Ascend adapter scaffolded + golden-tested
  (not yet run on NPUs).
• Scale-to-zero — Hearth gateway (buffering reverse proxy) + KEDA ScaledObject on gateway
  queue depth; 0→1→N→0, verified 1→2 across two GPU nodes.
• Cold-start handling — SSE keepalive heartbeats, reject mode, load-gated readiness, bounded
  queue (429) and activation timeout (503).
• Graceful drain — in-flight streams finish before scale-down.
• Model caching — HostPath and NodeLocalPVC (with pinnable cache.storageClassName,
  verified against Alibaba ESSD) + a prewarm Job.
• Observability — per-gateway Prometheus metrics, ServiceMonitor, and a Grafana dashboard.
• No-GPU test harness — a CPU vllm-stub and a kind + KEDA e2e that runs the full
  0→1→N→0 loop, backpressure (429/503), and graceful drain on every PR, no accelerator
  required, plus a no-GPU development guide.
• Packaging — Helm chart (operator + RBAC + CRDs) and multi-arch image build/release workflow.
• Project scaffolding — README, ROADMAP, CONTRIBUTING, CODE_OF_CONDUCT, SECURITY, MAINTAINERS,
  GOVERNANCE, issue/PR templates, and a DCO check.

Changed

• Operator skips no-op LLMService status updates, avoiding optimistic-concurrency churn.

---

🙏 Thanks to everyone who contributed to this release — the merged PRs and first-time
contributors are credited under What's Changed and New Contributors below.

What's Changed

🚀 Features

• feat: add version flags to binaries by @loopassembly in #9

🐛 Fixes

• controller: skip no-op LLMService status updates by @Jah-yee in #29

📖 Documentation

• docs: make LLMService sample storage class neutral by @wondr-wclabs in #13
• docs(chart): populate Helm notes by @wondr-wclabs in #14
• docs: add HostPath cache sample by @wondr-wclabs in #16
• docs: add LLMService CRD reference by @wondr-wclabs in #17
• docs: add observability guide by @wondr-wclabs in #15
• docs: add Helm v4 SSA CRD ownership conflict warning by @Jah-yee in #31

New Contributors

• @loopassembly made their first contribution in #9
• @wondr-wclabs made their first contribution in #13
• @Jah-yee made their first contribution in #29

Full Changelog: v0.1.0-rc.1...v0.1.0

v0.1.0-rc.1

Full Changelog: https://github.com/hearth-project/hearth/commits/v0.1.0-rc.1