v1.1.0

Changelog

• aa80f0a chore(release): 1.1.0 (#105)
• 98756bd feat(install-node): --schedule installs a load schedule; status shows the level (#103)
• 1159c42 feat(node-agent): load schedule by local time, pause with model unload and warm-up (#102)
• 0ec424d feat(nodes): capacity levels from the node heartbeat, PAUSED state and post-request cooldown (#99)
• 0374db4 fix(release): sign the chart after pushing it; ignore usage.v1's Go package (#104)

v1.0.0

Changelog

• 9b4c375 Merge remote-tracking branch 'origin/main' into release-1.0-sync
• e836d26 chore(release): 1.0.0
• 8bc4faf chore(release): 1.0.0 (#88)
• 40cf332 ci(buf): fail the BSR push without BUF_TOKEN instead of skipping it (#100)
• 83cf342 ci(helm): publish the chart as OCI on release tags; kind install test (#93)
• 2b336ba ci: usage.proto parity check with light-tokenmeter (#91)
• 1b56daa docs(proto): keep the cache-price and key-policy comments to three lines (#97)
• 39da9c0 docs(readme): the web console section lists every ktms-admin sign-in method
• 8ec32a7 feat(install-node): take over llm-proxy-node installs; refuse a foreign Ollama port (#98)
• 19c72f4 feat(keys): per-key policy: allowed models, RPM and TPM over all models (#95)
• 309237c feat(pricing): price prompt-cache reads and writes at their own rates (#94)
• 86c74af feat(usage): USAGE_COLLECTOR_TOKEN for an authenticated usage sink (#90)
• 2a680ee feat: GetSpend for keys, users and teams; prompt_cache for Anthropic (#96)
• 56c6129 test(upgrade): wait until the old release serves the seeded model (#101)

Known issue

• The Helm chart oci://ghcr.io/ai-process/charts/ktms 1.0.0 is published but not signed: the release job stopped after pushing it. Binaries (checksums.txt) and images are signed. Chart signatures start with 1.0.1.

v0.14.0: organizations and teams, signed releases

KTMS 0.14.0 is the last release before 1.0. It adds organizations and teams, and it is the first release published through the new compatibility gate and with signed artifacts.

Upgrading from 0.13.x: deployments that never set USAGE_PROJECT_ID now report usage under ktms instead of llm-proxy; set it to keep the old name. ktms-admin 0.4.0 pairs with this release.

Verify the images and checksums.txt as described in SECURITY.md (use V=v0.14.0).

Added

• Organizations and teams: team keys share a model allowlist and a monthly budget enforced
  by the gateway (CreateOrganization, CreateTeam, UpdateTeam, GetTeamSpend, ...).
  ListModels lists only the models a team may call.
• Operations: backups and the keyring, and a safe key rotation procedure, tested against
  Postgres; CI now runs the Postgres-backed tests (they used to skip).
• An upgrade test on every pull request and before each release: the previous release and the
  oldest release of the same major write a database, this build must keep and serve it, and the
  old release must still serve on the migrated database (rolling upgrades).
• Releases are gated: binaries and images are published only after the proto and upgrade checks
  against the previous release pass, and images only after the binaries.
• Signed releases: images and checksums.txt are signed with keyless cosign; images carry an
  SBOM and SLSA provenance, archives an SPDX SBOM. See SECURITY.md, Verifying a Release.
• govulncheck for linux and darwin on every pull request and before every release.
• The public API for 1.x is defined in the README (API stability),
  with experimental elements marked Experimental: in the proto.
• A startup warning when the deprecated LLMPROXY_ENCRYPTION_KEYS is used instead of
  KTMS_ENCRYPTION_KEYS.

Changed

• USAGE_PROJECT_ID now defaults to ktms, as the README already documented (it was
  llm-proxy). Set it explicitly to keep usage under the old project name; while it is unset
  and usage reporting is on, the gateway logs this at startup.
• Logs carry service=ktms instead of service=llm-proxy.
• The ffmpeg_missing message names the coyl/ktms:audio image.
• CI also checks usage.v1 for breaking changes. Adding the buf skip breaking label re-runs the check.
• A cache hit refused by the rpm limit returns the documented throttled:rpm reason instead of
  resource_exhausted: rpm limit exceeded.
• SENTRY_ENVIRONMENT defaults to development, as documented; it was sent empty.
• Postgres outages return the reasons auth_unavailable and storage_unavailable (were
  auth backend unavailable and storage unavailable), so the HTTP API puts them in code.
• DrainNode without node support returns FAILED_PRECONDITION instead of UNAVAILABLE.
• Docker images are built only for v* release tags (any tag used to produce images), and
  latest, slim and audio follow only the newest stable release.

Fixed

• RotateEncryption also re-encrypts the auth tokens of registered nodes, so dropping the old
  key version no longer breaks calls to nodes, and it locks the vendor key rows, so a key set
  during a rotation is no longer overwritten with the previous one.
• A vendor key the replica's keyring cannot read no longer stops that replica from loading
  config: it keeps the credential it had (or skips the key), logs and counts it. A keyring that
  reads no vendor key at all still refuses to start. RotateEncryption over such a key fails
  with FAILED_PRECONDITION instead of UNAVAILABLE storage_unavailable.
• With AUTO_SEED, starting a replica no longer rewrites unchanged seeded vendor keys, or keys
  a newer keyring already rotated, which broke rotations.

Deprecated

• LLMPROXY_ENCRYPTION_KEYS: use KTMS_ENCRYPTION_KEYS. It is removed in 2.0.
• X-LLM-Proxy-* headers: use X-KTMS-*. They are removed on their announced Sunset date,
  2027-03-01, the one removal inside 1.x.

v0.13.1

Changelog

• d2a7b2e feat(bench): benchmark all notable self-hosted gateways (#73)
• a6f8e5b feat(bench): reproducible gateway benchmark stand (k6, mock upstream, GCP) (#69)
• eed05da feat(router): honour vendor Retry-After and stop hidden SDK retries (#76)
• 658d50e fix(failure-modes): bound Redis, keep the node pool honest, queue usage events (#79)
• a7cb27d fix(failure-modes): keep serving known keys through a Postgres outage (#80)
• 29f2fab fix(nodes): report engine queue in heartbeats and keep unary leases (#78)
• 98b167c fix(schema): carry enum, number, format and additionalProperties (#77)
• 96ca1f5 fix(security): pin Go 1.27.3, Node 24.22.1 and Alpine 3.23 with ca-certificates >= 20261020-r0 (#81)
• 9dc0df2 fix: reuse vendor connections under load (#72)

v0.13.0

Changelog

• bbceed5 chore(oss): community health files (issue templates, PR template, SECURITY.md, CODE_OF_CONDUCT.md, badges) (#62)
• 9015663 ci: update Helm chart path to deploy/helm/ktms in test workflow
• f2abf30 feat(deploy): official Helm chart for Kubernetes deployment (#64)
• 5115785 feat(dx): zero-config quickstart, auto-seed and secure compose stack (#59)
• 0b3c09f feat(router): canary weights and async shadow traffic evaluation (#31) (#67)
• 6882351 feat: per-attempt upstream vendor request timeout for circuit breaker trip detection (#65)
• fd24350 feat: shadow routing (async traffic mirroring) (#61)
• 4b72203 feat: split docker images into slim and audio variants (#63)
• 78318f7 feat: upstream vendor circuit breaker for fail-fast error isolation (#60)
• f49cbff fix(circuitbreaker): atomic gauge update in transitionTo and clarify timeout tripping in docs
• fcb03af rebrand: migrate project to KTMS (Keystone: The Model Switchboard)

v0.12.0: JEPA, Latent Semantic Cache, Batch Rollout Judge, Multimodal Decisions & Spend Limits

Overview

Release v0.12.0 brings representation-space intelligence, latent semantic caching, batch trajectory evaluation for agentic workflows, text embeddings, multimodal decision support, monetary spend limits, and local inference edge capabilities to llm-proxy.

---

What's New in v0.12.0

1. Latent Semantic Cache & Representation-Space Guardrails (#46, #58)

• Redis-Backed Latent Semantic Cache: Caches responses based on the latent representation of the final user message (internal/cache/semantic.go) using configurable cosine similarity thresholds and bounded search scans. Keeps system instructions, conversation history, tools, and user configuration strictly isolated.
• Payload Encryption: Embeddings and response payloads are encrypted with AES-256-GCM via the proxy keyring before storing in Redis.
• Flexible Encoders (internal/latent):
  • HTTP embeddings encoder for JEPA edge nodes and OpenAI-compatible /v1/embeddings services.
  • Local hash encoder with zero-width character, homoglyph, whitespace normalization and character folding.
  • In-memory LRU memoization ensuring identical utterances across cache and guardrails are encoded once.
  • Fallback chain ensuring uninterrupted traffic when external encoders are degraded.
• Representation-Space Guardrails (internal/guardrails/latent.go): Detects prompt injections, jailbreaks, system prompt leaks, and exfiltration attempts against precomputed cluster centroids. Supports block/log modes with fail-open safety.
• Safety Filters: Automatic PII exclusion and guardrail screening before serving semantic cache hits.

2. Batch Trajectory & Rollout Evaluation for Agentic Workflows (#47, #57)

• EvaluateRollouts & JudgeStream RPCs: New gRPC endpoints on LLMProxyService to evaluate batches of candidate trajectories/actions against state and criteria for speculative decoding and Best-of-N selection.
• Bidirectional Streaming Decisions: JudgeStream provides bidirectional streaming decision loops for real-time agent evaluation and robotics control.
• Parallel Candidate Execution: Parallel candidate evaluation bounded by maxRolloutConcurrency = 16, with concurrent map safety and isolated goroutine metadata.
• Calibrated Scoring & Ranking: Monotonic composite scoring normalized across level counts and criteria, computing candidate ranks and identifying the best candidate.
• OpenAI-Compatible HTTP Endpoint: POST /v1/decisions/rollouts supporting text, structured JSON actions, and multimodal media.
• Circuit Breaker Protection: OpenAIAdapter and PoolAdapter support native rollout endpoints with safe fallback that preserves circuit-breaker state on nodes without rollout endpoints.

3. Text Embeddings API & Strict Vector Space Isolation (#51)

• GetEmbeddings RPC & POST /v1/embeddings: Native gRPC and HTTP endpoints for vector embeddings generation.
• Strict Vector Space Isolation: Disallows cross-model fallbacks for embeddings to prevent vector space contamination across disparate models.
• Multi-Vendor Adapter Support: OpenAI and Google adapters supporting float and base64 encoding formats, dimension truncation, and token counting.
• Input Screening: PII detection and token vault support for embedding inputs.

4. Multimodal Judge & Decision Model Support (#45, #50)

• Multimodal State Media: JudgeRequest.state_media supports vision (ImagePart), audio (AudioPart), and latent vector embeddings (EmbeddingPart).
• HTTP Decisions Support: POST /v1/decisions accepts arrays of OpenAI input parts (input_text, input_image, image_url, input_audio, input_embedding).
• Validation Ceilings: Strict validation for mime types, https-only URLs, finite vectors, and part/byte/dimension limits.
• New Model Capability: embedding_input capability for decision models that directly evaluate raw latent embeddings.

5. Monetary Spend Limits with Atomic Redis Reservations (#48)

• Hard USD Spend Limits: Enforces daily and monthly UTC spend caps per API key and end user via atomic reservations.
• Atomic Multi-Counter Reservation: Lua script reserves est_in*price_in + max_out*price_out across all applicable limits atomically before invoking the model, settling to actual costs post-completion.
• Nano-USD Pricing Math: Exact 128-bit pricing calculation (internal/pricing) across standard, peak, and batch execution modes.
• Admin Management API: SetSpendLimit, DeleteSpendLimit, and ListSpendLimits admin RPCs.
• Standardized Refusals: Returns RESOURCE_EXHAUSTED with structured insufficient_quota ErrorInfo and RetryInfo specifying period reset times.
• Usage Reporting: All usage events now carry cost_nanousd.

6. Local JEPA & Decision Model Inference on Node Agents (#44, #49)

• Node Agent Flags: --judge-model flag tags local models with the judge capability, routing /v1/decisions calls directly to local edge hardware.
• Gateway Forwarding: Node agent gateway forwards /v1/decisions and /v1/decisions/rollouts to local inference engines.
• Pool Adapter Integration: Leases slots, tracks concurrent capacity, and falls back gracefully when nodes do not expose rollout endpoints.

---

Commits in this Release

• d2100fe feat(jepa): latent semantic cache and representation-space guardrails (#46) (#58)
• 273bd35 feat(judge): batch trajectory and rollout evaluation for agentic workflows (#47) (#57)
• 3eb79bb feat: add embeddings support (GetEmbeddings and POST /v1/embeddings) (#51)
• ca3ddf3 feat(judge): multimodal state support (vision, audio, embeddings) in Judge API (#50)
• 25396ac feat: monetary spend limits with atomic Redis reservation (#48)
• 1dc8b79 feat(nodes): support local JEPA and decision models in node-agent and pool adapter (#44) (#49)

v0.11.1

Changelog

• cac0617 feat(proto): Buf Schema Registry export and multi-language client templates (#43)

v0.11.0

Changelog

• 569f2dc feat(router): smart auto model routing and node placement (#42)

v0.10.3

Changelog

• 1dc331f fix(router): local models need no vendor key (#41)

v0.10.2

Changelog

• 44993ad Merge pull request #21 from ai-process/feat/multimodal-support
• 94e1a11 Reach the usage sink over TLS (USAGE_COLLECTOR_TLS)
• c8ba7f7 feat(guardrails): add PII masking and prompt injection detection (#36)
• d399430 feat(registry): add Azure OpenAI vendor support with content filter fallback (#37)
• 3a67343 feat(vendor): add AWS Bedrock Converse and ConverseStream support (#38)
• d89b6b5 feat: add Prometheus metrics export and OpenTelemetry distributed tracing (#32)
• 5163b14 feat: add exact-match response cache in redis with streaming support (#33)
• 5c51e4e feat: add groq and mistral provider support (#35)
• 423547b feat: add weighted routing and canary traffic splitting to routing rules (#34)
• 6e3be8f feat: multimodal input support (vision, audio, documents in ChatMessage)
• fe59446 fix(batch): return ErrNoVendorKey, document error reasons, and sync config (#39)