Skip to content

v0.12.0: JEPA, Latent Semantic Cache, Batch Rollout Judge, Multimodal Decisions & Spend Limits

Choose a tag to compare

View all tags
@github-actions github-actions released this 16 Oct 04:10
· 72 commits to main since this release
v0.12.0
00eedfc

Overview

Release v0.12.0 brings representation-space intelligence, latent semantic caching, batch trajectory evaluation for agentic workflows, text embeddings, multimodal decision support, monetary spend limits, and local inference edge capabilities to llm-proxy.

What's New in v0.12.0

1. Latent Semantic Cache & Representation-Space Guardrails (#46, #58)

  • Redis-Backed Latent Semantic Cache: Caches responses based on the latent representation of the final user message (internal/cache/semantic.go) using configurable cosine similarity thresholds and bounded search scans. Keeps system instructions, conversation history, tools, and user configuration strictly isolated.
  • Payload Encryption: Embeddings and response payloads are encrypted with AES-256-GCM via the proxy keyring before storing in Redis.
  • Flexible Encoders (internal/latent):
    • HTTP embeddings encoder for JEPA edge nodes and OpenAI-compatible /v1/embeddings services.
    • Local hash encoder with zero-width character, homoglyph, whitespace normalization and character folding.
    • In-memory LRU memoization ensuring identical utterances across cache and guardrails are encoded once.
    • Fallback chain ensuring uninterrupted traffic when external encoders are degraded.
  • Representation-Space Guardrails (internal/guardrails/latent.go): Detects prompt injections, jailbreaks, system prompt leaks, and exfiltration attempts against precomputed cluster centroids. Supports block/log modes with fail-open safety.
  • Safety Filters: Automatic PII exclusion and guardrail screening before serving semantic cache hits.

2. Batch Trajectory & Rollout Evaluation for Agentic Workflows (#47, #57)

  • EvaluateRollouts & JudgeStream RPCs: New gRPC endpoints on LLMProxyService to evaluate batches of candidate trajectories/actions against state and criteria for speculative decoding and Best-of-N selection.
  • Bidirectional Streaming Decisions: JudgeStream provides bidirectional streaming decision loops for real-time agent evaluation and robotics control.
  • Parallel Candidate Execution: Parallel candidate evaluation bounded by maxRolloutConcurrency = 16, with concurrent map safety and isolated goroutine metadata.
  • Calibrated Scoring & Ranking: Monotonic composite scoring normalized across level counts and criteria, computing candidate ranks and identifying the best candidate.
  • OpenAI-Compatible HTTP Endpoint: POST /v1/decisions/rollouts supporting text, structured JSON actions, and multimodal media.
  • Circuit Breaker Protection: OpenAIAdapter and PoolAdapter support native rollout endpoints with safe fallback that preserves circuit-breaker state on nodes without rollout endpoints.

3. Text Embeddings API & Strict Vector Space Isolation (#51)

  • GetEmbeddings RPC & POST /v1/embeddings: Native gRPC and HTTP endpoints for vector embeddings generation.
  • Strict Vector Space Isolation: Disallows cross-model fallbacks for embeddings to prevent vector space contamination across disparate models.
  • Multi-Vendor Adapter Support: OpenAI and Google adapters supporting float and base64 encoding formats, dimension truncation, and token counting.
  • Input Screening: PII detection and token vault support for embedding inputs.

4. Multimodal Judge & Decision Model Support (#45, #50)

  • Multimodal State Media: JudgeRequest.state_media supports vision (ImagePart), audio (AudioPart), and latent vector embeddings (EmbeddingPart).
  • HTTP Decisions Support: POST /v1/decisions accepts arrays of OpenAI input parts (input_text, input_image, image_url, input_audio, input_embedding).
  • Validation Ceilings: Strict validation for mime types, https-only URLs, finite vectors, and part/byte/dimension limits.
  • New Model Capability: embedding_input capability for decision models that directly evaluate raw latent embeddings.

5. Monetary Spend Limits with Atomic Redis Reservations (#48)

  • Hard USD Spend Limits: Enforces daily and monthly UTC spend caps per API key and end user via atomic reservations.
  • Atomic Multi-Counter Reservation: Lua script reserves est_in*price_in + max_out*price_out across all applicable limits atomically before invoking the model, settling to actual costs post-completion.
  • Nano-USD Pricing Math: Exact 128-bit pricing calculation (internal/pricing) across standard, peak, and batch execution modes.
  • Admin Management API: SetSpendLimit, DeleteSpendLimit, and ListSpendLimits admin RPCs.
  • Standardized Refusals: Returns RESOURCE_EXHAUSTED with structured insufficient_quota ErrorInfo and RetryInfo specifying period reset times.
  • Usage Reporting: All usage events now carry cost_nanousd.

6. Local JEPA & Decision Model Inference on Node Agents (#44, #49)

  • Node Agent Flags: --judge-model flag tags local models with the judge capability, routing /v1/decisions calls directly to local edge hardware.
  • Gateway Forwarding: Node agent gateway forwards /v1/decisions and /v1/decisions/rollouts to local inference engines.
  • Pool Adapter Integration: Leases slots, tracks concurrent capacity, and falls back gracefully when nodes do not expose rollout endpoints.

Commits in this Release

• d2100fe feat(jepa): latent semantic cache and representation-space guardrails (#46) (#58)
• 273bd35 feat(judge): batch trajectory and rollout evaluation for agentic workflows (#47) (#57)
• 3eb79bb feat: add embeddings support (GetEmbeddings and POST /v1/embeddings) (#51)
• ca3ddf3 feat(judge): multimodal state support (vision, audio, embeddings) in Judge API (#50)
• 25396ac feat: monetary spend limits with atomic Redis reservation (#48)
• 1dc8b79 feat(nodes): support local JEPA and decision models in node-agent and pool adapter (#44) (#49)

Assets 11
Loading