Skip to content

feat(gateway): select deployments from bounded runtime signals - #163

Draft
tcballard wants to merge 1 commit into
mainfrom
codex/enterprise-deployment-selection
Draft

feat(gateway): select deployments from bounded runtime signals#163
tcballard wants to merge 1 commit into
mainfrom
codex/enterprise-deployment-selection

Conversation

@tcballard

Copy link
Copy Markdown
Contributor

Summary

  • Add bounded, process-local deployment observations for interchangeable members behind one public model alias.
  • Select eligible pool members using deterministic latency, throughput, availability, cost, or capacity policies.
  • Preserve weighted ordering as the compatibility default and as the fallback for stale, sparse, or incomplete observations.
  • Expose the concrete deployment and selection reason in response headers, route receipts, and prompt-free metrics.
  • Document the contract in WF-ADR-0062 and update the deployment-pool, gateway configuration, roadmap, and changelog sources of truth.

Refs #148
Parent: #146

Scope

Included:

  • typed gateway configuration for deployment-selection strategy, observation TTL, member cost, and hard price ceilings;
  • bounded rolling observation store with sample/key/cardinality limits and sanitization;
  • buffered and streaming observation capture for latency, TTFT, throughput, availability, rate limits, failures, and capacity pressure;
  • deterministic stable tie-breaking and fail-closed price filtering;
  • bounded Prometheus-compatible telemetry and concrete deployment response headers;
  • focused stale-data, tie, strategy, price, flood, and public-alias invariants.

Not included:

  • changes to the authoritative scored routing core;
  • changes to privacy, tier, capability, named-route, or pinned-public-alias decisions;
  • Redis schema or protocol changes;
  • Python gateway work.

Product / Architecture Decisions

  • Runtime signals can only reorder already-eligible interchangeable deployments behind one public alias.
  • A missing, stale, sparse, malformed, or conflicting signal set falls back to the existing weighted order.
  • A configured price ceiling is a hard filter; unknown member prices are excluded when the ceiling is active.
  • The existing shared Redis health/circuit path remains authoritative for fleet-wide admission. This local signal store is selection evidence, not a replacement for shared health.
  • The default policy remains weighted, so existing configurations and behavior are unchanged until an operator opts in.

User-Facing Contract

  • Successful responses expose x-wayfinder-router-deployment and x-wayfinder-router-deployment-selection.
  • Route receipts continue to identify the public model alias and now retain the concrete deployment/selection reason where available.
  • Streaming and buffered delivery use the same selection semantics.
  • Pinned destinations retain pin semantics; this feature does not silently switch a named route or cross model/privacy boundaries.

Verification

  • cargo fmt --manifest-path rust/Cargo.toml --all -- --check
  • git diff --check
  • cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --all-features --locked -- -D warnings
  • cargo test --manifest-path rust/Cargo.toml --workspace --all-features --locked (full workspace passed, including gateway and Redis-backed suites)
  • node clients/shared/test/parity.mjs (21/21 score, routing, and feature parity)

Review Path

Please review the policy boundary and fallback semantics first, then the buffered/streaming observation lifecycle, bounded metric labels, and configuration compatibility. The routing-core decision itself is intentionally untouched.

Notes for Reviewer

The process-local observation clock is monotonic and shared across config reloads. Runtime signals are prompt-free and bounded to avoid high-cardinality/flood behavior. The full test suite initially exposed an existing metrics compatibility fixture sensitivity; new metric families are emitted only when populated, preserving the empty exposition contract.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant