feat(stargate): add multi-cluster development deployment

[barrygreengus]

Why

Stargate needs a repeatable, dev-only multi-cluster environment for validating routing behavior, backend discovery, authentication, telemetry, and load-balancer performance before production rollout. The previous ad hoc deployment and local benchmark scripts did not provide one source of truth for topology, image pins, credentials, observability, workloads, or result capture.

What changed

• Add a Helmfile deployment for one Stargate hub and two MockDC clusters per region.
• Run three Stargate replicas, three Kubernetes backend-router replicas, and a minimal dev-only static-token auth service in the hub.
• Run two MockDynamo and Pylon pairs in each MockDC. Give every pair a unique inference server ID and cluster ID while retaining one model and routing key.
• Configure WaitAndWiden with utilization comparison, a stable two-backend affinity set, a 200 ms affinity hold, an input-token scale of 0.1, a global sample count of two, and one allowed queued request.
• Allow PowerOfN with a sample count of two and the TTFT comparator as a request-header override.
• Bound WaitAndWiden queue admission from 100 ms to 5,000 ms as request SLO budget is consumed.
• Add --disable-stats-stream to MockDynamo and disable the endpoint in the dev chart defaults. It returns HTTP 404 without opening a stats subscription. Pylon stays in auto mode so it detects the unsupported endpoint and enables its OpenAI fallback aggregators and estimators.
• Roll Stargate Pods when the startup-only load-balancer ConfigMap changes.
• Add Prometheus-compatible collection, Grafana provisioning, Stargate service dashboards, and backend traffic-balance panels.
• Add regional deployment, credential initialization, observability, and live verification tooling.
• Add canonical Spark suites for smoke, capacity, cache-affinity, and full comparisons. Generate deterministic workloads once per campaign so every configuration receives byte-identical prompts. The high-context arm uses 32 persistent session workers. Each worker owns an ordered list of six high-context, one medium-context, and one low-context session. Each session runs to completion before that worker starts its next session.
• Save workload fingerprints, deployed images, load-balancer configuration, Spark commands and reports, cache deltas, and time-scoped Grafana links.
• Make clean-cache comparisons restart all four backend Pods, refresh Stargate routing state, and wait for all replicas to observe all four backends before traffic starts. Run measured Spark traffic in a hub Pod through the ClusterIP service with --spark-pod; retain WebSocket port forwarding for local clients and fail an arm when its forwarder exits.
• Enable Pylon startup calibration with maximum concurrency 25, matching the MockDynamo profile. Disable periodic canaries and remove the fixed initial-TPS seed.
• Ignore terminating MockDC Pods during rollout verification.

The affinity policy and concurrency-aware queue estimates are implemented in #1660. This deployment PR supplies the environment and canonical workload suite used to validate that change.

Customer Release Notes

Not customer visible.

Plan Summary

The stack deploys into existing Kubernetes clusters. One regional hub receives routing, auth, and observability services. Two regional MockDC clusters each receive two independent mocked inference backends. Published service images are supplied through a protected values file and pinned by digest.

Usage

Initialize credentials once, then apply each phase with protected regional values:

python3 deploy/stacks/stargate-dev/scripts/deploy.py init --region REGION --credentials CREDENTIALS_JSON
python3 deploy/stacks/stargate-dev/scripts/deploy.py apply --region REGION --phase stargate --credentials CREDENTIALS_JSON --values VALUES_YAML
python3 deploy/stacks/stargate-dev/scripts/deploy.py apply --region REGION --phase mockdc --credentials CREDENTIALS_JSON --values VALUES_YAML
python3 deploy/stacks/stargate-dev/scripts/deploy.py apply --region REGION --phase observability --values VALUES_YAML
python3 deploy/stacks/stargate-dev/scripts/verify.py --region REGION --phase regional

List and run the canonical load-test suites:

python3 deploy/stacks/stargate-dev/scripts/loadtest.py --list
STARGATE_SPARK_IMAGE=IMAGE python3 deploy/stacks/stargate-dev/scripts/loadtest.py --suite canonical --region REGION --algorithm ALGORITHM --spark-pod SPARK_POD --output RESULTS

Repeat the same command with --resume to continue an interrupted campaign. WaitAndWiden requests include x-cache-affinity-key, x-input-tokens, x-request-slo-ms, and x-max-wait-ms through Spark.

Testing

• MockDynamo stats-stream toggle: 37 Rust tests passed, including disabled-endpoint and profile compatibility coverage. Clippy passed with warnings denied.
• python3 -m unittest discover -s deploy/stacks/stargate-dev/tests -v: 23 tests passed. The seven deployment invariant tests also passed after the calibration-cap wiring change.
• Ruff check, Ruff format check, and git diff --check: passed for the deployment changes.
• Deployment-time Helm lint and schema validation: passed for the Stargate and MockDC phases.
• Built and deployed the tested MockDynamo and Pylon images. The full campaign used Stargate f9058ea9f, Pylon 7ec0e9883, MockDynamo 0cef9214c, and deployment configuration ffb5c7ec8.
• Post-run regional verification: passed. The temporary Spark Pod was removed after result collection.
• The algorithm change at f9058ea9f passed 1,148 Cargo tests across Stargate, Pylon library, and protocol packages, plus Clippy with warnings denied. Both deployed images built successfully with Bazel.

The full canonical WAW and PowerOf2 suite completed: 28 measured reports, 88,277 successful requests, zero failed requests, and two client retries.

Full canonical results with fallback statistics

The September 11 canonical campaign completed 28 measured reports and 88,277 logical requests. All 88,277 succeeded, with zero terminal failures. WAW had zero client retries. PowerOf2 had two client retries in the 48 RPS capacity arm; both requests ultimately succeeded. Four one-request mixed-session warm-ups are excluded from these totals.

Tested images: Stargate f9058ea9f, Pylon 7ec0e9883, and MockDynamo 0cef9214c, deployed through ffb5c7ec8 and pinned by digest. Both algorithms used the same images, resources, Spark commit 3cb1478, and six byte-identical workload files.

MockDynamo returned HTTP 404 for its disabled stats endpoint. Pylon stayed in auto mode and activated fallback aggregators and estimators. --max-engine-concurrency=25 retained the engine limit, startup calibration was capped at 25, and periodic canaries were disabled. KV-cache HTTP polling remained enabled.

The suite covered smoke, five capacity levels, three normal-session repeats per algorithm, one high-context arm per algorithm, and two mixed-traffic repeats per algorithm. Each affinity arm reset caches, recalibrated Pylons, refreshed routing state, and waited for four active backends.

WAW used utilization comparison, two affinity candidates, input-token scale 0.1, a 200 ms affinity hold, n=2, max_queued=1, and queue bounds of 100-5,000 ms. PowerOf2 used two samples and the TTFT comparator. Each scenario used identical request timing settings for both algorithms. The high-context arms used a 60-second request SLO and routing wait, with a 90-second client timeout.

Capacity sweep

Each arm ran for 120 seconds. Target RPS is the generator rate cap; achieved throughput is reported separately. Every capacity arm completed with zero failed requests.

Target RPS	WAW achieved RPS	PowerOf2 achieved RPS
8	7.87	7.84
16	14.65	14.13
24	19.99	18.35
32	22.22	21.30
48	24.97	24.58

Session workloads

Counts below are sums across repeats. Throughput, cache-hit percentages, and latency percentiles are medians of per-run statistics, not pooled percentiles. Cache-hit percentages use successful requests with observed cache information.

Workload	WAW successful	PowerOf2 successful	WAW RPS	PowerOf2 RPS	WAW cache hit	PowerOf2 cache hit
Normal sessions (3 repeats)	18,432 / 18,432	18,432 / 18,432	18.947	16.835	69.56%	32.68%
High context	768 / 768	768 / 768	2.970	2.460	41.67%	19.14%
Mixed hot (2 repeats)	9,564 / 9,564	9,569 / 9,569	7.970	7.974	99.98%	99.94%
Mixed short (2 repeats)	4,785 / 4,785	4,786 / 4,786	3.987	3.988	42.34%	20.66%

The high-context workload used 32 persistent workers, each with eight ordered three-turn sessions: six high-context sessions at 80k/100k/120k input tokens, one medium session at 8k/16k/24k, and one low session at 1k/2k/4k. Both algorithms completed all 768 requests without retries. WAW finished in 258.558 seconds (4.31 minutes); PowerOf2 finished in 312.185 seconds (5.20 minutes).

Mixed arms ran concurrent hot- and short-session streams for ten minutes at rate caps of 8 and 4 RPS. All mixed requests succeeded without retries. Both smoke arms also passed all 32 requests without retries.

Latency and tradeoffs

Workload	WAW TTFT p99	PowerOf2 TTFT p99	WAW end-to-end p99	PowerOf2 end-to-end p99
Normal sessions (3 repeats)	426 ms	442 ms	3,572 ms	4,456 ms
High context	17,039 ms	17,039 ms	19,137 ms	19,399 ms
Mixed hot (2 repeats)	118 ms	108 ms	3,277 ms	2,818 ms
Mixed short (2 repeats)	259 ms	281 ms	2,769 ms	2,982 ms
Capacity at 48 RPS target	721 ms	287 ms	5,636 ms	6,095 ms

WAW improved normal-session throughput and cache reuse and completed the high-context workload faster. PowerOf2 had lower TTFT tails for the hot mixed stream and at the highest capacity target. At that target, WAW achieved slightly higher throughput but its TTFT p99 was 721 ms versus 287 ms. This tradeoff matters when selecting an affinity hold for a strict latency target.

Validation and limits

Post-run regional checks passed. All four MockDynamo stats endpoints still returned 404, Pylon recorded fallback activation with no received engine-stream events, and deployment images, arguments, replica counts, and resource limits matched the captured environment. The temporary Spark Pod was removed and diagnostic collectors were stopped.

All successful routing snapshots showed four active backends: 972 observations across measured run windows, with one auxiliary read timeout. The fallback audit covered 941 observations with no unexpected stream activity and one auxiliary read timeout. These two monitoring timeouts were separate from the workload, which had zero failed requests.

This is a single-region mock benchmark. Unlike the earlier high-context-only comparison, both algorithms were rerun under the same fallback-stats setup. Historical stats-stream runs are separate evidence. Client latency includes retry time. Backend-share and routing-counter exports use their recorded Prometheus observation windows; mixed backend counters combine the hot and short streams.

Suggested production WAW configuration

For a scoped rollout of session-oriented text-generation models, use the following as a starting configuration. It preserves the WAW settings exercised by the latest benchmark and makes the tested TTFT bucket defaults explicit. Replace MODEL_NAME with the routed model ID. Unlisted models continue using PowerOfN.

{
  "default": "power-of-n",
  "models": {
    "MODEL_NAME": {
      "algorithm": "wait-and-widen",
      "cache_affinity_backend_selection_count": 2,
      "cache_affinity_input_tokens_scale": 0.1,
      "cache_affinity_virtual_nodes": 150,
      "cache_affinity_wait_ms": 200,
      "comparator": "utilization",
      "max_queue_time_ceil_ms": 5000,
      "max_queue_time_floor_ms": 100,
      "max_queued": 1,
      "n": 2,
      "require_cache_affinity_key": true,
      "require_input_tokens": true,
      "ttft_bucket_size_ms": 20,
      "next_bucket_unlock_factor": 0.25
    }
  }
}

Use Stargate builds containing the free-slot queue correction (tested at f9058ea9f) and Pylon builds containing the explicit concurrency flag (tested at 7ec0e9883), or releases containing the merged PR #1660 changes. Validate the configuration against the real inference engine and representative traffic before expanding the rollout; the current performance evidence comes from MockDynamo.

Routing policy and tuning

• Check the affinity set first on every routing attempt. After 200 ms, an unsuccessful affinity check can proceed to global buckets containing all backends, including the affinity set at full prefill cost.
• Use utilization comparison and n=2 within unlocked TTFT buckets. Keep the same model configuration and affinity-hash settings across router replicas.
• Treat cache_affinity_input_tokens_scale=0.1 as the tested affinity scoring weight, not a measured 90% cache-hit rate. Tune it against observed prefill savings; use 1.0 when cache reuse is not established.
• Keep max_queued=1 as an initial bound: the capacity check allows the reported engine concurrency plus one pending request. This is separate from the queue-time allowance, which grows from 100 ms to 5,000 ms as the request SLO budget is consumed.
• A known free execution slot gives zero effective queue delay. Active prefill, decode, and pending assignments consume slots; the request's own prefill time still contributes to TTFT. Verify that the engine's concurrency limit and active-request count describe the same execution capacity.

Gateway and deadline settings

Use the per-model WAW policy by omitting an x-routing-method override for these requests. The example requires a stable x-cache-affinity-key and an x-input-tokens estimate. The gateway should derive them from the session/prefix and request content. For a model that also accepts requests without an affinity key, set require_cache_affinity_key=false; those requests use non-affinity routing.

Set x-request-slo-ms from the service latency target, x-max-wait-ms from its routing-wait budget, and x-stargate-max-wait-ms when an explicit proxy-retry budget is required. The SLO header controls queue-admission interpolation; it is not an end-to-end timeout. Budget routing waits, backend processing, response delivery, and retries within the client's deadline.

The high-context test used a 60-second request SLO, a 60-second maximum routing wait, and a 90-second client timeout per attempt. These are workload-specific reference values, not universal production defaults.

Pylon and engine prerequisites

Require a valid measured throughput estimate and an accurately reported engine concurrency limit. For an engine owned by one Pylon per logical cluster, the matching startup settings are:

--do-calibration
--calibration-max-concurrency=ENGINE_CONCURRENCY
--max-engine-concurrency=ENGINE_CONCURRENCY
--active-canary-interval-ms=0

Replace ENGINE_CONCURRENCY with the engine's actual limit; it was 25 in this environment. --max-engine-concurrency supplies capacity when the engine does not report it; an engine-reported limit takes precedence. Omit --initial-input-tps when startup calibration is enabled. Multiple Pylons sharing one engine require coordinated calibration or a separately measured bootstrap rate. Check the completed calibration ramp: the configured cap alone does not prove that calibration reached it.

During rollout, track request failures and deadlines, TTFT and decode latency, active requests versus engine capacity, cache reuse, backend traffic distribution, and Pylon queue-mismatch rejections. Exercise both ordinary traffic and sustained saturation; overload should produce bounded waits and failures within the configured budgets. Use production authentication and routing infrastructure for this configuration; the dev stack's static-token auth service is scoped to development.

Notes

Fallback mode is deployed with Pylon 7ec0e9883 and MockDynamo 0cef9214c; Stargate remains on f9058ea9f. Deployment commit ffb5c7ec8 supplies --max-engine-concurrency=25. All four MockDynamo stats endpoints returned HTTP 404, Pylon recorded automatic fallback activation with no received stream events, and calibration advertised the 25-slot limit. Startup calibration is enabled and periodic canaries are disabled. The full canonical comparison completed with 88,277 successful requests and no failed requests.

Repository defaults remain non-deployable placeholders so credentials and private registry coordinates are not committed. Live images are supplied by digest through a protected values file. The current dev deployment uses Stargate f9058ea9f, Pylon 7ec0e9883, and MockDynamo 0cef9214c, with max_queued=1, configured engine concurrency 25, startup calibration capped at 25, and periodic canaries disabled.

Issues

Relates to #1420

References

• https://github.com/NVIDIA/nvcf/releases/tag/src%2Flibraries%2Frust%2Fstargate%2Fv0.15.1

Related Pull Requests

• feat(stargate): add affinity waits and honor engine concurrency #1660
• https://github.com/nvidia-lpu/spark/pull/139
• https://github.com/nvidia-lpu/spark/pull/140

Dependencies

No new third-party dependencies. No license review or NOTICE change is required.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🛡️ CodeQL Analysis

🚨 Found 5 issue(s)

Severity Breakdown:

• 🔴 Errors: 0
• 🟡 Warnings: 0
• 🔵 Notes: 0

📋 Top Issues

🔗 View full details in Security tab

_{🕐 Last updated: 2026-09-03 17:42:41 UTC | Commit: d3ef5fd}