Showcase / RFC: Cordis multi-tenancy with LiteLLM and DeepInfer / SGLang

[skunpoj]

Governed Harness integration: current bounded description

This discussion describes an independent DSH deployment pattern. It is not an upstream-supported plugin contract or a claim that every discussed feature is deployed and qualified.

Architecture and ownership

The retained functional route is dsh-gateway -> litellm-gateway -> context-forwarder -> deepinfer-dspark. DeepInfer is the inference engine. sglang-dspark is a separately retained, operator-controlled alternate; the two four-GPU engines are not simultaneous services. Canonical Services are live, while historical Deployment identities remain temporarily for compatibility until every consumer and rollback reference is migrated.

The public setup home is skunpoj/dsh_setup. It contains sanitized reference configuration, documentation, tests, and runbooks. Site credentials, private overlays, build artifacts, cluster identities, and user data are intentionally excluded. Publishing this repository does not prove live deployment parity or completion of the private CI/CD cutover.

The GPU source ownership and release map is the canonical boundary: dsh_setup owns portable public product/configuration source, skunpoj/DeepInfer owns engine source, and a private site repository must own domains, registry, storage, identities, Secrets, image promotion, overlays, monitoring and deployment receipts. The running workload still requires a single reproducible receipt that pins all source commits and immutable image digests.

The engine source remains separate in skunpoj/DeepInfer. The running container uses lmsysorg/sglang as its dependency base, but PID 1 is the mounted DeepInfer serve/openai_server.py entry point and the expert-communication backend is NCCL. The private release receipt recorded 293 expected mounted Python paths, 293 read paths and zero mismatches against that private release source. A separate normalized-content comparison against later public successor c1b8cc6 found 17 exact-byte matches, 274 line-ending-only differences, two newer public fixes not yet mounted, and no missing Python path. The two content differences are strict boolean parsing and an explicit raw-string regular expression in the pinned humming-oracle compiler. Compiled artifacts, image build inputs, vendor dependencies, private overlays, and exact clean-checkout-to-image reproducibility remain open.

The one-file tokenizer privacy successor was promoted through the private operational source with controlled restart and rollback gates. The mounted hash matched the approved successor, the operational release comparison read 293 expected paths with zero mismatches, and one authenticated non-sensitive DSH -> LiteLLM -> DeepInfer smoke request returned the exact expected text without retry. That verifies the narrow release and route only; it does not qualify peer-copy, DeepEP, host changes, performance, or complete source-to-binary reproduction.

Current technical evidence

The single canonical technical note is the GPU serving comparison and administrator change plan. It contains:

• the direct-engine Test 1/Test 2/Test 3 method and complete measured matrix (raw retained IDs W1/W3/W4; W2 was not executed);
• a three-row executive table for measured SGLang, measured DeepInfer/NCCL, and a fully populated conditional DeepInfer/DeepEP sensitivity row;
• two separate apples-to-apples one-million-token tables for both engines: a bounded end-instruction check and a middle-retrieval/long-output check, each with a fully labelled conditional post-change sensitivity row;
• short-context and long-context graphs with machine-readable data;
• retained host, engine, LiteLLM, and context-forwarder readbacks;
• the distinction between measured results and the conditional 76.3-146.7 tok/s W1 host + DeepEP scenario, which is not an expected iommu=pt gain;
• the host change, integrity, restoration, and acceptance procedure;
• the relationship to relevant DeepInfer/DeepEP/Humming/Hopper references.

Graph and data index · SGLang + DSpark alternate · retained DeepInfer + DSpark/NCCL profile · proposed, unqualified DeepEP profile · host/recovery runbook.

The measured matrix used short, direct engine-local HTTP cohorts, not the authenticated DSH route, P99, or an upstream steady-state sweep. It therefore demonstrates neither complete product acceptance nor live parity. Two separate direct-engine matched one-million-token checks used matched payload construction, model snapshot and settings for both engines. The proposed DeepEP profile is not deployed or qualified. Current readiness and endpoint evidence is recovery evidence only; it is not peer-copy, DeepEP, full-model correctness, authenticated-path, restart, or complete source-to-binary qualification. No host reboot, IOMMU/ACS/BIOS change, runtime restart, model call, or post-change speedup is claimed.

Prompt and concurrency scope is explicit in the report: the matched short matrix used 18-26 input / 119-512 output tokens at offered C=1/2/4/8/16 with active cap 8. The first C1 check used exactly 1,000,000 server-reported prompt tokens and nine output tokens for each engine. SGLang measured 1,407.020 s TTFT / 1,407.350 s elapsed; DeepInfer/NCCL measured 318.888 s TTFT / 318.995 s elapsed. Both returned HTTP 200 and the exact required response. Machine-readable bounded results.

The second C1 check also used exactly 1,000,000 server-reported prompt tokens, with a value embedded in the middle and a required long structured answer. Both engines returned HTTP 200, recovered the middle value and passed the long-output rule. DeepInfer produced 1,024 tokens at 328.608 s TTFT / 344.360 s elapsed / 64.945 decode tok/s; SGLang produced 981 tokens at 1,481.680 s TTFT / 1,495.694 s elapsed / 69.933 decode tok/s. Machine-readable middle/long results and the long-context graph are published. Each 1M experiment is one direct-engine run per engine; neither is cache-controlled, multi-user, statistical, authenticated-route, or post-change evidence.

The report now states the bounded evidence chain explicitly: NVIDIA documents translated-IOMMU risk for Linux bare-metal PCIe peer transfer; Linux documents passthrough/bypass semantics; generic NVIDIA multi-GPU guidance uses iommu=pt when IOMMU remains enabled; and DeepEP requires working GPU communication paths but does not make iommu=pt a DeepInfer/DSpark/DeepEP requirement. The 1.32-2.54x range applies only if peer integrity, DeepEP qualification and the assumed expert-path timing all succeed. Applying it across the executive table produces sensitivity bounds only, not measured results or multi-user forecasts.

DeepEP does not read or configure iommu=pt. The relationship is indirect: the host argument changes the lower DMA-translation policy; that may restore a byte-correct GPU peer path; DeepEP can then be tested on that path for dispatch/combine, model correctness and performance. It is not automatic DeepEP enablement and does not itself establish a speedup.

The operational plan uses one technical maintenance session with staged configurations: measure iommu=pt alone first; only if the agreed gates are still missed, test a pre-reviewed ACS or BIOS combination as a separate recorded configuration. Every stage retains an immediate route back to the known-working boot and service configuration.

Governance controls and open integration questions

The intended product boundary remains: external human authentication at the gateway; per-user/workspace separation; an internal LiteLLM policy and model catalog; final-payload context admission; explicit cancellation; retained audit evidence; and operator-controlled model-engine changes. Directory separation alone is not a security boundary. Source checks, unit tests, readiness, and a backend answer are not substitutes for authenticated end-to-end acceptance.

Open upstream design questions include supported extension points for managed model catalogs, external identity, usage accounting, long-context admission, cancellation, durable workspace recovery, and governed agent/tool integration. The authenticated /agent-tasks broker is deployed for the managed DSH Python SDK, Claude Code and Deep Agents paths. Each runtime now has one bounded internal Test marker with exact result and guard/server/spend-log identity reconciliation; this is not full authenticated delegation, approval/resume, arbitrary tool execution, cancellation/recovery, immutable-image provenance, Production qualification, or OpenShell acceptance. OpenShell remains disabled in the live UI and has not passed its 25 live product gates. Kilo and TrueForge remain governed clients rather than substitutes for the broker controls. Every runtime/backend combination must pass the same identity, isolation, authorization, cancellation, network, audit and recovery gates before being described as operational.

The canonical transport naming gap is compatibility-complete at the Service layer. Current public dsh_setup main is exact revision b7dd7d5; public-source run 41548909262 passed that exact revision. The exact-revision upstream-candidate run 41548909206 also completed successfully; it proves only the checked upstream-candidate workflow and does not establish private runtime parity or live OpenShell acceptance. Portable managed-client defaults use litellm-gateway, and portable application-hostname configuration uses dsh-gateway; retained compatibility selectors continue to name historical immutable workloads until their consumers and rollback references are migrated. The public README, GPU profiles, runbook and hosted report now distinguish the private 293/293 release receipt from the later public DeepInfer successor. The Production managed transport remains an immutable ConfigMap sourced from private release 946a17593a2c541e26c80b251867b37292fea21b; its predecessor remains the rollback object. No new model request, GPU-engine restart, host change, DeepEP activation or benchmark was performed by this documentation/provenance update.

This body replaces earlier implementation/status narratives. Older comments in this discussion are being reduced to explicit superseded pointers so they cannot be mistaken for current configuration or deployment instructions.

Related H20 community release (separate evidence)

A later PR #92 comment links the Apache-2.0 OpenAI-Insights/deepinfer-h20-release, which packages an 8 x H20-3e, DP4 / CP2 / EP8 integration of PRs #92, #144 and #147 with an optional PR #246-inspired local Engram fallback. This is useful implementation evidence, but its hardware, topology, GPU count, parallelism and workload differ from our 4 x H100 NVL, DP4 / EP4, NCCL measurements. It therefore does not change our measured or conditional numeric rows and is not evidence that DeepEP, iommu=pt, ACS/BIOS or peer-copy integrity is qualified on our host. This distinction is included in the hosted report at exact dsh_setup commit 88aef797.

[skunpoj]

Current verified status; earlier comment history superseded

The single current report is https://github.com/skunpoj/dsh_setup/blob/main/docs/gpu-platform/GPU_PLATFORM_REVIEW.pdf. Supporting material is the graph and data index, source ownership map, and host-change and recovery runbook.

The controlled count-only tokenizer privacy successor was released through the private operational source. Its release receipt recorded 293 expected mounted Python paths, 293 read paths and zero mismatches; after recovery, one authenticated, non-sensitive DSH to LiteLLM to DeepInfer request returned the exact expected text with HTTP 200 and no retry. That receipt proves the private release against its own approved source. A separate comparison with the later public DeepInfer successor c1b8cc6 found 17 exact-byte matches, 274 line-ending-only differences, two newer public fixes not yet mounted, and no missing Python path. It must not be described as zero-mismatch parity with that later public commit.

This verifies the narrow tokenizer privacy release and authenticated route only. The current DeepInfer serving path continues to use DSpark with the NCCL expert-communication backend. No host reboot, iommu=pt, ACS/BIOS change, DeepEP activation, peer-copy qualification, performance benchmark, or complete source-to-binary reproduction is claimed. The public profiles are sanitized references; private deployment identities, credentials, endpoints, and operational history are not published.

Related H20 community release (separate evidence)

A later PR #92 comment links the Apache-2.0 OpenAI-Insights/deepinfer-h20-release, which packages an 8 x H20-3e, DP4 / CP2 / EP8 integration of PRs #92, #144 and #147 with an optional PR #246-inspired local Engram fallback. This is useful implementation evidence, but its hardware, topology, GPU count, parallelism and workload differ from our 4 x H100 NVL, DP4 / EP4, NCCL measurements. It therefore does not change our measured or conditional numeric rows and is not evidence that DeepEP, iommu=pt, ACS/BIOS or peer-copy integrity is qualified on our host. This distinction is included in the hosted report at exact dsh_setup commit 88aef797.