fix(nvcf-self-managed-stack): set the LLM request router worker address - #653
Closed
vrv3814 wants to merge 3 commits into
Closed
fix(nvcf-self-managed-stack): set the LLM request router worker address#653vrv3814 wants to merge 3 commits into
vrv3814 wants to merge 3 commits into
Conversation
The self-managed stack never emitted api.remoteConfig.configData, so nvcf.llm-request-router.worker-address stayed at the helm-nvcf-api chart default of "". NVCA resolves the LLM request router only from the worker environment and fails fast when it is unset, so every LLM function deploy aborted during launch-spec translation with "LLM request router address is not set" and no worker pod was created. Emit the field from the api release: default to the in-cluster request router at llm-request-router.nvcf.svc.cluster.local:50071 when the LLM addon is enabled, and honor a new global.workerEndpoints.llmRequestRouterAddress override for compute planes that run outside the control plane cluster. Add an offline render test that asserts all four cases through the real api release, plus a test target on the stack Makefile. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: vemireddyv <vemireddyv@nvidia.com>
📝 WalkthroughWalkthroughThe self-managed stack now supports configurable LLM request-router worker addresses, external gRPC and QUIC dialing, optional Gateway API exposure, and advertised per-pod hostnames. Helm templates, documentation, and offline rendering tests cover default, enabled, disabled, and invalid configurations. ChangesLLM request-router exposure
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant StackConfig
participant APIremoteConfig
participant RouterDeployment
participant GatewayRoutesChart
participant ExternalPylon
StackConfig->>APIremoteConfig: publish worker router address
StackConfig->>RouterDeployment: configure external dial addresses and hostname
RouterDeployment->>ExternalPylon: establish gRPC and QUIC connections
StackConfig->>GatewayRoutesChart: configure Gateway listeners and routes
GatewayRoutesChart->>ExternalPylon: forward gRPC and QUIC traffic
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Comment |
Contributor
…nal compute planes Compute planes outside the control plane cluster cannot resolve llm-request-router.nvcf.svc.cluster.local, and nothing in the stack exposed the router. Setting only the worker address pointed workers at an unreachable name. Pylon uses two channels: a gRPC control channel on the router TCP port, and a QUIC reverse tunnel on the router UDP port after the router issues a tunnel target. Both have to reach the router, so exposure needs routes for each plus addresses the router can advertise. Add to the gateway-routes chart an opt-in TCPRoute for port 50071, a UDPRoute for port 50072, and a shared ReferenceGrant, under nvcfGatewayRoutes.routes.llmRequestRouter. The stack gates them on addons.llm.enabled so a route never references a Service that was not deployed, and requires the Gateway name and namespace when they are enabled. Add to the request router chart transport.grpcPylonDialAddr and transport.reverseTunnelPylonDialAddr for the shared addresses pylon dials. Both default to empty and keep the existing in-cluster behavior, including the POD_IP reverse tunnel address at replicaCount above 1. Surface them plus kubernetes.advertisedHostnameTemplate through addons.llm.requestRouter.external. The advertised hostname stays the per-pod gRPC authority and QUIC SNI. These routes forward on port alone, so they suit a single replica; above one, the front end must route on that identity. Documented in the chart READMEs and in the LLM function enablement guide, which also drops an earlier claim that setting the worker address alone was enough for split-cluster. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: vemireddyv <vemireddyv@nvidia.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@deploy/stacks/self-managed/tests/llm-request-router-external-exposure.sh`:
- Around line 117-126: Update the enabled-render assertions in the test loop
around the gateway manifest to verify the specific LLM resource names for the
TCPRoute, UDPRoute, and ReferenceGrant, rather than only their kinds; ensure
`llm-request-router-grpc`, `llm-request-router-quic`, and
`allow-routes-to-llm-request-router` are each present. Update the
disabled-render check to assert none of those three named resources render when
the router is disabled, while preserving the existing QUIC listener attachment
assertion.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 6b94d6e7-056e-44d6-894f-10a23eeeaa2b
📒 Files selected for processing (14)
deploy/helm/gateway-routes/README.mddeploy/helm/gateway-routes/chart/templates/referencegrant-llm-request-router.yamldeploy/helm/gateway-routes/chart/templates/tcproute-llm-request-router.yamldeploy/helm/gateway-routes/chart/templates/udproute-llm-request-router.yamldeploy/helm/gateway-routes/chart/values.yamldeploy/helm/llm-request-router/README.mddeploy/helm/llm-request-router/llm-request-router/templates/deployment.yamldeploy/helm/llm-request-router/llm-request-router/values.yamldeploy/stacks/self-managed/Makefiledeploy/stacks/self-managed/environments/base.yamldeploy/stacks/self-managed/global.yaml.gotmpldeploy/stacks/self-managed/tests/llm-request-router-external-exposure.shdocs/user/csp-end-to-end-example-installation.mddocs/user/llm-function-enablement.md
🚧 Files skipped from review as they are similar to previous changes (2)
- docs/user/csp-end-to-end-example-installation.md
- deploy/stacks/self-managed/Makefile
…s a GRPCRoute The pylon control channel is gRPC over h2c, so a TCPRoute was the wrong shape for it. Pylon dials the shared address handed to workers but sets the router's advertised per-pod hostname as the request authority, which is exactly what a GRPCRoute matches on. Replace the TCPRoute with a GRPCRoute carrying hostnames. It attaches to the shared Gateway like the nvcf-api and nvct-api gRPC routes, so the control channel no longer needs its own TCP listener. Only the QUIC reverse tunnel still needs a dedicated UDP listener, so gateways.llmRequestRouter becomes gateways.llmRequestRouterQuic and covers just that hop. This also gives the control channel a path to multi-replica: hostname matching can reach a specific router pod, which port-based forwarding cannot. The QUIC hop is unchanged and still forwards on port alone, so it remains the reason external installs should stay at one replica. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: vemireddyv <vemireddyv@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@deploy/helm/gateway-routes/chart/values.yaml`:
- Around line 254-284: Add Helm validation for the llmRequestRouter
configuration so enabling ingress.gatewayApi.routes.llmRequestRouter with
addons.llm.requestRouter.replicaCount greater than one fails rendering unless
SNI-aware per-pod routing is configured; preserve the existing single-replica
behavior. Add a regression test covering the invalid multi-replica combination.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f078e293-ec51-4e83-8e40-859f739c6da7
📒 Files selected for processing (9)
deploy/helm/gateway-routes/README.mddeploy/helm/gateway-routes/chart/templates/grpcroute-llm-request-router.yamldeploy/helm/gateway-routes/chart/templates/referencegrant-llm-request-router.yamldeploy/helm/gateway-routes/chart/templates/udproute-llm-request-router.yamldeploy/helm/gateway-routes/chart/values.yamldeploy/stacks/self-managed/environments/base.yamldeploy/stacks/self-managed/global.yaml.gotmpldeploy/stacks/self-managed/tests/llm-request-router-external-exposure.shdocs/user/llm-function-enablement.md
🚧 Files skipped from review as they are similar to previous changes (5)
- deploy/stacks/self-managed/global.yaml.gotmpl
- deploy/helm/gateway-routes/README.md
- deploy/stacks/self-managed/environments/base.yaml
- deploy/stacks/self-managed/tests/llm-request-router-external-exposure.sh
- docs/user/llm-function-enablement.md
Comment on lines
+254
to
+284
| # LLM request router routes for compute planes outside the control plane | ||
| # cluster. Pylon opens a gRPC control channel, registers, receives a tunnel | ||
| # target, then opens a QUIC reverse tunnel. Both hops are required. | ||
| # Disabled by default because most installs keep the router in-cluster. | ||
| # | ||
| # The gRPC hop is a GRPCRoute on the shared Gateway, matched by hostname. | ||
| # Pylon sets the router's advertised per-pod hostname as the request | ||
| # authority, so hostnames here must line up with the router's | ||
| # advertisedHostnameTemplate. | ||
| # | ||
| # The QUIC hop is a UDPRoute and forwards on port alone. At replicaCount 1 | ||
| # that reaches the only router pod. Above 1 the router expects the reverse | ||
| # tunnel to come back to the pod named by the QUIC SNI, which a UDPRoute | ||
| # cannot honor. | ||
| llmRequestRouter: | ||
| enabled: false | ||
| grpc: | ||
| name: llm-request-router-grpc | ||
| hostnames: | ||
| - "llm-router.{{ .Values.nvcfGatewayRoutes.domain }}" | ||
| backend: | ||
| name: llm-request-router | ||
| namespace: nvcf | ||
| port: 50071 | ||
| quic: | ||
| name: llm-request-router-quic | ||
| backend: | ||
| name: llm-request-router | ||
| namespace: nvcf | ||
| port: 50072 | ||
| routeAnnotations: {} |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
# Locate the router replica setting and any validation that couples it to
# external Gateway exposure.
rg -n -C 8 \
'llmRequestRouter|llm-request-router|replicaCount|UDPRoute|external.*exposure' \
deploy/stacks/self-managed deploy/helm
# Expect: a rendered-config test or template validation rejects enabled
# llmRequestRouter routes when the router replica count is greater than one.Repository: NVIDIA/nvcf
Length of output: 50370
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- guidance ---'
find .. -name AGENTS.md -print
printf '%s\n' '--- relevant files ---'
git ls-files \
'deploy/helm/llm-request-router/**' \
'deploy/helm/gateway-routes/**' \
'deploy/stacks/self-managed/**' | \
grep -E '(^|/)(values|Chart|templates|README|test|schema|.*stack.*)\.' | \
head -200
printf '%s\n' '--- exact references ---'
rg -n -C 5 \
'llmRequestRouter|llm-request-router|advertisedHostnameTemplate|replicaCount' \
deploy/stacks/self-managed deploy/helm/llm-request-router deploy/helm/gateway-routes \
-g '!README.md' -g '!*.lock' -g '!charts/**' | \
head -500Repository: NVIDIA/nvcf
Length of output: 50367
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- repository guidance ---'
cat AGENTS.md 2>/dev/null || true
printf '%s\n' '--- stack router release ---'
sed -n '155,190p' deploy/stacks/self-managed/helmfile.d/02-core.yaml.gotmpl
printf '%s\n' '--- stack router and route value propagation ---'
sed -n '250,330p' deploy/stacks/self-managed/global.yaml.gotmpl
printf '%s\n' '--- router chart values ---'
sed -n '1,190p' deploy/helm/llm-request-router/llm-request-router/values.yaml
printf '%s\n' '--- router chart validation and replica templates ---'
rg -n -C 12 \
'fail|required|replicaCount|disableDnsDiscovery|advertisedHostnameTemplate|StatefulSet' \
deploy/helm/llm-request-router/llm-request-router/templates \
deploy/helm/llm-request-router/llm-request-router/values.yaml
printf '%s\n' '--- route chart templates ---'
git ls-files deploy/helm/gateway-routes/chart/templates | xargs -r -n1 sh -c '
echo "--- $0"
sed -n "1,240p" "$0"
'
printf '%s\n' '--- external exposure test ---'
cat deploy/stacks/self-managed/tests/llm-request-router-external-exposure.shRepository: NVIDIA/nvcf
Length of output: 50370
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- all relevant value declarations and uses ---'
rg -n \
'(^|[^A-Za-z])llmRequestRouter([^A-Za-z]|$)|(^|[^A-Za-z])replicaCount([^A-Za-z]|$)|UDPRoute|GRPCRoute' \
deploy/stacks/self-managed deploy/helm/gateway-routes deploy/helm/llm-request-router \
-g '*.yaml' -g '*.gotmpl' -g '*.tpl' -g '*.sh' |
grep -vE 'README|charts/' |
head -400
printf '%s\n' '--- router StatefulSet template ---'
find deploy/helm/llm-request-router/llm-request-router/templates -maxdepth 1 -type f -print |
sort |
while read -r file; do
if grep -qE 'replicaCount|disableDnsDiscovery|advertisedHostnameTemplate|fail ' "$file"; then
echo "--- $file"
nl -ba "$file" | grep -C 18 -E 'replicaCount|disableDnsDiscovery|advertisedHostnameTemplate|fail '
fi
done
printf '%s\n' '--- Gateway route templates ---'
find deploy/helm/gateway-routes/chart/templates -maxdepth 1 -type f -print |
sort |
while read -r file; do
if grep -qE 'llmRequestRouter|UDPRoute|GRPCRoute' "$file"; then
echo "--- $file"
nl -ba "$file" | grep -C 20 -E 'llmRequestRouter|UDPRoute|GRPCRoute'
fi
done
printf '%s\n' '--- focused external-exposure test ---'
nl -ba deploy/stacks/self-managed/tests/llm-request-router-external-exposure.sh |
sed -n '1,180p'Repository: NVIDIA/nvcf
Length of output: 36228
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- stack router values ---'
sed -n '680,750p' deploy/stacks/self-managed/global.yaml.gotmpl
printf '%s\n' '--- stack Gateway route values ---'
sed -n '1060,1180p' deploy/stacks/self-managed/global.yaml.gotmpl
printf '%s\n' '--- self-managed defaults ---'
sed -n '225,270p' deploy/stacks/self-managed/environments/base.yaml
sed -n '375,425p' deploy/stacks/self-managed/environments/base.yaml
printf '%s\n' '--- router deployment guard and replica wiring ---'
sed -n '20,115p' deploy/helm/llm-request-router/llm-request-router/templates/deployment.yaml
printf '%s\n' '--- Gateway route templates ---'
sed -n '1,90p' deploy/helm/gateway-routes/chart/templates/grpcroute-llm-request-router.yaml
sed -n '1,85p' deploy/helm/gateway-routes/chart/templates/udproute-llm-request-router.yaml
sed -n '1,70p' deploy/helm/gateway-routes/chart/templates/referencegrant-llm-request-router.yaml
printf '%s\n' '--- all cross-value fail checks ---'
grep -R -n -E 'fail |required .*llmRequestRouter|replicaCount.*llmRequestRouter|llmRequestRouter.*replicaCount' \
deploy/stacks/self-managed deploy/helm/gateway-routes deploy/helm/llm-request-router \
--include='*.yaml' --include='*.gotmpl' --include='*.tpl' --include='*.sh' || trueRepository: NVIDIA/nvcf
Length of output: 39316
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
import re
global_template = Path("deploy/stacks/self-managed/global.yaml.gotmpl").read_text()
router_deployment = Path(
"deploy/helm/llm-request-router/llm-request-router/templates/deployment.yaml"
).read_text()
grpc_route = Path(
"deploy/helm/gateway-routes/chart/templates/grpcroute-llm-request-router.yaml"
).read_text()
udp_route = Path(
"deploy/helm/gateway-routes/chart/templates/udproute-llm-request-router.yaml"
).read_text()
exposure_test = Path(
"deploy/stacks/self-managed/tests/llm-request-router-external-exposure.sh"
).read_text()
checks = {
"stack maps an independent router replica value": (
'replicaCount: {{ dig "addons" "llm" "requestRouter" "replicaCount" 3 .Values }}'
in global_template
),
"stack maps an independent route enable value": (
'$llmRequestRouterRouteEnabled := dig "ingress" "gatewayApi" "routes" '
'"llmRequestRouter" "enabled" false .Values'
in global_template
),
"stack does not contain a combined route/replica guard": not re.search(
r"(llmRequestRouterRouteEnabled|routes.*llmRequestRouter).*"
r"(replicaCount|requestRouter)",
global_template,
re.DOTALL,
),
"GRPCRoute gates only on route enablement": (
"and .Values.nvcfGatewayRoutes.enabled "
".Values.nvcfGatewayRoutes.routes.llmRequestRouter.enabled"
in grpc_route
),
"UDPRoute gates only on route enablement": (
"and .Values.nvcfGatewayRoutes.enabled "
".Values.nvcfGatewayRoutes.routes.llmRequestRouter.enabled"
in udp_route
),
"router chart guard does not mention Gateway routes": (
"llmRequestRouter.discovery.disableDnsDiscovery" in router_deployment
and "gatewayApi.routes" not in router_deployment
and "UDPRoute" not in router_deployment
),
"external exposure test has no replica-count rejection case": (
"replicaCount" not in exposure_test
),
}
for name, result in checks.items():
print(f"{'PASS' if result else 'FAIL'}: {name}")
if not all(checks.values()):
raise SystemExit(1)
PYRepository: NVIDIA/nvcf
Length of output: 528
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
global_lines = Path("deploy/stacks/self-managed/global.yaml.gotmpl").read_text().splitlines()
router_lines = Path(
"deploy/helm/llm-request-router/llm-request-router/templates/deployment.yaml"
).read_text().splitlines()
grpc_route = Path(
"deploy/helm/gateway-routes/chart/templates/grpcroute-llm-request-router.yaml"
).read_text()
udp_route = Path(
"deploy/helm/gateway-routes/chart/templates/udproute-llm-request-router.yaml"
).read_text()
exposure_test = Path(
"deploy/stacks/self-managed/tests/llm-request-router-external-exposure.sh"
).read_text()
def window(lines, start_marker, end_marker):
start = next(i for i, line in enumerate(lines) if start_marker in line)
end = next(i for i in range(start + 1, len(lines)) if end_marker in lines[i])
return lines[start:end]
router_block = window(
global_lines,
"llmRequestRouter:",
"llmApiGateway:",
)
route_block = window(
global_lines,
"$llmRequestRouterRouteEnabled",
"$nvcfApiGrpcRouteHostnames",
)
fail_windows = []
for index, line in enumerate(global_lines):
if "fail " in line or "required " in line:
context = "\n".join(global_lines[max(0, index - 5): index + 6])
if "llmRequestRouterRouteEnabled" in context and (
"replicaCount" in context or "requestRouter" in context
):
fail_windows.append(context)
checks = {
"router replica value is independent of route enablement": (
any("requestRouter" in line and "replicaCount" in line for line in router_block)
and any("llmRequestRouterRouteEnabled" in line for line in route_block)
),
"stack has no bounded route/replica validation window": not fail_windows,
"GRPCRoute gates only on route enablement": (
"and .Values.nvcfGatewayRoutes.enabled "
".Values.nvcfGatewayRoutes.routes.llmRequestRouter.enabled"
in grpc_route
),
"UDPRoute gates only on route enablement": (
"and .Values.nvcfGatewayRoutes.enabled "
".Values.nvcfGatewayRoutes.routes.llmRequestRouter.enabled"
in udp_route
),
"router chart has no Gateway-route validation": (
"gatewayApi.routes" not in "\n".join(router_lines)
and "UDPRoute" not in "\n".join(router_lines)
),
"external exposure test has no replica-count rejection case": (
"replicaCount" not in exposure_test
),
}
for name, result in checks.items():
print(f"{'PASS' if result else 'FAIL'}: {name}")
if not all(checks.values()):
if fail_windows:
print("\nUnexpected candidate validation windows:")
print("\n---\n".join(fail_windows))
raise SystemExit(1)
PYRepository: NVIDIA/nvcf
Length of output: 274
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
global_text = Path("deploy/stacks/self-managed/global.yaml.gotmpl").read_text()
global_lines = global_text.splitlines()
router_deployment = Path(
"deploy/helm/llm-request-router/llm-request-router/templates/deployment.yaml"
).read_text()
grpc_route = Path(
"deploy/helm/gateway-routes/chart/templates/grpcroute-llm-request-router.yaml"
).read_text()
udp_route = Path(
"deploy/helm/gateway-routes/chart/templates/udproute-llm-request-router.yaml"
).read_text()
exposure_test = Path(
"deploy/stacks/self-managed/tests/llm-request-router-external-exposure.sh"
).read_text()
candidate_windows = []
for index, line in enumerate(global_lines):
if "fail " in line or "required " in line:
context = "\n".join(global_lines[max(0, index - 8): index + 9])
if (
"llmRequestRouterRouteEnabled" in context
and "replicaCount" in context
):
candidate_windows.append(context)
checks = {
"stack emits an independent router replica value": (
'replicaCount: {{ dig "addons" "llm" "requestRouter" "replicaCount" 3 .Values }}'
in global_text
),
"stack computes an independent route enable value": (
'$llmRequestRouterRouteEnabled := dig "ingress" "gatewayApi" "routes" '
'"llmRequestRouter" "enabled" false .Values'
in global_text
),
"stack has no route/replica validation window": not candidate_windows,
"GRPCRoute gates only on route enablement": (
"and .Values.nvcfGatewayRoutes.enabled "
".Values.nvcfGatewayRoutes.routes.llmRequestRouter.enabled"
in grpc_route
),
"UDPRoute gates only on route enablement": (
"and .Values.nvcfGatewayRoutes.enabled "
".Values.nvcfGatewayRoutes.routes.llmRequestRouter.enabled"
in udp_route
),
"router chart has no Gateway-route validation": (
"gatewayApi.routes" not in router_deployment
and "UDPRoute" not in router_deployment
),
"external exposure test has no replica-count rejection case": (
"replicaCount" not in exposure_test
),
}
for name, result in checks.items():
print(f"{'PASS' if result else 'FAIL'}: {name}")
if not all(checks.values()):
if candidate_windows:
print("\nCandidate validation windows:")
print("\n---\n".join(candidate_windows))
raise SystemExit(1)
PYRepository: NVIDIA/nvcf
Length of output: 518
Add stack validation for multi-replica external router routes
When ingress.gatewayApi.routes.llmRequestRouter.enabled is true and addons.llm.requestRouter.replicaCount is greater than one, fail rendering or require SNI-aware per-pod routing. Add a regression test.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deploy/helm/gateway-routes/chart/values.yaml` around lines 254 - 284, Add
Helm validation for the llmRequestRouter configuration so enabling
ingress.gatewayApi.routes.llmRequestRouter with
addons.llm.requestRouter.replicaCount greater than one fails rendering unless
SNI-aware per-pod routing is configured; preserve the existing single-replica
behavior. Add a regression test covering the invalid multi-replica combination.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
On a self-managed install, every LLM function deploy failed during launch-spec
translation and no worker pod was created. The stack never emitted
api.remoteConfig.configData, sonvcf.llm-request-router.worker-addressstayedat the
helm-nvcf-apichart default of"". This emits the field from theapirelease and adds an override for split-cluster installs.
Additional Details
NVCA resolves the LLM request router address only from the worker environment
(
LLM_REQUEST_ROUTER_ADDRESS, withSTARGATE_ADDRESSas a legacy alias) andfails fast when neither is set. That fail-fast is deliberate: the address is a
control-plane concern that must ride down from nvcf-api, so NVCA carries no
built-in default.
Nothing in the self-managed stack supplied it.
global.yaml.gotmplhad noremoteConfigpassthrough on theapirelease, leaving the chart default empty.Deploys ended as
ERRORwith the agent logging:and the
ICMSRequestinnvcf-backendsitting atRequestFailureAcknowledged.How the pieces connect:
global.yaml.gotmplnow emitsapi.remoteConfig.configData.nvcf.llm-request-router.worker-address. Itdefaults to
llm-request-router.nvcf.svc.cluster.local:50071, matchingllmRequestRouter.service.grpcPortand the request router release namespace,whenever
addons.llm.enabledis true.global.workerEndpoints.llmRequestRouterAddressoverrides that default andapplies regardless of the addon flag. Compute planes outside the control plane
cluster cannot resolve the in-cluster service name, so they need a reachable
address here. It sits alongside the existing worker endpoint values, which serve
the same purpose for the other worker-facing services.
default is untouched and the stack does not advertise a router it never
deployed.
Docs:
docs/user/llm-function-enablement.mdgains a "Request Router WorkerAddress" section and a troubleshooting entry for the terminal error;
docs/user/csp-end-to-end-example-installation.mdlists the new key in itsmulti-cluster
workerEndpointssample.Summary by CodeRabbit
New Features
Documentation
Tests