refactor(inference): simplify routing — introduce inference.local, remove implicit catch-all

[pimlock]

Summary

Closes #133

Simplifies the inference routing system from a multi-route CRUD model to a single managed cluster route (inference.local). This is a large architectural cleanup that removes ~1,750 lines of dead/unnecessary code while making the system easier to reason about.

Key changes

• Remove inference route CRUD: Delete Create/Update/Delete/ListInferenceRoutes RPCs, CLI commands, and all supporting code. Cluster inference is now configured solely via nemoclaw cluster inference set/get/update.
• Remove InspectForInference OPA action: Policy evaluation is now binary allow/deny. Inference routing is triggered explicitly by CONNECT to inference.local, not by an implicit catch-all on unmatched connections.
• Provider-agnostic router: Introduce AuthHeader enum and InferenceProviderProfile in navigator-core. The router no longer checks provider types — auth style (Bearer vs custom header) and default headers are carried on ResolvedRoute.
• Proto cleanup: Rename routing_hint → name, SandboxResolvedRoute → ResolvedRoute, GetSandboxInferenceBundle → GetInferenceBundle. Replace InferenceRouteSpec (8 fields, 6 always empty) with ClusterInferenceConfig (2 fields: provider_name, model_id). Drop unused sandbox_id parameter from bundle request.
• Dead code removal: Delete stale checked-in navigator.inference.v1.rs (~800 lines), remove InferencePolicy/InferenceApiPattern from proto and policy crate, clean up OPA Rego rules.
• Add nemoclaw cluster inference update: New CLI command for partial updates (get-then-set) — change provider or model without re-specifying the full config.
• Architecture docs: Updated sandbox.md, README.md, gateway.md, inference-routing.md, security-policy.md to reflect new design.

Configuration flow

# 1. Create a provider with credentials
nemoclaw provider create --name nvidia-build --type nvidia --from-existing

# 2. Set cluster-level inference (provider + model)
nemoclaw cluster inference set --provider nvidia-build --model meta/llama-3.1-8b-instruct

# 3. Verify configuration
nemoclaw cluster inference get
# provider: nvidia-build
# model:    meta/llama-3.1-8b-instruct
# version:  1

# 4. Switch models without re-specifying the provider
nemoclaw cluster inference update --model meta/llama-3.3-70b-instruct

# 5. Switch to a different provider entirely
nemoclaw provider create --name openai-prod --type openai --from-existing
nemoclaw cluster inference set --provider openai-prod --model gpt-4o

Using inference from a sandbox

Important: inference.local only works over HTTPS. The sandbox proxy intercepts
HTTPS CONNECT requests to inference.local and routes them to the configured backend.
Plain HTTP requests are not intercepted.

# Inside any sandbox — no credentials needed, model is overridden by cluster config
curl https://inference.local/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anything",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
# → routed to the configured provider with injected API key
# → "model" field is rewritten to the cluster-configured model

The proxy intercepts the CONNECT tunnel to inference.local, performs a TLS handshake internally, parses the HTTP request inside the tunnel, and forwards it to the configured upstream provider with the correct credentials and model override.

Plan

See the plan for more details.

Follow-up

Deferred items tracked in #148:

• Default NVIDIA inference fallback when no cluster config is set
• Inject inference.local instructions into sandboxes for agent discovery

Related issues

Closes #21
Closes #20

[pimlock]

One design question to think through: should provider lifecycle be coupled to cluster inference configuration?

Right now SetClusterInference stores only a reference to the provider. That means if the referenced provider is later deleted, inference requests will fail at bundle resolution / request time.

The alternative would be to snapshot provider details when inference is configured, but that has a downside too: provider fixes would stop propagating automatically. For example, if I create a provider with an invalid API key, then correct the key on the provider, I would also need to re-run nemoclaw cluster inference set for inference to start working again.

Curious whether we want to:

1. prevent deletion of providers that are currently referenced by cluster inference,
2. allow deletion but make the failure mode / UX explicit, or
3. snapshot provider config into inference and accept the loss of automatic propagation.

Feels like option 1 or 2 is probably cleaner than snapshotting, but wanted to call out the tradeoff explicitly.

---

There is a similar consideration for providers that are used with sandboxes -> once we are injecting things on the fly, is an update to a provider applied to any running sandboxes?

[pimlock]

Migration skill (agent guide for upgrading existing clusters) is on a separate branch: https://github.com/NVIDIA/NemoClaw/blob/feat/migrate-inference-routing-skill/.agents/skills/migrate-inference-routing/SKILL.md

[johntmyers]

This is great!

Re: your question:

prevent deletion of providers that are currently referenced by cluster inference,

I kind of lean that way.

Does this same problem exist for providers that are in use by a sandbox too?

[drew]

somewhat arbitrarily starting a thread to discuss

prevent deletion of providers that are currently referenced by cluster inference,

I'm inclined to punt on this for now, and then solve referential integrity across the platform as a separate effort. could probably be a post gtc thing.

[pimlock]

From @johntmyers above

I kind of lean that way.
Does this same problem exist for providers that are in use by a sandbox too?

Will capture this as a ticket.