Proposal: user-defined model policy, complementary to provider entitlement filtering

[OPaimon]

Problem

DSH has two different model-availability questions:

1. Capability / entitlement: can the configured credential use this model?
2. Policy: does this user or deployment permit DSH to use this model?

Provider-owned catalog filtering can answer the first question. It cannot answer the second.

Examples of user policy:

• prevent unattended subagents or background tasks from selecting costly routes;
• keep compliance-sensitive code away from a provider while retaining its configuration;
• quarantine a broken, deprecated, misleading, or temporarily unreliable model;
• remove unwanted catalog noise without deleting credentials;
• deny one model id across every provider route.

Today the only built-in hard-disable lever at the model layer is removing a provider route configuration. That is destructive, provider-wide, and conflates temporary policy with configuration lifecycle.

Relationship to #1806

#1806 identifies a real adapter defect: pi-ai catalog providers may define credential-specific filterModels, but in the DSH 0.1.1-rc.2 path investigated there, the pi-ai route wrapper does not preserve that hook and PiAiAdapter.listModels() reads the complete catalog through getModels().

That should be fixed. It answers:

Which models did this provider credential entitle the account to use?

This proposal answers a different question:

Which otherwise usable models does this user or deployment permit DSH to use?

The two concerns should compose as an intersection:

provider/credential availability ∩ user policy

	Provider entitlement filtering	User-defined policy
Decision owner	Provider / credential	User / administrator
Source	OAuth or provider metadata, such as availableModelIds	Persistent DSH settings
Scope	One provider implementation	Every DSH adapter and route
Behavior	Automatic capability filtering	Explicit deny policy
Dispatch enforcement	Not implied by catalog filtering	Required

Even after entitlement filtering is wired correctly, it cannot express “this model is valid for my account but must not be used here.”

Proposed semantics

A settings-backed policy with:

• disabledProviders for complete provider routes;
• exact provider/model entries;
• */model rules for one model id across all providers.

It is enforced at two distinct faces:

1. Discovery filtering. Subtract denied entries from LlmRuntime.listProviders() and listModels(provider), which feed model selectors, catalog RPCs, task-model-style tools, and introspection surfaces.
2. Dispatch enforcement. Because DSH model catalogs are advisory, hiding a model does not make an exact model id invalid. A denied request reaching the public llm/stream waterfall terminates in-protocol with one { type: "finish", reason: { kind: "error", failure: { code: "MODEL_DISABLED" } } } chunk, before adapter dispatch and network egress.

The dispatch failure has the same shape as an adapter failure and remains composable with failover handling through agent/request-error.

Explicit non-goals:

• providers remain installed and configurable;
• the policy does not infer provider entitlements;
• direct HTTP use outside the DSH LLM service is out of scope;
• clone routes remain separate provider/model keys unless a */model rule covers them;
• no automatic fallback route is introduced.

Working experiment

To validate the semantics before proposing a core API, I implemented them as a third-party bundle plugin: dsh-model-gate (npm).

The plugin is usable on its own, but its main value to this discussion is implementation evidence:

• discovery filtering and dispatch enforcement are both necessary;
• settings changes apply without restart;
• the guard covers main sessions, subagents, explicit model ids, and background LLM calls that use the shared DSH service;
• rejected requests terminate before provider network dispatch;
• policy survives restart, while uninstall removes the runtime filtering and dispatch behavior without leaving wrappers behind.

It currently uses:

• instance-level wrapping of listProviders / listModels because DSH exposes no catalog-filter registration point;
• a normal llm/stream listener for the enforcement backstop;
• installSettingsSection plus a dedicated Web settings section for hot-applied persistence and toggling.

Install for users who need the behavior today:

dsh plugin --profile web add dsh-model-gate

Verification against DSH 0.1.x-rc currently includes:

• 19 automated tests covering matching, settings patches, wire validation, and UI contract details;
• 11 manual acceptance scenarios, including hot application, dispatch rejection, restart persistence, catalog restoration, and zero-residue uninstall.

Why a native seam would be better

The plugin proves the behavior, but instance-level method wrapping is not an ideal ecosystem contract:

• it relies on callers continuing to resolve the shared methods late;
• independent plugins cannot safely compose wrappers without an ordering and ownership protocol;
• an administrative UI needs a supported way to read the host baseline catalog without recursively applying its own filter;
• a first-class settings key would require no uninstall hygiene.

A native implementation could either own this policy directly in LlmRuntime or expose composable registration seams for catalog filters and dispatch policy.

If a general filter seam is preferred, it should define:

• deterministic ordering for multiple filters;
• whether failures are fail-open or fail-closed;
• provider/model context supplied to each filter;
• disposal and catalog-update notifications;
• a supported baseline-catalog read for administrative surfaces;
• the boundary between advisory discovery and mandatory dispatch enforcement.

Possible ownership models

1. Core-owned policy. DSH owns the settings schema, discovery subtraction, and dispatch rejection. This provides the strongest and simplest invariant.
2. Composable plugin seams. Plugins register catalog filters and dispatch guards; DSH owns ordering, lifecycle, and update semantics.
3. Adapter filtering only. This is appropriate for provider entitlement, but cannot provide cross-adapter user policy or mandatory dispatch enforcement.

My preference is core-owned policy, with composable seams still being useful for policy extensions.

Questions for maintainers

1. Does explicit user-defined model policy belong in core, or should DSH expose a composable plugin seam?
2. Should a policy-hidden model remain resolvable through a separate administrative baseline catalog?
3. Should MODEL_DISABLED become a standardized LLM failure code?
4. Should policy apply only at discovery and dispatch, or should exact metadata resolution also report the model as unavailable?
5. How should provider entitlement filters and user policy updates notify catalog consumers?

The request is not to adopt the plugin implementation as-is. It is to decide whether user-defined per-model policy is a core invariant and, if so, which seam should own it. I can turn the working behavior into a patch-shaped API and behavior spec if the direction is useful.