Global (api-level) Policies for LLM Providers and Proxies

[ashera96]

Problem

On a REST API you can attach a policy at the API level (spec.policies) and it applies to every resource at once. For rate limiting that gives you one shared bucket for the whole API — use up the limit on one resource and the rest are limited too.

You can't do this on an LLM provider or proxy. Every policy has to be tied to specific paths, so it only ever covers those resources — there's no way to apply something across the whole provider, like one rate limit shared by all resources or one guardrail that covers everything.

Under the hood, when an LLM artifact is turned into a REST API its policies are always attached per-resource, never at the API level.

Solution

Introduce a global policy scope for LLM providers and proxies: policies applied across all resources as a single shared scope, evaluated before any resource-level policy. The capability is policy-agnostic (accepts any policy type); rate limiting is the primary motivating use case. Existing path-scoped policies are unchanged.

• Stateful policies (rate-limit by request/token/cost, quota, spend caps) → one shared bucket/counter for the whole provider.
• Any policy → declared once, guaranteed to cover every resource (including the catch-all and resources added later), evaluated ahead of resource-level policies.

Design

Options considered

Option	How a global policy is declared	Schema impact	Backward compatibility
A — Separate globalPolicies section ✅ (proposed)	A new list parallel to policies, reusing the existing generic Policy schema (name/version/params, no paths).	Add one field; LLMPolicy untouched.	Fully additive — existing policies and its validation are intact.
B — Path-less entry in policies	Omit paths on an entry in the existing policies list ⇒ global.	Mutate LLMPolicy: make paths optional, add top-level params; add a rule to reject paths+params together.	Weakens validation — an accidentally omitted paths silently becomes "global" instead of erroring.
C — Explicit scope flag	Add scope: global|resource to a policy entry.	Add a field to LLMPolicy.	Backward-compatible (defaults to resource), but paths is dead weight when scope=global.
D — Reinterpret a wildcard path	Treat a policy on / or /* with methods ["*"] as global.	None.	Silently changes the behaviour of existing wildcard-path policies.

Proposed: Option A — a separate globalPolicies section

Keep the existing path-bound spec.policies (resource-level) exactly as-is, and add a new optional spec.globalPolicies list that reuses the generic Policy schema already used by REST APIs:

spec:
  # ...
  globalPolicies:                   # global: applies to ALL resources, evaluated first
    - name: basic-ratelimit
      version: v1
      params:
        limits: [ { requests: 100, duration: "1m" } ]
  policies:                         # resource-level: path-scoped (unchanged)
    - name: token-based-ratelimit
      version: v1
      paths:
        - path: /chat/completions
          methods: [POST]
          params: { totalTokenLimits: [ { count: 1000, duration: "1m" } ] }

Why Option A

1. Most minimal and backward-compatible. LLMPolicy is not touched, so paths stays required and no existing artifact, validation rule, or behaviour changes. (Option B has to mutate LLMPolicy and add an ambiguity rule.)
2. Clean separation and preserved order. Two lists, one meaning each. The global list maps directly onto the derived API's api-level chain, which is prepended ahead of resource-level policies — the "global scope before resource scope" model is automatic.
3. True reuse. globalPolicies is []Policy (the generic policy type), so a global policy of any kind is routed straight into the api-level chain with no conversion and no per-policy special-casing.

Naming: the field is globalPolicies to match the existing "Global Guardrails & Policies" UI label, and it reads naturally for both providers and proxies. The existing path-bound policies is the "Resource-wise" list and is left unchanged. (Note the cross-artifact asymmetry: on LLM artifacts policies is resource-level, while on REST policies is api-level — forced by backward-compat, since LLM's policies is already path-bound.)

Implementation outline

• Controller: collect globalPolicies into the derived REST API's spec.policies during the LLM→REST transform; path-scoped policies keep their per-operation attachment. No change to the REST transformer or the policy-engine — global policies reuse the existing attachedTo: "api" api-level path (tagged LevelAPI, prepended to every route, api-scoped bucket key for stateful policies).
• Schema: add globalPolicies: []Policy to LLMProviderConfigData and LLMProxyConfigData; regenerate types.

Discussion points

• Allowed policies: should globalPolicies accept any policy type (as REST api-level does), or be restricted to a curated set that is meaningful globally?
• Token/cost policies: confirm token/cost extraction identifiers (from the template) resolve at the api level (they describe extraction location, which is resource-independent).
• Field name — proposed globalPolicies; alternatives welcome:
  • globalPolicies (proposed) — matches the existing "Global Guardrails & Policies" UI label; reads naturally for both providers and proxies.
  • sharedPolicies — emphasises the single shared scope/bucket.

[ashera96]

Following up with the MCP findings and the revised proposal.

1. Does this extend to MCP — what does "global" mean for MCP proxies?
2. For LLM, what's the shape — one policies list or two — and when global and resource policies sit in the same list, how do they order?

• Keep MCP out of this controller change. It already attaches policies server-wide, its "global" is policy-bound rather than route-bound, and a flat server-wide cap is already available via basic-ratelimit. A capability-aware "server-wide" mode, if ever needed, is a small per-policy addition — not a cross-cutting controller field.
• For LLM, group — don't interleave: global/api-level policies run first, then resource-level, exactly like REST. Then comes the separate-lists-vs-one-list decision.

Design

The key distinction

	Differentiated by	Who sees the discriminator	api-level attachment	A single "global bucket" across sub-units
REST	URL path + method (route)	the controller (route key)	✅ spec.policies	✅ controller drops the route from the bucket key
LLM	upstream resource path → route	the controller (route key)	❌ (the gap this proposal fixes)	to-be-implemented
MCP	JSON-RPC capability (tool/resource/prompt/method) in the payload	the policy (parses the body)	✅ spec.policies	❌ none today; the policy always keys per-capability

It comes down to where the discriminator lives. REST/LLM split on the route, which the controller owns — so the controller can offer a generic "global = drop the route from the key." MCP splits on the payload, which only the policy can read — so scope is the policy's call, and there's no generic controller switch for it.

MCP: api-level works, but "global" is policy-bound

MCP already attaches policies server-wide: spec.policies on an MCP proxy is the generic Policy (no paths), tagged attachedTo: "api", so it applies to every call.

The catch is that MCP rate-limit scope is decided by the policy, not the route. Every MCP call is JSON-RPC over the same endpoint, so mcp-ratelimit reads the envelope and limits per capability — tool (params.name), resource (params.uri), prompt (params.name), or raw method. Per the mcp-ratelimit policy doc, the capability id is always appended to the bucket key — even under *:

"methods": [ { "name": "*", "limits": [ { "limit": 5, "duration": "1m" } ] } ]

That's 5/min per distinct method (separate buckets for tools/list, tools/call, …), not one shared bucket. For a single server-wide cap, basic-ratelimit already does the job — it's capability-agnostic, so on MCP's single endpoint it counts everything together.

So a generic "make any policy global" doesn't fit MCP: the unit being counted is defined and parsed by the policy itself, and each policy differs. Api-level support here is really policy-bound — the controller already treats every policy as api-level; global vs per-capability is up to the policy.

LLM: one list vs two, and ordering

• Option A — separate lists (globalPolicies + policies): clean separation, ordering unambiguous by construction; cost = a new field, and globalPolicies could read as misleading.
• Option B — one policies list, where an entry with no paths means api-level: no new field; the one list now carries both scopes (path = resource, no path = api-level), which raises the ordering question.

Point 1 — how do mixed entries order?

• (i) Honour author order — run entries as written, filtered to {global} ∪ {resource entries matching this route}. [resourceX, global1, resourceY] → resourceX runs, then global1.
• (ii) Group by scope — global first, then resource, regardless of interleaving. Exactly what REST does today.

Preference: (ii) grouping, for the reasons below.

Point 2 — implementation cost (grouping vs custom order)

• Grouping reuses what REST already does: collectAPIPolicies builds the api-level set, buildPolicyChain prepends it, then appends route-level. Option B just splits the one list by path-presence and feeds the same function — almost no new code, identical behaviour to REST.
• Custom order is a real change: the chain is level-grouped today and doesn't track position across the global/resource boundary. Honouring order means carrying each entry's index, rebuilding the chain as an index-ordered per-route merge, and squaring that with attachedTo and the policy-engine. More code, and a lasting split from the REST path.

Point 3 — would custom ordering pull LLM away from REST and MCP?

Yes. REST groups (api → resource); MCP is single-level. Custom interleaving makes LLM a third model:

• Inconsistent — running REST + LLM + MCP means three mental models for "what runs first."
• UI — the "Global Guardrails & Policies" / "Resource-wise" split already implies grouping; interleaving doesn't fit two sections.
• Portability — moving a policy set between artifact types would quietly change behaviour.

Discussion points

• Separate lists (with a clearer name) or one list with documented grouping?
• If we keep a dedicated field, what name reads best across providers and proxies — globalPolicies, sharedPolicies, …?
• MCP: any real demand for a true server-wide cap across all capabilities, or do per-capability limits (plus basic-ratelimit for a flat cap) cover it?

[ashera96]

Following is the finalised feature design:

• When considering the separate lists or single list, the finalised option is having 2 separate lists, while deprecating the current policies list so as to maintain backward compatability.
• MCP will be considered out of scope for this feature as policies supported right now are anyway attached at the API level, and bucketing logic is handled at the policy implementation level.

Design

spec:
  # ...
  globalPolicies: # global: applies to ALL operations, evaluated first
    - name: basic-ratelimit
      version: v1
      params:
        limits: [ { requests: 100, duration: "1m" } ]
  operationPolicies: # operation-level: evaluate after global policies
    - name: token-based-ratelimit
      version: v1
      paths:
        - path: /chat/completions
          methods: [POST]
          params: { totalTokenLimits: [ { count: 1000, duration: "1m" } ] }
  policies: # current operation-level policy array (deprecated, but supported for backward compatability)
    - name: token-based-ratelimit
      version: v1
      paths:
        - path: /chat/completions
          methods: [POST]
          params: { totalTokenLimits: [ { count: 1000, duration: "1m" } ] }

A pending action item was to determine the appropriate wording for resource-level policies:

• operationPolicies: operation -> GET /orders/{orderId} (https://swagger.io/docs/specification/v3_0/paths-and-operations/?sbsearch=Operations#operations)
• pathPolicies: path -> /orders/{orderId}
• resourcePolicies

[ashera96]

Update on the feature design:

• Following schema change was finalised; where Policy schema used reused for LLM providers and proxies. A new schema will be introduced for operationPolicies named OperationPolicy. Previous LLMPolicy associated policies list was deprecated.

globalPolicies:
  type: array
  description: Global (api-level) policies applied across ALL operations as one shared scope, evaluated before operation-level policies.
  items:
    $ref: "#/components/schemas/Policy"
operationPolicies:
  type: array
  description: Operation-level policies scoped to specific paths/methods, evaluated after global policies.
  items:
    $ref: "#/components/schemas/OperationPolicy"
policies:
  deprecated: true
  type: array
  description: DEPRECATED - use operationPolicies. Still honoured (treated identically to operationPolicies).
  items:
    $ref: "#/components/schemas/LLMPolicy"

• We will handle on-the fly migration for previously created LLM providers and proxies. Once a save action is performed, the previously saved policies list will be migrated to operationPolicies and globalPolicies. For a newly onboarded provider or proxy, we will be saving as operationPolicies and globalPolicies.
• Considering the scenario where previous gateway version and the new version are both active, at the deploy step, we need to collect the current gateway version and transform the saved data to honour the gateway version preferred approach. Note: if no version sent, it's assumed to be 1.0.0. Versions 1.1.0 is correctly being sent for the 1.1.0 release. For the current working version it'll be 1.2.0-SNAPSHOT; where policies should no longer be used, rather the 2 newly introduced lists.

[ashera96]

Update on the backward-compatibility discussion:

• Update the gateway api version from 0.9.0 to v1alpha1 to match the current apiVersion: gateway.api-platform.wso2.com/v1alpha1. However, since we are doing a modification to the spec, it should now be v1alpha2
• Update the platform api version from 1.0.0 to v1alpha2
• In terms of data transformation, if there is an inbound platform api call made with version 1.0.0, with the policies list, we retain the data by transforming to the 2 separate lists. But with the response payload, it should return the policies list after data merging. If a separate GET call is fired with version 1.0.0, we should always response with the policies list by considering the data transformation aspect. On the other hand, if the request is made with version v1alpha2, regardless of whether a single policies list or 2 separate lists are attached, the response should return 2 lists, as the request was made in v1alpha2 and the preferred policy handling in that version is 2 separate lists.
• All data must be retained in the new and preferred way, but the response should honour request version.
• AI workspace UI should always call the platform API's latest version. Optionally, if no version was specified it is assumed to be for the latest.

Also, the following were discussed to be added as new packages to handle these data transformations:

• Inbound platform API calls, data transformation layer should come into play. Introduce a new package to the platform api component.
• Outgoing gateway calls should undergo data transformation to modify the payload to the gateway compatible format.