Repository navigation
agentgateway demo
AgentGateway performs authorization decisions at request-processing points: it determines whether an action on a resource is allowed given context expressed in CEL. This design instruments those decision points to make the underlying capability observable. External GovOps consumers enrich these observations with Authorization Capability Catalog (ACC) context to provide observable governance data.
Core Proposition: When AgentGateway decides whether an operation is allowed, it already knows the action and resource. GovOps makes that capability explicit and observable by emitting semantic observations that external governance systems can enrich with enterprise context.
Key Insight: AgentGateway observes protocol-level capabilities (what the system does). Governance defines enterprise capabilities (what the business authorizes). The separation enables protocol fidelity and enterprise context independently.
AgentGateway sees a protocol operation:
mcp:tools/call : github/archive_repository
↓
ALLOW via CEL
↓
CapabilityObservation
↓
OTel log record
↓
GovOps consumer finds no ACC mapping
↓
UNMAPPED CAPABILITY
Governance reviews and maps it to enterprise capability:
archive : repository
risk: high
owner: github-team
Subsequent observations are immediately enriched without any AgentGateway configuration change.
This demonstrates the GovOps feedback loop: observe → inventory → classify → govern.
- Emit semantic capability observations at authorization-aware decision points
- Expose the action and resource AgentGateway evaluates during authorization
- Enable external GovOps enrichment with ACC context
- Provide observation_id for future runtime correlation
- Maintain protocol fidelity (emit actual protocol operations, not CRUD abstractions)
- Change CEL evaluation logic or authorization behavior
- Parse CEL policies to infer capabilities
- Load or cache ACC in AgentGateway
- Perform kernel observation or runtime tracing
- AgentGateway does not assign business meaning, risk, ownership, or ACC membership to observed capabilities
- Modify MCP/LLM routing or execution behavior
AgentGateway
authorization: action + resource → outcome
↓
CapabilityObservation
action : resource
↓
OTel emission
↓
External GovOps Consumer
/ \
/ \
ACC.yaml Unmapped
lookup candidates
\ /
└─────┬────┘
↓
Enriched Events
(governance context)
↓
Dashboard / Audit
AgentGateway: Emits capability observations at authorization decision points via OpenTelemetry log records. No governance logic.
GovOps Consumer: External service that loads ACC.yaml (capability inventory with risk, ownership, business impact), matches observations to enterprise capabilities, tracks unmapped candidates for governance review.
Governance Layer: ACC.yaml defines enterprise capabilities with business metadata. Updated by governance team, not runtime infrastructure.
Future - Runtime Correlation: observation_id enables correlation to kernel evidence (eBPF/Tetragon/ETW) in Phase 2.
Authorization functions emit capability observations at decision points. The action and resource are derived from the protocol operation being authorized:
| AgentGateway Operation | Action | Resource |
|---|---|---|
| MCP tool call | mcp:tools/call |
mcp.tool:github/delete_repository |
| MCP prompt | mcp:prompts/get |
mcp.prompt:server/prompt |
| MCP resource read | mcp:resources/read |
mcp.resource:inventory/prod |
| MCP resource subscribe | mcp:resources/subscribe |
mcp.resource:inventory/prod |
| MCP task update | mcp:tasks/update |
mcp.task:server/task |
| LLM chat | llm:chat |
llm.model:openai/gpt-5 |
| LLM embed | llm:embed |
llm.model:openai/text-embedding-3 |
Critical Requirement: Same resource, different actions → different capabilities. For example:
mcp:resources/read : inventory/prodmcp:resources/subscribe : inventory/prod
These are distinct capabilities with potentially different risk profiles.
fn authorize_capability(
action: CapabilityAction,
resource: CapabilityResource,
auth_context: &AuthorizationContext
) -> AuthorizationDecision {
let decision = evaluate_cel_policy(action, resource, auth_context);
emit_capability_observation(action, resource, decision, auth_context);
decision
}The emitter generates a unique observation_id (UUID v7), computes a deterministic capability_id (SHA-256 of canonical action + resource), extracts principal and credential information, and emits an OTel log record. Telemetry failure is logged and metered but never blocks authorization.
Implementation details (function signatures, capability_id computation, OTel attributes) in Appendices C, A, and B.
The GovOps consumer is an external service (not part of AgentGateway) that:
- Ingests CapabilityObservation events from OTel collector
- Loads ACC.yaml capability inventory
- Matches observations against ACC entries
- Enriches matched observations with governance metadata
- Records unmapped capabilities as candidates for review
ACC capabilities define enterprise meaning with business metadata:
capability_id: archive-repository
# Enterprise capability
action: archive
resource: repository
# Governance metadata
risk_tier: high
owner: github-team
# Protocol-level mappings (what AgentGateway observes)
observed_capabilities:
- action: mcp:tools/call
resource:
type: mcp.tool
target: github
id: archive_repositoryKey Separation: One enterprise capability (archive : repository) may map to multiple protocol operations. The consumer matches AgentGateway's protocol-level action/resource against the observed_capabilities list to determine ACC membership.
The consumer maintains a registry of unmapped capabilities:
Action: mcp:tools/call
Resource: github/archive_repository
Status: UNMAPPED
First Seen: 2025-01-15T08:12:34Z
Last Seen: 2025-01-15T10:45:22Z
Count: 37
Governance reviews unmapped candidates and decides:
- Is this an implementation of an existing ACC capability?
- Is it a new enterprise capability requiring governance?
- Is it infrastructure noise that should not enter ACC?
- What business action/resource, risk, and ownership apply?
The consumer discovers candidates. Humans govern the catalog.
AgentGateway observes mcp:tools/call : github/archive_repository. The GovOps consumer cannot find a corresponding mapping in ACC, so it records an unmapped capability candidate. Governance reviews the candidate and maps it to archive : repository, adding risk and ownership metadata. Subsequent observations are immediately enriched without any AgentGateway configuration change.
Before ACC Mapping:
Action: mcp:tools/call
Resource: github/archive_repository
Status: UNMAPPED
Count: 37
After ACC Mapping:
Observed: mcp:tools/call : github/archive_repository
Governed: archive : repository
Risk: high
Owner: github-team
Additional Demo Scenarios:
-
Killer Demo Requirement: Same resource, different actions
-
mcp:resources/read : inventory/prod→ capability_id A -
mcp:resources/subscribe : inventory/prod→ capability_id B
-
-
Hot Reload: Update ACC.yaml with new mapping, consumer immediately enriches future observations (no AgentGateway restart)
-
Multiple Outcomes: Observations emitted for ALLOW (CEL policy), DENY (CEL policy), ALLOW (no policy configured), DENY (API key restriction)
The architecture comprises three core components: AgentGateway instruments authorization decision points and emits CapabilityObservation events via OpenTelemetry log records without performing governance logic. GovOps Consumer is an external service that ingests observations, loads ACC.yaml capability inventory, matches protocol-level observations to enterprise capabilities, and maintains an unmapped capability registry for governance review. Governance Layer defines enterprise capabilities in ACC.yaml with business metadata (risk, ownership) that the consumer uses for enrichment, updated by governance teams independently of runtime infrastructure.
The primary interface is the CapabilityObservation event emitted as an OTel log record with stable attributes (observation_id, action, resource, outcome, authorization metadata). Full data type definitions including Resource, AuthorizationInfo, Principal, and Credential structures are provided in Appendix C.
struct CapabilityObservation {
observation_id: String, // UUID v7
action: String, // mcp:tools/call, llm:chat
resource: Resource, // parsed into type:target/id
capability_id: String, // SHA-256 of canonical form
outcome: Outcome, // Allow | Deny
authorization: AuthorizationInfo,
principal: Option<Principal>,
credentials: Vec<Credential>,
observer: String, // "AgentGateway"
timestamp: DateTime<Utc>,
}
struct Resource {
resource_type: String, // mcp.tool, llm.model
resource_target: String, // github, openai
resource_id: String, // delete_repository, gpt-5
}Compact Notation: action : resource (e.g., mcp:tools/call : github/delete_repository)
struct AuthorizationInfo {
basis: AuthorizationBasis, // CelPolicy | ApiKey | Default | None
policy_refs: Vec<PolicyRef>,
}
struct PolicyRef {
policy_id: String,
policy_version: Option<String>,
attachment_scope: Option<String>,
effect: Option<String>, // allow | deny
}
enum AuthorizationBasis {
CelPolicy, // CEL policy evaluated
Default, // No policy, default allow
ApiKey, // API key authorization
None, // No authorization configured
}Authorization Semantics Examples:
- CEL policy allowed:
outcome: Allow, basis: CelPolicy, policy_refs: ["mcp-admin"] - No policy configured:
outcome: Allow, basis: Default, policy_refs: [] - CEL policy denied:
outcome: Deny, basis: CelPolicy, policy_refs: ["github-delete-restriction"] - API key restriction:
outcome: Deny, basis: ApiKey, policy_refs: []
Full data type definitions in Appendix C.
A property is a characteristic or behavior that should hold true across all valid executions of a system—essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.
For any authorization decision point in AgentGateway, the emitted CapabilityObservation action SHALL equal the protocol operation being authorized (never a CRUD abstraction or transformation).
Validates: Requirements 1.1, 1.2, 1.3, 1.4, 1.5, 1.6
For any authorization decision, the emitted CapabilityObservation resource SHALL equal the resolved resource sent upstream (not client alias or transformed identifier).
Validates: Requirements 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 7.1, 7.2, 7.3, 7.4
For any authorization evaluation, the emitted outcome SHALL equal the authorization result that controlled execution (allow or deny).
Validates: Requirements 3.1, 3.2, 3.3, 3.4
For any authorization evaluation, exactly one observation_id SHALL be generated, and all observation_ids SHALL be unique across the system lifetime.
Validates: Requirements 2.7
For any two CapabilityObservations with identical action and resource, the computed capability_id SHALL be identical regardless of time, policy changes, or AgentGateway version.
Validates: Requirements 2.1, 2.2, 2.3, 2.4, 2.5, 2.6
For any two CapabilityObservations where action differs OR resource differs, the computed capability_id SHALL be different.
Validates: Requirements 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7
For any telemetry emission failure or GovOps consumer unavailability, AgentGateway authorization outcome SHALL remain unaffected and authorization flow SHALL complete successfully.
Validates: Requirements 5.3, 5.5, 5.7, 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7
For any CapabilityObservation emission failure, an ERROR log entry SHALL be created AND the govops_telemetry_loss_total metric SHALL be incremented.
Validates: Requirements 5.1, 5.2
For all authorization decision bases (CEL allow, CEL deny, default allow, API key deny), a CapabilityObservation SHALL be emitted with correct outcome and authorization_basis.
Validates: Requirements 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7
For any CapabilityObservation containing credentials, the emitted observation SHALL include credential type, issuer, and identifier, but SHALL NOT include raw credential values (JWT tokens, API keys, passwords).
Validates: Requirements 6.5, 6.6
For all CapabilityObservations, high-cardinality fields (observation_id, resource_id, principal_id, capability_id) SHALL appear only in OTel log record attributes and SHALL NOT appear as metric labels.
Validates: Requirements 10.2, 10.3, 10.4, 10.5, 10.6
For any CEL-based authorization decision, the policy_refs array SHALL contain all evaluated policy identifiers with effect fields, and for any non-CEL authorization decision, the policy_refs array SHALL be empty.
Validates: Requirements 3.5, 3.6, 3.7, 11.1, 11.2, 11.6, 11.7
Telemetry emission failures are logged and metered but never block authorization flow, with OTel SDK handling transient retries automatically. External GovOps consumer failures do not affect AgentGateway authorization, with OTel collector buffering providing durability during consumer downtime. Canonical form computation errors result in ERROR logs with capability_id set to "INVALID" while authorization proceeds unaffected. Full error scenarios, responses, and recovery procedures in Appendix E.
Unit tests verify canonical form generation (determinism, case sensitivity, special characters), CapabilityObservation construction (observation_id uniqueness, timestamp format, metadata extraction), and telemetry loss handling. Integration tests validate observation emission at authorization points, OTel log export with stable attributes, and metric emission. End-to-end tests cover the full pipeline from authorization through enrichment including mapped/unmapped scenarios, different actions on same resource, policy changes, and consumer restart resilience. Full test plans and example code in Appendix F.
- No sensitive data in telemetry: Include principal type/id and credential type/issuer/id only. Never emit raw JWT tokens, passwords, or API keys.
-
Correlation key for runtime observability:
observation_idenables future kernel evidence correlation without coupling authorization to execution. - External enrichment: AgentGateway does NOT load ACC.yaml, preventing governance data from affecting authorization logic.
- High-cardinality data in logs, not metrics: Resource IDs, principal IDs, observation IDs in OTel log records only (prevents cardinality explosion).
- Telemetry failure does not block authorization: OTel emission failure logged and metered, authorization proceeds.
- Existing OpenTelemetry/tracing infrastructure in AgentGateway
- OTel log exporter configured (OTLP, file, or stdout)
- External OTel collector for event aggregation (optional for MVP)
- External GovOps consumer for ACC enrichment (built in Week 3)
After capability observability works, add kernel correlation using observation_id:
Authorization Observation:
observation_id: obs-123
action: mcp:tools/call
resource: github/delete_repository
outcome: allow
Runtime Evidence (eBPF/Tetragon):
observation_id: obs-123
connect: 10.0.4.32:443
observation_id: obs-123
read: /etc/ssl/cert.pem
This proves: "After capability X was authorized, what did the runtime actually do?"
Week 1: Instrument Authorization Decision Points
- Add
CapabilityObservationstruct to AgentGateway - At each authorization function, derive action/resource and emit CapabilityObservation via OTel log records
- Include: observation_id, action, resource, outcome, authorization_basis, policy_refs, principal, timestamp
- Compute capability_id as SHA-256(canonical action + resource)
- Handle telemetry loss gracefully (log ERROR, increment metric, continue authorization)
Week 2: Demo Governance Loop
- Test same resource, different actions (
subscribevsreadoninventory/prod) - Test unmapped capability (
archive_repositorynot in ACC) - Verify capability_id stability across multiple authorizations
- Hot-reload: Add ACC mapping for unmapped capability, verify immediate enrichment
Week 3: Build External GovOps Consumer
- Implement OTel log record ingestion from collector
- Load ACC.yaml and perform capability matching
- Maintain unmapped capability registry (first_seen, last_seen, count)
- Emit enriched observations with governance metadata
- Generate dashboard showing mapped/unmapped capabilities with risk/ownership
- Demo complete governance feedback loop: observe → inventory → classify → govern
To ensure stable capability identity across deployments, policies, and time, we define a canonical form for action + resource pairs.
Canonical Form v1:
govops-observed:v1|<action>|<resource_type>|<resource_target>|<resource_id>
Rules:
- UTF-8 encoding
- Lowercase: action, resource_type
- Preserve case: resource_target, resource_id
- Empty resource_target: ""
- Separator: pipe (|)
- SHA-256 hash of canonical form = capability_id
Examples:
mcp:tools/call : github/delete_repository
→ govops-observed:v1|mcp:tools/call|mcp.tool|github|delete_repository
→ SHA-256 → capability_id
llm:chat : openai/gpt-5
→ govops-observed:v1|llm:chat|llm.model|openai|gpt-5
→ SHA-256 → capability_id
Determinism: Same logical capability exercise always produces the same capability_id, regardless of CEL policy syntax, AgentGateway version, or time.
Event Name: govops.capability.observation
Event Type: OTel log record (preserves high-cardinality fields)
Stable Attributes:
govops.observation.id: String
govops.action: String
govops.resource.type: String
govops.resource.target: String
govops.resource.id: String
govops.capability.id: String (SHA-256)
govops.observation.outcome: String (allow | deny)
govops.observation.authorization_basis: String
govops.observer: String
govops.policy.refs: JSON array
govops.principal.type: String
govops.principal.id: String
govops.credentials: JSON array
govops.timestamp: String (ISO8601)
Metrics (Bounded Cardinality):
govops_observations_total{outcome, action, resource_type, resource_target}
CRITICAL: Do NOT use high-cardinality fields (resource_id, principal_id, observation_id) as metric labels.
Example OTel Log Record:
{
"timestamp": "2025-01-15T10:23:45.123Z",
"body": "Authorization observation for mcp:tools/call",
"attributes": {
"govops.observation.id": "01963a7c-4b2e-7890-a123-456789abcdef",
"govops.action": "mcp:tools/call",
"govops.resource.type": "mcp.tool",
"govops.resource.target": "github",
"govops.resource.id": "delete_repository",
"govops.capability.id": "a3f8b9c...",
"govops.observation.outcome": "deny",
"govops.observation.authorization_basis": "cel",
"govops.observer": "AgentGateway",
"govops.principal.type": "user",
"govops.principal.id": "alice"
}
}Telemetry Loss Handling:
match emit_capability_observation(observation) {
Err(e) => {
error!("Telemetry loss: {}", e);
metrics::increment_counter!("govops_telemetry_loss_total");
}
Ok(_) => {}
}struct CapabilityObservation {
observation_id: String, // UUID v7
action: String, // mcp:tools/call, llm:chat
resource: Resource,
capability_id: String, // SHA-256 of canonical form
outcome: Outcome, // Allow | Deny
authorization: AuthorizationInfo,
principal: Option<Principal>,
credentials: Vec<Credential>,
observer: String, // "AgentGateway"
timestamp: DateTime<Utc>,
}
struct Resource {
resource_type: String, // mcp.tool, llm.model
resource_target: String, // github, openai
resource_id: String, // delete_repository, gpt-5
}
struct AuthorizationInfo {
basis: AuthorizationBasis, // CelPolicy | ApiKey | Default | None
policy_refs: Vec<PolicyRef>,
}
struct PolicyRef {
policy_id: String,
policy_version: Option<String>,
attachment_scope: Option<String>,
effect: Option<String>, // allow | deny
}
struct Principal {
type_: String, // user | agent | workload | system
id: String,
}
struct Credential {
type_: String, // jwt | api_key | spiffe | mtls
issuer: Option<String>,
id: String, // jti, key_id, SPIFFE ID, cert fingerprint
}
enum AuthorizationBasis {
CelPolicy, // CEL policy evaluated
Default, // No policy, default allow
ApiKey, // API key authorization
None, // No authorization configured
}
enum Outcome {
Allow,
Deny,
}
enum CapabilityAction {
ToolsCall,
PromptsGet,
ResourceRead,
ResourceSubscribe,
ResourceUnsubscribe,
TasksGet,
TasksUpdate,
TasksCancel,
Chat,
Embed,
}The twelve correctness properties defined in this design ensure observability fidelity, governance independence, and system reliability. See the main "Correctness Properties" section for full property definitions and requirements mappings.
Summary of Properties:
- Action Fidelity: Emitted action equals protocol operation (never abstraction)
- Resource Fidelity: Emitted resource equals resolved upstream resource
- Outcome Fidelity: Emitted outcome equals authorization result that controlled execution
- Observation Uniqueness: One evaluation produces one unique observation_id
- Capability Stability: Same action/resource always produces same capability_id
- Action-Resource Distinctiveness: Different action OR resource produces different capability_id
- Governance Independence: Telemetry/consumer failure cannot affect authorization
- Telemetry Loss Observability: Emission failure logged and metered
- Observability Completeness: All authorization outcomes observable
- Metadata Security: No raw credential values in observations
- Cardinality Boundary: High-cardinality data in logs only, not metrics
- Policy Reference Accuracy: CEL decisions include policy_refs, non-CEL decisions have empty array
Telemetry Emission Failure
Condition: OTel log exporter fails to emit CapabilityObservation event (network error, buffer full, misconfiguration).
Response:
- Log ERROR with context:
error!("Telemetry loss: {}", e) - Increment metric:
govops_telemetry_loss_total - Continue authorization flow (do NOT block on telemetry failure)
Recovery:
- OTel SDK retries transient failures automatically
- Persistent failures visible via telemetry loss metric
- Operators alerted to investigate OTel exporter configuration
ACC Enrichment Consumer Failure
Condition: External GovOps consumer crashes or cannot reach OTel collector.
Response:
- AgentGateway authorization continues unaffected
- CapabilityObservation events accumulate in OTel collector buffer
- Consumer restarts and processes buffered events
Recovery:
- OTel collector provides durable buffering
- Consumer implements idempotent event processing (using observation_id)
- No data loss if consumer downtime < buffer retention period
Canonical Form Computation Error
Condition: Invalid action or resource values prevent canonical form generation.
Response:
- Log ERROR with invalid input:
error!("Cannot compute capability_id: invalid resource {:?}", resource) - Emit CapabilityObservation with
capability_id = "INVALID" - Continue authorization flow
Recovery:
- Validation logic prevents malformed action/resource from reaching authorization evaluation
- If reached, telemetry captures error for debugging
- Authorization outcome unaffected by telemetry computation failure
Unit Testing
Canonical Form Generation:
- Test deterministic SHA-256 for same action/resource pairs
- Test case sensitivity rules (lowercase action, preserve resource case)
- Test empty resource_target handling
- Test special characters in resource IDs
CapabilityObservation Construction:
- Test observation_id uniqueness (UUID v7)
- Test timestamp generation (ISO8601 format)
- Test principal/credential extraction from AuthorizationContext
- Test policy_refs population
Example Test:
#[test]
fn test_canonical_form_stability() {
let action = CapabilityAction::ToolsCall;
let resource = Resource {
resource_type: "mcp.tool".into(),
resource_target: "github".into(),
resource_id: "delete_repository".into(),
};
let id1 = compute_capability_id(&action, &resource);
let id2 = compute_capability_id(&action, &resource);
assert_eq!(id1, id2);
}
#[test]
fn test_different_actions_different_ids() {
let resource = Resource {
resource_type: "mcp.resource".into(),
resource_target: "inventory".into(),
resource_id: "prod".into(),
};
let id_read = compute_capability_id(&CapabilityAction::ResourceRead, &resource);
let id_subscribe = compute_capability_id(&CapabilityAction::ResourceSubscribe, &resource);
assert_ne!(id_read, id_subscribe);
}Integration Testing
Authorization Instrumentation:
- Test CapabilityObservation emission at each authorization function
- Test observation_id correlation across authorization chain
- Test telemetry loss handling (mock OTel exporter failure)
- Test authorization continues when telemetry fails
OTel Log Export:
- Test event emission to stdout exporter
- Test stable attribute presence in log records
- Test metric emission (
govops_observations_total) - Test telemetry loss metric (
govops_telemetry_loss_total)
Example Test:
#[tokio::test]
async fn test_authorize_emits_capability_observation() {
let (gateway, otel_receiver) = setup_test_gateway_with_telemetry();
let result = gateway.authorize_tool_call(
"github",
"delete_repository",
&test_auth_context()
).await;
assert_eq!(result.outcome, Outcome::Deny);
let events = otel_receiver.received_events();
assert_eq!(events.len(), 1);
let event = &events[0];
assert_eq!(event.attributes["govops.action"], "mcp:tools/call");
assert_eq!(event.attributes["govops.resource.id"], "delete_repository");
assert_eq!(event.attributes["govops.observation.outcome"], "deny");
}End-to-End Testing
Full Pipeline:
- AgentGateway authorizes MCP tool call
- CapabilityObservation emitted to OTel collector
- External GovOps consumer enriches with ACC metadata
- Enriched event written to dashboard storage
Test Scenarios:
- MAPPED capability:
tools/call : github/delete_repository→ enriched with ACC risk_tier - UNMAPPED capability:
tools/call : github/archive_repository→ flagged as unmapped - Different actions:
resources/readvsresources/subscribe→ different capability_ids - Policy change: Update CEL policy, verify same capability_id emitted
- Consumer restart: Stop consumer, emit events, restart, verify enrichment resumes
- No policy configured: Verify observation emitted with
authorization_basis: Default