feat(guardrails): per-execution latency histogram aisix_guardrail_latency_seconds

[jarvis9443]

What

Implements the guardrail-observability-metrics part of api7/AISIX-Cloud#1076 (scope agreed in the issue comments: align capability with LiteLLM/Kong; the offline precision/recall eval set is out of scope per the Partner review note).

Today guardrail observability is two request-level aggregate counters — aisix_guardrail_blocks_total (no labels) and aisix_guardrail_bypasses_total{reason}, chat-completions only. There is no per-guardrail attribution, no phase/result split, and no latency measurement at all.

This PR adds aisix_guardrail_latency_seconds — a real bucketed histogram recorded once per consulted guardrail per hook pass, on every endpoint:

label	values
env_id	constant per DP process (SLO-histogram convention)
guardrail	configured guardrail (row) name
kind	keyword / pii (local, in-process) vs bedrock / lakera / presidio / … (remote)
phase	input / output
result	allowed / blocked / masked / bypassed / would_block / would_mask
error_type	bounded failure tag (lakera_timeout, openai_moderation_5xx, …) when result="bypassed", else none

Buckets span 1 ms – 30 s (guardrail-scale, below the SLO buckets), so histogram_quantile yields P50/P95/P99 per guardrail/phase/result across instances, and the _count series doubles as an execution/block/bypass counter.

How

• Recording sits in the GuardrailChain folds — the single point every handler's guardrail evaluation converges (chat, messages, responses, completions, embeddings, audio, streaming end-of-stream and window scans, cache-hit output checks, and the segment moderation pass). All endpoints are covered with zero handler churn, and the existing chat-only limitation of the aggregate counters does not carry over.
• No new crate dependency: aisix-guardrails records through a GuardrailMetricsSink trait defined in aisix-core (beside the existing GuardrailMonitorHit/AppliedGuardrail telemetry types). aisix-obs implements it on Metrics; the proxy/server bootstrap injects it via LiveGuardrailIndex::new_with_sink, and resolve() attaches it to every resolved chain.
• result is the enforced outcome: a monitor-mode member's suppressed decision records as would_block/would_mask (from its monitor hits) while the request proceeds — so a staged policy's hit rate and latency are measurable in Prometheus before flipping it to block. Non-segment folds skip segment-moderating members (they are consulted — and timed — via the segment pass), so each member records exactly once per pass.
• Each member's timing includes its decorators (monitor/mandatory) and internal timeout handling, i.e. the latency the request actually paid.

LiteLLM comparison (baseline per repo policy, checked at BerriAI/litellm@5d4c4d0fce)

LiteLLM exposes litellm_guardrail_latency_seconds{guardrail_name, status, error_type, hook_type} plus litellm_guardrail_requests_total / litellm_guardrail_errors_total. Capability is matched or exceeded; the shape intentionally differs in three ways:

1. One histogram instead of three metrics — with real buckets, _count subsumes both counters (documented PromQL equivalents in the paired docs PR).
2. Cleaner status taxonomy — LiteLLM's Prometheus layer records every content-policy block as status="error" (only its logging layer distinguishes guardrail_intervened from guardrail_failed_to_respond). We follow the intent of its logging-layer taxonomy instead: blocked vs bypassed(error_type) are first-class results.
3. Fail-open visibility — LiteLLM has no fail-open signal at all; we record bypassed with the bounded failure tag (consistent with the existing aisix_guardrail_bypasses_total{reason} values).

Kong (the other alignment target) records per-plugin input/output processing latency and block reasons in audit logs only, with no dedicated guardrail Prometheus histogram — covered by this metric plus the existing usage-event fields.

Known limits (deliberate)

• The synchronous per-field redactors (local pii mask rewrites, microsecond-scale) are not timed — their per-field call pattern would skew per-execution counts; detection/scan time for those kinds is still recorded via the check folds.
• A fail-closed remote failure records as blocked with error_type="none": the structured failure tag only exists on Bypass verdicts, and threading it through Block would touch ~65 construction/destructure sites for a rare path. Its latency signature (clustering at the provider timeout) remains visible.
• Existing counters are unchanged (no dashboard/scrape breakage).

No config/schema surface is added (metrics are emitted unconditionally, gated only by the existing observability.metrics.prometheus.enabled), so no CP-side wiring is required.

Tests

• chain.rs unit tests: per-member recording with name/kind/phase, enforced-result classification (allowed/blocked/bypassed+tag), segment-pass masked, non-segment skip of segment members, no-sink no-op.
• build.rs unit tests: sink injection through LiveGuardrailIndex::new_with_sink → resolved chains; monitor-mode would_block/would_mask recording end-to-end through the real decorators.
• metrics.rs unit test: rendered exposition has _bucket{le=...} series (not a summary) with the full label set and error_type="none" defaulting.
• New E2E guardrail-metrics-e2e.test.ts (real aisix + etcd + mock moderation endpoint + two mock upstreams): asserts allowed, local blocked, remote blocked, bypassed with error_type="openai_moderation_5xx", monitor-mode would_block, and output-phase blocked series with correct counts, plus bucketed rendering and env_id/kind labels.

Fixes api7/AISIX-Cloud#1076

Summary by CodeRabbit

• New Features

  • Added per-guardrail execution telemetry, including latency, phase, outcome, guardrail type, and error classification.
  • Added Prometheus histogram metrics for guardrail execution latency.
  • Added optional metrics integration for guardrail processing across proxy and server configurations.
  • Added support for monitoring enforced, bypassed, masked, and monitor-mode outcomes.

• Tests

  • Added unit and end-to-end coverage for telemetry labels, latency buckets, and execution outcomes.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 48c34f0c-0e0b-49b1-87f2-d6d51a2a7f85

📥 Commits

Reviewing files that changed from the base of the PR and between 7f6a96a and d49831f.

📒 Files selected for processing (9)

• crates/aisix-core/src/lib.rs
• crates/aisix-core/src/models/guardrail.rs
• crates/aisix-core/src/models/mod.rs
• crates/aisix-guardrails/src/build.rs
• crates/aisix-guardrails/src/chain.rs
• crates/aisix-obs/src/metrics.rs
• crates/aisix-proxy/src/state.rs
• crates/aisix-server/src/main.rs
• tests/e2e/src/cases/guardrail-metrics-e2e.test.ts

---

📝 Walkthrough

Walkthrough

Adds per-guardrail execution telemetry with bounded labels, outcome classification, elapsed timing, optional sink propagation, Prometheus histogram export, application wiring, and unit/E2E coverage.

Changes

Guardrail latency observability

Layer / File(s)	Summary
Telemetry contract crates/aisix-core/src/models/guardrail.rs, crates/aisix-core/src/models/mod.rs, crates/aisix-core/src/lib.rs	Defines and exports GuardrailExecution and GuardrailMetricsSink.
Chain execution recording crates/aisix-guardrails/src/chain.rs	Times guardrail members, classifies results, records phases and errors, handles segment moderation, and tests sink behavior.
Prometheus latency histogram crates/aisix-obs/src/metrics.rs	Adds the bucketed aisix_guardrail_latency_seconds histogram, bounded labels, sink implementation, and unit coverage.
Sink wiring and end-to-end validation crates/aisix-guardrails/src/build.rs, crates/aisix-proxy/src/state.rs, crates/aisix-server/src/main.rs, tests/e2e/src/cases/guardrail-metrics-e2e.test.ts	Propagates metrics into resolved chains and validates allowed, blocked, bypassed, monitored, and output executions.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related issues

• api7/AISIX-Cloud issue 1072 — Adds the per-guardrail latency observability described by the issue through chain timing and a Prometheus histogram.

Suggested reviewers: moonming

🚥 Pre-merge checks | ✅ 5 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	The PR implements per-guardrail latency observability, but it does not deliver the linked issue's offline evaluation set or Precision/Recall/FPR tooling.	Add the labeled offline PII/prompt-injection evaluation set and scripts that compute Precision, Recall, FPR, and threshold checks.

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding a per-execution guardrail latency histogram.
Out of Scope Changes check	✅ Passed	The changes stay focused on guardrail latency metrics, plumbing, and tests, with no obvious unrelated feature additions.
E2e Test Quality Review	✅ Passed	E2E uses real app+etcd+metrics, with justified mocks for externals, and covers allowed/blocked/bypassed/would_block/output phases cleanly.
Security Check	✅ Passed	No sensitive-data exposure found: new guardrail metrics/logging use bounded tags only, and bypass/error labels are fixed tags rather than raw request or secret values.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/guardrail-latency-metrics

---

_{Comment @coderabbitai help to get the list of available commands.}

[jarvis9443]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.