Evaluation and Audit

Evaluation And Audit

PolicyAware connects evaluation results to governance outcomes. The goal is not only to measure model quality, but to verify that requests, policies, routing, and outputs behaved correctly.

Evaluation Types

• Policy compliance checks.
• Sensitive-data leakage checks.
• Citation-required checks for RAG.
• Prompt regression tests.
• Tool misuse detection.
• Approval path validation.
• Golden dataset execution.

Eval Suite Example

suite: executable_governance_policy_eval

checks:
  - type: expected_policy_decision
  - type: expected_reason_codes
  - type: sensitive_data_leakage

cases:
  - id: support_ticket_allowed
    input: "Summarize this customer request."
    user:
      id: eval_user
      role: support_agent
    context:
      region: us
      risk: low
      task_type: support
    expected:
      decision: allow
      reason_codes:
        - POLICY.ALLOW_MATCHED

  - id: pii_is_redacted
    input: "Email jane@example.com about the claim."
    user:
      id: eval_user
      role: support_agent
    context:
      region: us
      risk: low
      task_type: support
    expected:
      decision: conditional_allow
      reason_codes:
        - DATA.PII_DETECTED
        - POLICY.TRANSFORM_APPLIED

Run:

policyaware eval run examples/evals/executable_governance_cases.yaml \
  --policy-file examples/policies/basic.yaml

Audit Trace Contents

Each audit trace can include:

• trace ID and request ID
• tenant, app, user, task type
• policy decision and matched rules
• reason codes
• risk tier and risk score
• selected model
• latency and cost estimate
• evaluation scores
• request and response snapshots
• tool calls

Audit Flow

flowchart LR
    A["Gateway response"] --> B["Audit trace"]
    B --> C["JSONL storage"]
    B --> D["SQLite storage"]
    B --> E["Audit bundle"]
    B --> F["Trace viewer"]
    B --> G["Prometheus metrics"]
    B --> H["OpenTelemetry-shaped export"]

Loading

Export Audit Bundle

policyaware audit bundle .policyaware/traces.jsonl --output audit_bundle

View Audit Traces

policyaware audit view .policyaware/traces.jsonl --output trace-viewer.html

Export Metrics

policyaware observability prometheus .policyaware/traces.jsonl
policyaware observability otel-json .policyaware/traces.jsonl