observability vtl

---

title: Observability & PHI Safety Pipeline acronyms: [VTL] created: 2026-07-09 updated: 2026-07-15 type: concept tags: [sdk, compliance, infrastructure] confidence: high source_files:

• integrity-sdk/integrity_sdk/security/redactor.py
• integrity-sdk/integrity_sdk/integrations/openai_integrity.py
• integrity-sdk/integrity_sdk/integrations/langchain_callback.py
• integrity-sdk/integrity_sdk/telemetry/tracing.py
• integrity-sdk/tests/unit/test_redactor.py
• integrity-sdk/tests/unit/test_tracing.py
• integrity-sdk/tests/unit/test_openai_integrity.py
• integrity-sdk/tests/unit/test_langchain_callback.py
• integrity-oracle/backend/src/phi.rs
• integrity-oracle/backend/src/handlers.rs
• integrity-oracle/backend/migrations/0002_markets_and_judge.sql

---

This page replaces the old wiki's observability-vtl.md/phi.md. It documents the corrected, current design for how PHI/PII/secrets are kept out of oracle storage — the design decided in this rewrite after cross-checking prior Xibalba Solutions spec docs, not the old wiki's speculative "Verifiable Trust Layer" (GuardrailEngine, StateStore time-travel, a separate integrity-observability-backend service, a ReputationLedger) — none of that exists in this repo; see the note at the bottom.

Table of contents

• The real gap this design fixes
• Redactor — targeted, not blanket masking (built, tested)
• LLM-as-judge evaluation — storage/ingestion plumbing built; rubric [PLANNED]
• Oracle-side defense in depth — built
• Dual-mode storage — [PLANNED], bigger roadmap item
• What this page does NOT claim (correcting the old wiki)

The real gap this design fixes

integrity_sdk/integrations/openai_integrity.py used to set GenAIAttributes.PROMPT/COMPLETION (raw prompt/completion text) as OTel span attributes with zero redaction. If a span like that were ever flushed through client.py's telemetry path to the oracle's POST /v1/telemetry/ingest, raw agent content — for a healthcare-vertical agent, actual PHI — would land in oracle storage. That is exactly the leak this design exists to close, and it was live in already-shipped code.

Redactor — targeted, not blanket masking (built, tested)

integrity_sdk/security/redactor.py, alongside the existing security/attestation.py/security/vault.py. Performs client-side masking of specific sensitive-content categories before any prompt/ completion text leaves the agent's process — not a blanket hash-everything strip. An earlier draft of this component specified exactly that blanket rule; it was wrong, because oracle-side LLM-as-judge evaluation (below) needs structurally-intact, if-redacted trace content to actually judge reasoning/tool-use shape — a blanket hash would make that impossible.

Regex/heuristic categories matched today: PRIVATE_KEY (PEM blocks), API_KEY (OpenAI sk-..., AWS AKIA..., Bearer ...), SSN, CREDIT_CARD, EMAIL, PHONE, MRN (medical record number, labeled convention). Each match is replaced with [REDACTED:{CATEGORY}]. Honest scope, stated in the source itself: this is real, working masking for common high-confidence cases — not a certified HIPAA Safe Harbor 18-identifier de-identification system, and it will miss PHI with no structural marker (e.g. a patient's name in free text). This is why the oracle-side backstop below exists.

Wired into the two named integrations, but as of 2026-07-15, opt-in, not unconditional: both openai_integrity.py's IntegrityOpenAI and langchain_callback.py's IntegrityLangChainCallback accept a redact_phi: bool constructor parameter that now defaults to False (a real, deliberate behavior change from this section's prior claim that redaction ran unconditionally). When True, openai_integrity.py calls redact_text(...) on prompt text, completion text (both streaming and non-streaming), and streamed chunks before any span attribute is set or local-metrology signals are derived from it; langchain_callback.py redacts text_output and reasoning_content the same way. Any Integrity Health / healthcare-vertical agent MUST pass redact_phi=True explicitly — neither wrapper can infer an agent's compliance_vertical on its own, so nothing auto-enables this for a healthcare deployment; leaving it at the default logs a logger.warning naming the agent, but nothing currently enforces the flag being set correctly (a real, accepted residual risk — see PRODUCTION_GAPS.md §3, and Telemetry Ingestion Pipeline §2 for the full writeup). telemetry/tracing.py's trace_run/traceable API is unaffected by this flag — it still redacts every string leaf unconditionally, always. Tested in integrity-sdk/tests/unit/test_redactor.py (clean text passes through untouched; SSN/email/phone/credit-card/API-key patterns are matched and masked) and, new for the redact_phi behavior itself, tests/unit/test_openai_integrity.py/test_langchain_callback.py (default leaves text raw; redact_phi=True redacts prompt/completion/ reasoning text; 13 tests total, neither integration had any coverage before this).

Real gap this page previously overstated, closed 2026-07-11: this section used to claim redaction was "wired into both instrumentation paths" as if that were the complete picture. It wasn't — telemetry/ tracing.py's trace_run/traceable/client.traceable(...) (the SDK's own documented, recommended general-purpose tracing API, not one of the two "instrumentation paths" this page was scoped to) captured a wrapped function's raw arguments and return value with zero redaction. Any consumer decorating their own LLM-calling function with @client.traceable(...) — the normal, recommended way to use this SDK — was forwarding raw, unredacted content toward the oracle. Fixed: a new _redact_value helper in tracing.py recursively applies redact_text to every string leaf in TraceRun.set_outputs's value and _capture_inputs's captured arguments, however deeply nested. Tested in the new integrity-sdk/tests/unit/test_tracing.py (previously zero dedicated tests existed for this module at all).

LLM-as-judge evaluation — storage/ingestion plumbing built; rubric [PLANNED]

integrity-oracle/backend's judge_evaluations table now exists (migrations/0002_markets_and_judge.sql: id, agent_id FK'd to agents, run_id, judge_model, verdict, score, rationale_summary, telemetry_event_id, created_at), and POST /v1/telemetry/ingest's TelemetryIngestRequest gained an optional judge_evaluation field (handlers::JudgeEvaluationDto) that persists into it when present. Deliberately NOT part of the signed envelope — the signable JSON ingest_telemetry builds for crypto::verify_agent_signature does not include judge_evaluation, so adding one never retroactively breaks an already-signed payload from a client that predates this field; it rides along as an unauthenticated sidecar on an otherwise-authenticated request.

Still [PLANNED]: the actual judge implementation. There is no client.py submit_judge_evaluation(...) SDK helper, and no judge prompt/rubric anywhere in this repo — "Xibalba Solutions defines" the rubric and it isn't specified yet. This pass built the storage + ingestion plumbing only, exactly the scope asked for; not wired into the AIS formula in scoring-core (that would be its own weights/ formula decision requiring explicit sign-off).

flowchart LR
    Agent["Agent process<br/>(prompt/completion text)"]
    Redactor["Redactor (SDK, client-side)<br/>SSN / API-key / email / phone / MRN / etc<br/>-- only if redact_phi=True, default False"]
    Envelope["Signed telemetry envelope<br/>(otel_spans, judge_evaluation)"]
    PHIrs["oracle phi.rs backstop<br/>(server-side, same categories, ALWAYS runs)"]
    DB["Postgres<br/>(telemetry_events, judge_evaluations)"]
    Reject["400 PhiDetected<br/>(before any DB/RPC work)"]

    Agent -->|redact_phi=True| Redactor --> Envelope
    Agent -->|redact_phi=False, default| Envelope
    Envelope -->|POST /v1/telemetry/ingest| PHIrs
    PHIrs -->|unredacted pattern found| Reject
    PHIrs -->|clean| DB

Loading

Oracle-side defense in depth — built

POST /v1/telemetry/ingest now independently rejects (400, AppError::PhiDetected, real HTTP status — not a silent drop) any payload carrying a recognized unredacted raw PHI/PII/secret pattern, via integrity-oracle/backend/src/phi.rs. This mirrors (not reinvents) redactor.py's regex categories (PRIVATE_KEY, API_KEY, SSN, CREDIT_CARD, EMAIL, PHONE, MRN) closely enough to be a real backstop, not a decorative one — same intent per category, translated to Rust's regex crate syntax. Runs first in the handler, before any DB/RPC work, over every JSON string leaf in otel_spans plus the optional judge_evaluation (numeric/structural fields are skipped — the SDK's Redactor never touches those either, so scanning them would just false-positive on values nobody was ever going to redact). Verified to NOT re-flag the SDK's own [REDACTED:{CATEGORY}] output (a compliant, already-redacted payload must pass). Unit-tested in phi.rs itself (every category, the non-reflag case, the numeric/string scan boundary) and exercised at the real HTTP layer in tests/e2e.rs (a raw SSN in an otel span → real 400, before signature verification even runs). The oracle no longer relies solely on the SDK-side Redactor for this invariant — belt and suspenders, as designed, not a substitute for it (this is still regex/heuristic, not certified de-identification; a name mentioned in free text with no structural marker still gets through both layers).

Dual-mode storage — [PLANNED], bigger roadmap item

Mode 1 (transparent): full trace storage for non-regulated verticals, prioritizing developer debugging visibility. Mode 2 (Sovereign ZK-mode, for Integrity Health/healthcare and any PHI-adjacent vertical): raw content never leaves local hardware at all — only a hash and a ZK proof of correct measurement leave the agent's process. Not built as a toggle; tracked in the root README.md's "Vision & long-term roadmap" section.

What this page does NOT claim (correcting the old wiki)

The old wiki's observability-vtl.md described a Trace Aggregator, a Risk Analyzer (privilege escalation / agentic loop / hallucination detection), a GuardrailEngine with pluggable BaseGuardrail classes, a StateStore supporting agent-state "time travel" (rewind/edit/fork), a ReputationLedger, and a standalone integrity-observability-backend service. None of these exist in integrity-sdk or integrity-oracle today. Only the Redactor and the (not-yet-built) judge-evaluation ingestion path are real/planned per the current design; treat the rest of that old page as unbuilt product ideation, not documentation.

Related: Telemetry Ingestion Pipeline (the full end-to-end collection→batching→signing→oracle-pipeline writeup this page's redact_phi section is one slice of), local metrology, AIS, ComplianceGate, integrity-sdk, AIS API — Versioned Wire Spec (the telemetry-envelope signing fix that made this pipeline actually reach the oracle for the first time is documented there and on integrity-oracle).