v0.1.18 - Guardrail Detection

Overview

v0.1.18 introduces Guardrail Detection — a passive, zero-overhead observability layer for agent security and compliance. It ships as an OTel span processor that auto-registers on traccia.init(), requires no changes to existing agent code, and writes structured findings directly onto trace spans so all exporters see them.

The feature covers the full lifecycle: detecting guardrails that exist (explicit annotation, provider signals, heuristics), evaluating which expected guardrail categories are missing given the agent's observed capabilities, and writing a rich GuardrailSummary to the root span.

What's new

Guardrail detection engine (new)

Auto-registered processor. GuardrailDetectorProcessor is registered automatically by traccia.init(). No manual processor setup required. It is trace-ID scoped, so concurrent agent runs in the same process produce isolated results.

Three detection tiers:

Tier	source_type	confidence	How
A — Explicit	explicit	high / medium	@observe(as_type="guardrail") or guardrail_span()
B — Provider-native	provider_native	high / medium	LLM finish reason, stop reason, safety ratings
C — Heuristic	heuristic	always low	Denial keywords in tool error messages

Missing-guardrail evaluator. At root span end, infers agent capabilities (calls_llm, handles_user_text, produces_user_text, uses_tools) from trace spans and reports which guardrail categories are expected but absent.

Span output. All data is written as span attributes:

• Per guardrail-signal span: guardrail.findings (JSON), guardrail.finding.count
• Root span: guardrail.summary (full JSON), guardrail.summary.coverage_confidence, guardrail.summary.missing_count, guardrail.summary.detected_categories

---

Explicit annotation — Tier A

guardrail_span() context manager — recommended for inline guardrail checks:

from traccia.guardrails import guardrail_span

with guardrail_span("pii_check", category="pii", enforcement_mode="warn") as span:
    result = run_pii_scanner(user_input)
    span.set_attribute("guardrail.triggered", result.found_pii)

@observe(as_type="guardrail") decorator — for function-level guardrails. If the function returns a bool, guardrail.triggered is set automatically — no manual span access needed:

from traccia import observe

@observe(
    as_type="guardrail",
    attributes={
        "guardrail.name": "injection_check",
        "guardrail.category": "prompt_injection",
        "guardrail.enforcement_mode": "block",
    },
)
def check_injection(text: str) -> bool:
    return any(kw in text.lower() for kw in INJECTION_KEYWORDS)
    # True → triggered, False → not triggered. Set automatically from bool return.

Pre-setting guardrail.triggered in attributes={} takes precedence and is never overridden. Works for both sync and async functions.

Confidence: high when all three required attributes (guardrail.name, guardrail.category, guardrail.triggered) are present; downgrades to medium otherwise with a warning logged to traccia.guardrails.

---

Provider-native detection — Tier B (automatic)

Detected automatically from existing LLM span attributes — no annotation required:

Provider	Signal	Attribute	Confidence
OpenAI	finish_reason = "content_filter"	llm.finish_reason	HIGH
Azure OpenAI	finish_reason = "content_filtered"	llm.finish_reason	HIGH
Google GenAI	finish_reason = "SAFETY"	llm.finish_reason	HIGH
Anthropic	stop_reason = "content_filter" or "content_filtered" and llm.vendor is anthropic or claude	llm.stop_reason + llm.vendor	HIGH
Anthropic	Policy phrases in error.message and vendor anthropic/claude and span looks like an LLM call (span.type=llm or llm.model / llm.prompt / llm.completion / finish or stop reason present)	error.message + LLM signals	MEDIUM
Google / LangChain	"blocked": true or "probability": "HIGH" in safety ratings	llm.safety_ratings	MEDIUM
Any	Incomplete response + refusal text in completion	llm.response.status + llm.completion	MEDIUM

---

Heuristic detection — Tier C

When a tool/function span raises an error with denial-like keywords in the message (permission, denied, unauthorized, forbidden, not allowed), a tool_permission finding is emitted at low confidence.

Infrastructure error exclusions. The following error types are excluded regardless of message content to avoid false positives from timeouts and network failures: TimeoutError, asyncio.TimeoutError, ConnectionError, ConnectionRefusedError, ConnectionResetError, OSError, socket.timeout, requests.exceptions.Timeout, requests.exceptions.ConnectionError, httpx.TimeoutException, httpx.ConnectError.

Tier C findings do not count as coverage. A low-confidence heuristic finding appears in detected_categories but does not remove the category from missing_categories. The reasoning: seeing a possible denial is not the same as having a guardrail. Both signals are preserved.

Disabling Tier C via init() (no custom processors). Tier A and Tier B stay enabled.

from traccia import init

init(guardrail_heuristics=False)

Also: environment variable TRACCIA_GUARDRAIL_HEURISTICS=false, or traccia.toml:

[instrumentation]
guardrail_heuristics = false

GuardrailDetectorProcessor(heuristics_enabled=...) is wired from this setting automatically.

---

Capability inference and missing-guardrail evaluation

The evaluator infers capabilities from all spans in a trace and maps them to expected guardrail categories:

Observed capability	Expected categories	Missing confidence
calls_llm + handles_user_text	input_validation, prompt_injection	MEDIUM
produces_user_text	output_validation, moderation	MEDIUM
uses_tools	tool_permission	HIGH

Why MEDIUM for input_validation / prompt_injection? llm.prompt is present on any LLM call — including batch pipelines over internal documents. The capability signal is reliable; the inference that the prompt is user-provided is not. MEDIUM accurately communicates this uncertainty.

why_required text: "Agent makes LLM calls with prompt data (may be user-provided)"

evaluate_run(..., heuristics_enabled=False) drops Tier C findings from the summary (detected/triggered/coverage/limitations), matching the processor when heuristics are off. Useful for direct programmatic use of the evaluator.

---

SDK-level suppression for false positive missing warnings

Batch pipelines and internal-only agents can suppress specific missing-guardrail categories without disabling detection:

from traccia.guardrails import guardrail_span, ATTR_GUARDRAIL_SUPPRESS_MISSING

# Convenience parameter on guardrail_span
with guardrail_span(
    "pipeline_root",
    category="unknown",
    suppress_missing=["prompt_injection", "input_validation"],
):
    run_batch_pipeline(documents)

# Or set directly on any span
span.set_attribute(
    ATTR_GUARDRAIL_SUPPRESS_MISSING,
    ["prompt_injection", "input_validation"],
)

Can be set on any span in the run. Affects only missing_categories — detected findings and triggered categories are unaffected.

---

New public API surface

traccia.guardrails module exports:

Name	Type	Description
guardrail_span()	Context manager	Create a guardrail-typed span with pre-filled attributes
GuardrailCategory	Enum	input_validation, prompt_injection, pii, moderation, tool_permission, output_validation, rate_limit, unknown
GuardrailFinding	Dataclass	Single guardrail signal with category, confidence, triggered state, evidence
GuardrailSummary	Dataclass	Aggregated run-level summary
MissingGuardrail	Dataclass	Expected-but-absent guardrail category
SourceType	Enum	explicit, provider_native, heuristic
Confidence	Enum	high, medium, low
EnforcementMode	Enum	block, warn, log_only, unknown
ATTR_GUARDRAIL_*	str constants	Reserved span attribute keys
ATTR_GUARDRAIL_SUPPRESS_MISSING	str	"traccia.guardrail.suppress_missing"
validate_guardrail_attributes()	Function	Returns warnings for incomplete guardrail spans; optional require_triggered=False for decorator use

@observe(as_type="guardrail") — decorator validates with require_triggered=False so missing guardrail.triggered in attributes={} does not warn when it will be set from a bool return value.

Processor resilience. GuardrailDetectorProcessor logs WARNING on on_end / summary failures (traccia.guardrails.processor); trace export continues. Set that logger to DEBUG for tracebacks. Failed processor registration logs via traccia.auto instead of failing silently.

Config file init() kwargs. load_config_with_priority flat-key mapping now includes openai_agents, crewai, and guardrail_heuristics under [instrumentation] (previously openai_agents / crewai could be dropped when passed flat from init()).

---

New constants in traccia.guardrails.constants

• ATTR_LLM_STOP_REASON = "llm.stop_reason" — Anthropic stop reason attribute
• ATTR_LLM_SAFETY_RATINGS = "llm.safety_ratings" — Google/LangChain safety ratings JSON
• ATTR_GUARDRAIL_SUPPRESS_MISSING = "traccia.guardrail.suppress_missing" — per-run suppression

---

Hard limits — what cannot be captured

These are fundamental limits of trace-based inference:

• Out-of-band guardrails (API gateways, proxies, external validators) are invisible unless they write span attributes.
• Presence ≠ correctness — a misconfigured guardrail that always returns False looks identical to a working one.
• Absence ≠ missing — no guardrail span could mean no guardrail exists, or it exists but is out-of-band. The evaluator always surfaces this in limitations.
• llm.prompt ≠ user input — batch pipelines and user-facing agents look identical. Suppression handles this.
• Prompt injection cannot be inferred from model output — only an explicit span proves the check ran.
• Tier C misses unusual phrasing — exact keyword matching only. Unusual denial messages require explicit annotation.