safe-fetch

Secure async web fetching for AI agents. Combines pre-request secret/PII scanning, SSRF blocking, and post-fetch prompt injection detection into a single safe_fetch() call.

Quick Start

from safe_fetch import safe_fetch, SafeFetchConfig, Policy

result = await safe_fetch("https://example.com/article")
print(result.content)          # wrapped, LLM-ready markdown
print(result.raw_content)      # unwrapped markdown
print(result.safe_content)     # unwrapped safe markdown
print(result.metadata.final_url)
print(result.risk.level)
print(result.extraction_method)  # how it was obtained
print(result.response_findings)  # injection findings (empty if clean)

# Custom policy
config = SafeFetchConfig(
    request_policy=Policy.STRICT,   # raise on secrets/PII in URL/headers
    response_policy=Policy.WARN,    # log + redact injections, don't raise
)
result = await safe_fetch("https://example.com/docs", config=config)

Policy Modes

Both request_policy and response_policy accept a Policy enum value.

Policy	Request behavior	Response behavior
STRICT	Raise SecretLeakError / PIILeakError on any finding	Raise InjectionDetectedError on any finding
WARN	Log finding, record in request_findings, continue	Log finding, redact matched content, record in response_findings
PERMISSIVE	Record in request_findings, continue	Record in response_findings, return content unmodified

Note: SSRF blocking is always enforced regardless of request_policy. SSRFBlockedError is never suppressed.

Defaults: request_policy=STRICT, response_policy=WARN

SafeFetchConfig

@dataclass
class SafeFetchConfig:
    request_policy: Policy = Policy.STRICT
    response_policy: Policy = Policy.WARN
    connect_timeout: float = 10.0        # seconds
    read_timeout: float = 30.0           # seconds
    total_timeout: float | None = 60.0
    max_response_bytes: int = 10_000_000
    max_redirects: int = 5
    allow_http: bool = True
    allowed_hosts: set[str] = field(default_factory=set)
    blocked_cidrs: set[str] = field(default_factory=set)
    allowed_content_types: set[str] = field(default_factory=lambda: {
        "text/html", "text/plain", "text/markdown", "application/xhtml+xml"
    })
    safe_markdown: bool = True
    page_links: bool = True              # append a "Links on this page" section
    max_page_links: int = 100
    llm_client: Any = None               # optional; must implement classify_injection(text) -> bool
    classifier_timeout: float = 5.0
    classifier_failure_policy: Policy = Policy.WARN
    user_agent: str = "safe-fetch/1.0 (LLM-agent)"
    extra_headers: dict = field(default_factory=dict)

Preset constructors are available:

SafeFetchConfig.agent_default()
SafeFetchConfig.strict_enterprise()
SafeFetchConfig.permissive_research()

Fetch Pipeline

Content is retrieved in this order, returning at the first success:

1. Content negotiation — Accept: text/markdown header; if server returns markdown/plain text, used directly (extraction_method="content-negotiation")
2. .md probe — for HTML responses, probes the same path with .md appended while extraction runs (extraction_method="md-probe")
3. trafilatura — extracts main content from HTML as markdown (extraction_method="trafilatura")
4. readability + markdownify — fallback HTML extraction (extraction_method="readability+markdownify")

The pipeline canonicalizes every caller URL, redirect target, and .md probe before network access. By default it blocks non-global destination addresses, local hostnames, ambiguous IP encodings, URL credentials, fragments, malformed ports, unsupported content types, non-2xx statuses, oversized responses, and redirect chains beyond max_redirects.

Safe Markdown and Metadata

result.raw_content preserves extracted Markdown after response scanning. result.safe_content neutralizes Markdown images, reference images, raw HTML, comments, SVG/script/template/noscript blocks, autolinks, and active links while preserving visible text. result.content wraps safe_content in <web_content> boundary tags with a redacted source URL and timestamp.

result.metadata, result.integrity, result.safety_events, and result.risk provide provenance, SHA-256 content hashes, sanitizer/neutralization/classifier events, and an advisory risk score with reasons. The risk score is not a trust guarantee; treat fetched content as untrusted data and keep tools, credentials, and approvals least-privileged.

Links on this page

While inline links are neutralized by default, safe-fetch appends a generated ## Links on this page section listing every discovered link as - [text](url), so an agent can walk a site (e.g. follow docs navigation) one fetch at a time:

...extracted page content...

## Links on this page

- [Getting Started](https://docs.example.com/start)
- [API Reference](https://docs.example.com/api)

Links are harvested from the whole sanitized page (navigation and sidebars included; hidden-element links are already stripped) — or from the Markdown source for .md responses — then resolved against the final URL and filtered through the same scheme/host/CIDR policy as link_policy="allow-safe": javascript:/file: schemes, private addresses, and blocked hosts never appear. Link text is treated as untrusted (invisible characters stripped, Markdown structure removed, length-capped). The section is capped at max_page_links (default 100) and reported via page_links safety events (links_appended, links_dropped_policy, links_truncated, harvest_capped), which do not affect the risk score. Set page_links=False to disable.

CLI Environment Controls

The CLI is configured only through SAFE_FETCH_* environment variables, including policy, timeout, response size, redirect, host/CIDR, content-type, status, redaction, safe Markdown, and classifier controls:

SAFE_FETCH_MAX_RESPONSE_BYTES=1048576 \
SAFE_FETCH_ALLOWED_HOST_SUFFIXES=.example.com \
SAFE_FETCH_SAFE_MARKDOWN=true \
safe-fetch --json https://docs.example.com/page

Run safe-fetch --help for the full environment variable and exit code table.

Exception Reference

All exceptions inherit from SafeFetchError.

Exception	When raised
InvalidSchemeError	URL scheme is not http or https
InvalidURLError	URL is malformed, ambiguous, or unsafe
HostPolicyError	Host or CIDR policy rejected the target
SSRFBlockedError	URL resolves to a private/reserved IP (always raised)
SecretLeakError	Secret detected in URL query params or headers (STRICT)
PIILeakError	PII (email, phone, credit card, SSN) detected in URL or headers (STRICT)
FetchTimeoutError	Connect, read, or total timeout
RedirectLimitError	Too many HTTP redirects
ResponseTooLargeError	Response exceeded max_response_bytes
UnsupportedContentTypeError	Content type is not allowlisted
HTTPStatusError	Status policy rejected the response
ClassifierError	Classifier escalation failed under strict policy
ExtractionFailedError	All extraction methods failed
InjectionDetectedError	Injection finding in response (STRICT)

Threat Model and Residual Risk

safe-fetch is designed for untrusted web retrieval in agent and RAG workflows. It reduces SSRF, request secret leakage, hidden-content injection, Markdown exfiltration, and prompt-injection risks, but it does not prove remote content is safe to obey. Downstream systems should still separate instructions from retrieved data, restrict tool permissions, avoid passing secrets into retrieval URLs, prefer source allowlists, and require human approval for sensitive actions.

Detection limits: The regex-based injection scanner catches known patterns and control tokens but cannot detect paraphrased or novel injection attempts on its own. When an llm_client is wired, the LLM classifier becomes the primary detector for novel and paraphrased injection — it runs over the full fetched content on every fetch, independent of whether structural heuristics fired. Code-block scanning uses a narrow high-signal subset for obvious control tokens. Robust agent architectures should:

1. Use content boundary markers (result.content_marker) so the LLM can distinguish retrieved data from trusted instructions.
2. Wire in an LLM classifier (SafeFetchConfig(llm_client=...)) so paraphrased and novel injections are caught semantically, not just by pattern matching.
3. Design tools with least-privilege — retrieved web content should not have the authority to invoke high-impact actions without a human approval step.

Patterns that remain out of scope by design: arbitrary external-CSS-hidden content (requires a rendered browser) and injections in embedded media or binary formats.

from safe_fetch import SafeFetchError

try:
    result = await safe_fetch(url)
except SafeFetchError as e:
    print(f"safe-fetch error: {e}")

LLM Classifier

When an llm_client is provided, safe-fetch classifies fetched content directly for prompt injection on every fetch — catching paraphrased and novel injections that regex patterns cannot detect. The classifier is invoked in up to classifier_max_chunks overlapping windows of classifier_max_chars characters (default: 4 calls × 12,000 chars). A positive verdict produces a HIGH-confidence finding handled per response_policy.

Cost note: Each fetch with an llm_client makes up to classifier_max_chunks classifier calls (default 4). For cost-sensitive deployments, set classifier_mode="escalate" to revert to the cheaper legacy behavior (classifier fires only when structural heuristics produced a MEDIUM finding).

The package exports a reference prompt template and a fail-closed verdict parser to help build the llm_client:

import anthropic
from safe_fetch import (
    REFERENCE_CLASSIFIER_PROMPT,
    SafeFetchConfig,
    parse_classifier_verdict,
    safe_fetch,
)

client = anthropic.Anthropic()

class MyClassifierClient:
    async def classify_injection(self, text: str) -> bool:
        # Substitute content into the reference prompt template
        prompt = REFERENCE_CLASSIFIER_PROMPT.replace("{content}", text)
        response = client.messages.create(
            model="claude-haiku-4-5-20251001",
            max_tokens=5,
            messages=[{"role": "user", "content": prompt}],
        )
        reply = response.content[0].text
        # Fail-closed: only exact "CLEAN" (stripped, case-insensitive) is not adversarial.
        # Any verbose, malformed, or content-hijacked reply is treated as adversarial.
        return parse_classifier_verdict(reply)

config = SafeFetchConfig(llm_client=MyClassifierClient())
result = await safe_fetch(url, config=config)

parse_classifier_verdict is fail-closed: it returns False (not adversarial) only for the exact token CLEAN (whitespace-stripped, case-insensitive). Any other reply — including verbose explanations or content that hijacks the classifier's output — returns True (adversarial). This bias toward false positives is intentional for security.

Installation

uv add safe-fetch
# With optional Playwright support for JS-heavy pages:
uv add "safe-fetch[playwright]"