A programmable LLM gateway with a brain. Drop it in front of OpenAI / Anthropic / Gemini / Ollama and get intent-aware routing, prompt templates, PII masking, semantic caching, structured-output validation, cost guardrails, and OpenTelemetry traces — behind a single OpenAI-compatible endpoint.
┌──────────────────────────────────────────────────────────┐
Client ─┤ /v1/chat/completions (OpenAI-compatible) │
│ /api/v1/chat (PIL-native, extended) │
└────────────────────────┬─────────────────────────────────┘
│
┌────────────────▼─────────────────┐
│ Auth · Rate-limit · Budget │
└────────────────┬─────────────────┘
│
┌─────────┐ ┌─────────┐ ┌───▼────┐ ┌────────┐ ┌──────────┐ ┌────────┐
│ Intent │─▶│ PII │─▶│ RAG │─▶│ Prompt │─▶│ Token │─▶│ Cache │
│ Classify│ │ Scrubber│ │Retrieve│ │ Builder│ │ Budget │ │ Lookup │
└─────────┘ └─────────┘ └────────┘ └────────┘ └──────────┘ └───┬────┘
│
┌───────────┐ ┌─────────┐ ┌──────────┐ ┌▼─────────┐
│ Audit │◀─│Validate │◀─│ LLM │◀─│ Model │
│ Logger │ │Response │ │ Client │ │ Router │
└───────────┘ └─────────┘ └──────────┘ └──────────┘
│
▼
Postgres · Redis · OTEL
LLM apps die in production from a handful of repeatable causes: cost runaway, PII leakage to third parties, brittle structured outputs, no observability, no safe way to ship prompt or model changes. There are good tools for slices of this (LiteLLM, Langfuse, Presidio) — but no one ships the full path as one coherent system.
PIL is the layer that gives you:
- Cost control — semantic cache, intent-aware routing to cheap models when possible, per-org daily/monthly budgets with circuit breakers.
- Privacy — Presidio-based PII detection with a pluggable custom-recognizer point; reversible per-request masking before any external API call;
privacy_modepins requests to local Ollama with the fallback chain cleared. - Reliability — provider fallback chains, retry with backoff, structured-output validation, replay endpoint for debugging.
- Observability — OpenTelemetry traces for every stage; Prometheus metrics; a dashboard for cost, latency, intent mix, model mix.
- Safety to ship changes — hot-reloadable YAML routing policies, shadow mode for new models, golden-set evaluation harness.
| Feature | LiteLLM | Portkey | Langfuse | Helicone | PIL |
|---|---|---|---|---|---|
| Provider routing | ✅ | ✅ | ❌ | ❌ | ✅ |
| Cost / token tracking | partial | ✅ | ✅ | ✅ | ✅ |
| Prompt templates (hot-reload) | ❌ | partial | ✅ | ❌ | ✅ |
| Intent-aware routing | ❌ | ❌ | ❌ | ❌ | ✅ |
| PII masking | ❌ | partial | ❌ | ❌ | ✅ |
| Semantic cache | ❌ | ✅ | ❌ | partial | ✅ |
| Structured-output validation | ❌ | ❌ | ❌ | ❌ | ✅ |
| Shadow mode + replay | ❌ | partial | ❌ | ❌ | ✅ |
| OpenTelemetry native | partial | partial | ✅ | ✅ | ✅ |
| OSS self-host | ✅ | partial | ✅ | ✅ | ✅ |
The wedge: PIL is the only OSS gateway that treats the prompt itself as a first-class controllable artifact — not just the request envelope.
- Quick Start
- Core Concepts
- Architecture
- API
- Routing Policy
- Prompt Templates
- Privacy: PII Masking
- Semantic Cache
- Validation
- Shadow Mode and Replay
- Observability
- Configuration
- Tech Stack
- Repository Layout
- Roadmap
- Non-Goals
- Contributing
- License
Status: M3 (retrieval + validation). Everything M0–M2 ships, plus a document ingestion pipeline (PDF / DOCX / TXT / MD / HTML → chunks → embeddings) with status-tracked async ingestion, a retriever with optional hybrid (pgvector + tsvector) and an optional local cross-encoder reranker, a swappable Qdrant adapter behind the same
VectorStoreprotocol, a JSON-Schema + citation-grounding response validator with one retry-on-fail, and a buffered-streaming path that activates automatically whenstream=truecollides with JSON output. Right-to-delete now purges documents + chunks too. Shadow mode and the dashboard land in M4.
- Docker + Docker Compose
- An OpenAI-compatible API key (OpenAI direct, or set
OPENAI_BASE_URL=https://openrouter.ai/api/v1for OpenRouter, etc.)
git clone https://github.com/xRyuk/prompt-intelligence-layer.git
cd prompt-intelligence-layer
cp .env.example .env
# Edit .env: set OPENAI_API_KEY. To use OpenRouter, also set
# OPENAI_BASE_URL=https://openrouter.ai/api/v1
docker compose up --buildYou now have:
- API at
http://localhost:8000(OpenAPI docs:/docs). - Jaeger UI at
http://localhost:16686for traces. - Prometheus exposition at
http://localhost:8000/api/v1/metrics.
ORG=$(curl -s -X POST localhost:8000/api/v1/orgs \
-H "Content-Type: application/json" \
-d '{"name":"acme","daily_budget_usd":5.0}' | jq -r .id)
KEY=$(curl -s -X POST localhost:8000/api/v1/orgs/$ORG/keys \
-H "Content-Type: application/json" \
-d '{"label":"dev"}' | jq -r .plaintext)
echo $KEY # pil_sk_… (returned exactly once)from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="pil_sk_…", # the PIL key from above, NOT OpenAI's
)
resp = client.chat.completions.create(
model="gpt-4o-mini", # explicit model = byte-perfect passthrough
messages=[{"role": "user", "content": "ping"}],
)
print(resp.choices[0].message.content)
# Cost, request_id, trace_id, and cache-hit live on the response headers:
# x-pil-request-id, x-pil-trace-id, x-pil-cost-usd, x-pil-cache-hitPass model="auto" to drive the full pipeline (intent classify →
template render → token budget → route → call). The response is still
OpenAI-shaped; routing metadata rides on a non-standard pil block:
resp = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "Summarize the French Revolution in 3 bullets."}],
)
# resp.pil → {"intent": "summarization", "routing_rule": "default",
# "provider_used": "openai", "fallback_used": false, ...}Or call the PIL-native endpoint directly with extended fields:
curl -X POST localhost:8000/api/v1/chat \
-H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-d '{"query":"Summarize the French Revolution in 3 bullets."}'curl -s -H "Authorization: Bearer $KEY" localhost:8000/api/v1/requests | jqSee Docs/operating.md for how to read logs, find traces in Jaeger, and query Prometheus.
| Concept | What it means |
|---|---|
| Intent | The classified task type of a request (summarization, legal_risk_analysis, etc.). |
| Template | A YAML system + task scaffold rendered per intent. |
| Routing rule | A when → model mapping; first match wins. Hot-reloadable. |
| Pipeline stage | One step in the request lifecycle. Each stage has a fail-open / fail-closed policy. |
| Shadow mode | A candidate model receives a copy of traffic; its output is logged for comparison, not served. |
| Replay | Re-run a past request_id with an overridden config. Diff appears in the dashboard. |
| Org / API key | Every request is scoped to an org_id and api_key_id. Budgets and rate limits live here. |
PIL runs as a stateless FastAPI service backed by Postgres (with pgvector), Redis, and any LLM providers you configure.
Client → PIL Gateway → Pipeline
├── Auth + rate-limit + budget
├── Intent classifier (skippable)
├── PII scrubber (fail-closed by default)
├── Context retriever (RAG, optional)
├── Prompt builder (YAML templates)
├── Token budget manager
├── Semantic cache lookup (return on hit)
├── Model router (YAML policy)
├── LLM client (streaming)
├── Response validator (schema + grounding)
├── PII restore (if reversible mode)
└── Audit logger + OTEL span
↓
LLM provider (OpenAI / Anthropic / Gemini / Ollama)
See Docs/prd.md §9 for the full request lifecycle and §10 for the data model.
GET /healthPOST /v1/chat/completionsDrop-in for openai-python, openai-node, langchain, etc. PIL adds x-pil-* response headers exposing the routing decision, cost, and trace ID.
POST /api/v1/chatExtended fields enable the full pipeline:
{
"query": "Review this contract and identify risky clauses.",
"document_ids": ["contract_789"],
"task_type": "auto",
"output_format": "json",
"privacy_mode": true,
"max_cost_usd": 0.05
}Response:
{
"request_id": "req_01HZ...",
"intent": "legal_risk_analysis",
"model_used": "claude-opus-4-7",
"fallback_used": false,
"response": {
"summary": "The document contains multiple clauses that may create legal risk.",
"risks": [
{
"clause": "Termination",
"risk_level": "Medium",
"reason": "Allows termination without notice.",
"suggestion": "Add a minimum notice period.",
"evidence": "The company may terminate this agreement at any time without prior notice."
}
]
},
"usage": {
"prompt_tokens": 4210,
"completion_tokens": 780,
"total_tokens": 4990,
"estimated_cost_usd": 0.032,
"latency_ms": 2430,
"cache_hit": false
},
"safety": { "pii_detected": true, "pii_masked": true, "categories": ["PERSON", "PHONE"] },
"validation": { "passed": true, "errors": [] },
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}POST /api/v1/documents # upload PDF / DOCX / TXT / HTML / MD
GET /api/v1/documents/{id}
DELETE /api/v1/documents/{id}POST /api/v1/replay/{request_id} # re-run with optional config overrides
GET /api/v1/requests/{request_id} # full audit detail
GET /api/v1/metrics # Prometheus
GET /api/v1/dashboard # built-in UI (P1)POST /api/v1/orgs # create org
POST /api/v1/orgs/{id}/keys # issue API key
PUT /api/v1/orgs/{id}/budget # set daily / monthly budget
DELETE /api/v1/orgs/{id}/data # right-to-deleteRouting lives in config/routing.yaml. Edit it, save it, and PIL hot-reloads — no restart.
default_model: gpt-4o-mini
fallback_chain: [gpt-4o-mini, claude-haiku-4-5, llama3.1:8b]
rules:
- name: privacy_mode_local_only
when: { privacy_mode: true }
model: llama3.1:8b
provider: ollama
- name: legal_to_strong_model
when: { intent: legal_risk_analysis }
model: claude-opus-4-7
- name: code_gen_to_sonnet
when: { intent: code_generation }
model: claude-sonnet-4-6
- name: summary_to_mini
when:
intent: [summarization, classification]
max_tokens_lte: 800
model: gpt-4o-mini
shadow:
enabled: true
candidate_model: claude-haiku-4-5
traffic_percent: 5Rules are evaluated top to bottom; first match wins. Anything that doesn't match falls through to default_model.
Templates live in app/templates/*.yaml and are keyed by intent. They are also hot-reloadable.
# app/templates/legal_risk_analysis.yaml
name: legal_risk_analysis
version: 3
system_prompt: |
You are a legal-document analysis assistant.
Use ONLY the provided context. Do not invent facts.
If information is insufficient, say so clearly.
task_prompt: |
Analyze the following document sections and identify legal risks.
output_schema: schemas/legal_risk_v3.json
output_format: json
few_shot_examples: examples/legal_risk_examples.yamlVersioning is required. PIL stores the template version against every request so a regression can be traced to a prompt change.
PII detection runs before any external API call. PIL ships with:
- Presidio defaults for the international PII set:
PERSON,EMAIL_ADDRESS,PHONE_NUMBER,IP_ADDRESS,IBAN_CODE,CREDIT_CARD,LOCATION,URL. - A plugin point at
app/core/pii/recognizers/— drop a.pyfile that exportsRECOGNIZERS: list[EntityRecognizer]and PIL auto-registers it at startup. No locale-specific recognizers ship in core (recognizer packs are post-v1); seeDocs/examples/recognizers/for a workingORDER_IDexample you can copy.
Three modes (set pii_mode on the request):
reversible(default) — the mapping{placeholder: original}is stored in Redis atpii:map:<request_id>withmin(300s, request_timeout)TTL, and originals are restored into the response before it leaves the gateway. The key is deleted explicitly after the response is sent.one_way— entities are replaced with stable placeholders (<PERSON_1>,<EMAIL_ADDRESS_1>); no mapping is stored; the caller sees the masked text.off— no scrubbing. Use only when the request body is known PII-free.
Per-request scoping is the M2 guarantee — the mapping lives under a request-id-namespaced key and is unreachable without the request_id. There is no global mapping table.
privacy_mode: true is a stronger guarantee: the request is pinned to the local Ollama provider with the fallback chain cleared. If Ollama is unreachable PIL surfaces 503 PRIVACY_MODE_NO_LOCAL_PROVIDER rather than falling back to a remote provider — your data never leaves the local network.
The scrubber's default fail-policy is closed: if Presidio crashes, the request is aborted with 502 PII_SCRUBBER_UNAVAILABLE and no LLM call is made. Every other pipeline stage has its own configurable policy in config/fail_policy.yaml — hot-reloadable, no restart.
Embedding-similarity cache keyed by (org_id, intent, prompt_embedding). Default threshold cosine ≥ 0.97; configurable.
Per-org isolation is enforced — caches never leak across organizations. TTL is configurable; default 24h. Hit rate is exposed on the dashboard and as a Prometheus gauge.
Cache is bypassed when:
cache: falseis passed on the request,- the intent is in the
no_cache_intentslist, - the request includes
document_idsthat have changed since the cached response was generated.
Upload a document, PIL parses + chunks + embeds it, and any future
request with document_ids: [<id>] pulls relevant chunks into the
prompt as {{ context }}. Supported formats: PDF, DOCX, TXT,
MD, HTML.
# Upload — returns immediately with status="pending"
curl -X POST localhost:8000/api/v1/documents \
-H "Authorization: Bearer $KEY" \
-F "file=@contract.pdf"
# Poll until ready (status: pending → parsing → chunking → embedding → ready)
curl -H "Authorization: Bearer $KEY" \
localhost:8000/api/v1/documents/$DOC_ID
# Ask the document a question. The template's `requires_context: true`
# field triggers retrieval automatically.
curl -X POST localhost:8000/api/v1/chat \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{
"query": "List risky clauses in this contract.",
"task_type": "legal_risk_analysis",
"document_ids": ["'$DOC_ID'"]
}'config/retrieval.yaml is hot-reloadable:
defaults:
k: 8
rerank: false
hybrid: false
intents:
legal_risk_analysis:
k: 12
rerank: true
rerank_to_n: 4- Hybrid blends pgvector cosine NN with Postgres
tsvectorlexical search. Off by default; turning it on improves recall for keyword- shaped queries at the cost of a second SQL query per request. - Reranker is a local cross-encoder (
BAAI/bge-reranker-base), lazy-loaded on first use. Off by default — paying the model load is the operator's call. - Backend switches between
pgvector(default, reuses the M0 pg) andqdrantviaVECTOR_BACKEND=qdrant. Both implement the sameVectorStoreprotocol; the rest of the pipeline doesn't know which one is wired in.
export PIL_BASE_URL=http://localhost:8000
export PIL_API_KEY=pil_sk_...
uv run python scripts/demo_legal_risk.pyUploads tests/fixtures/sample_contract.md, polls until ready,
asks legal_risk_analysis to find risky clauses, validates the
structured response, prints the resolved citations, and exits 0 only
if all M3 acceptance checks pass (validation OK, at least one grounded
citation, no fabrications).
Two modes:
- Streaming pass-through — tokens stream straight back to the client. Best UX, no validation.
- Buffered + validated — full response is collected, then validated against the template's
output_schema(JSON Schema). On failure, PIL retries once with a stricter prompt; on second failure, returns the validation error to the caller. Use for any structured output you depend on downstream.
When the caller passes stream: true but the resolved template's
output_format is json, PIL automatically switches to buffered mode
(streaming partial JSON would defeat the validator). The response
header x-pil-streaming-buffered: true tells the client that
happened.
Validators include:
- JSON Schema conformance (Draft 2020-12)
- Required-field presence
- Enum / range validation (e.g.
risk_level ∈ {Low, Medium, High}) - Citation grounding — every cited chunk ID must exist in the retrieved context; flags fabricated citations
- Length bounds
Retry is capped at one in code (MAX_VALIDATION_RETRIES = 1 in
app/core/validator.py) — even if config/fail_policy.yaml sets a
higher number, the second attempt's failure surfaces as
422 VALIDATION_FAILED.
PIL does not claim to detect hallucinations in free-form text. That problem is unsolved. Citation grounding is the closest we can honestly get.
Shadow mode sends a configurable percentage of live traffic to a candidate model in parallel. The candidate's response is logged but not served. The dashboard shows side-by-side cost, latency, and (if the request has a schema) validation pass rate. This is how you safely qualify a model change before flipping the switch.
Replay re-runs any past request with an optional config override:
curl -X POST http://localhost:8000/api/v1/replay/req_01HZ... \
-H "Content-Type: application/json" \
-d '{"override_model": "claude-haiku-4-5", "override_template_version": 4}'The replay result and a diff against the original are stored against the original request_id for inspection.
- OpenTelemetry traces — every pipeline stage is a span with semantic conventions (
gen_ai.*attributes). Export to Jaeger, Tempo, Honeycomb, or any OTEL collector. - Prometheus metrics at
/api/v1/metrics:pil_requests_total{org, intent, model, status}pil_request_duration_seconds{stage}(histogram)pil_tokens_total{org, model, kind}pil_cost_usd_total{org, model}pil_cache_hits_total{org}/pil_cache_misses_total{org}pil_validation_failures_total{intent, error_type}
- Structured JSON logs to stdout — no file logging.
- Built-in dashboard (P1) — cost, latency P50/P95/P99, intent mix, model mix, cache hit rate, error rate, top spending orgs.
.env — only secrets and connection strings live here.
APP_ENV=development
APP_LOG_LEVEL=info
# Providers (at least one required)
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
GEMINI_API_KEY=
OLLAMA_HOST=http://ollama:11434
# Storage
DATABASE_URL=postgresql+asyncpg://pil:pil@postgres:5432/pil
REDIS_URL=redis://redis:6379/0
VECTOR_BACKEND=pgvector # or: qdrant
QDRANT_URL=
# Toggles (defaults shown)
ENABLE_PII_MASKING=true
ENABLE_SEMANTIC_CACHE=true
ENABLE_RESPONSE_VALIDATION=true
PII_FAIL_POLICY=closed
# Observability
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
PROMETHEUS_ENABLED=trueEverything else — routing, templates, fail policies per stage, budget defaults — lives in config/ as hot-reloadable YAML.
| Layer | Choice | Why |
|---|---|---|
| HTTP / API | FastAPI + Uvicorn | Async-native, OpenAPI for free |
| Validation | Pydantic v2 | Speed + type safety |
| HTTP client | httpx (async) | Streams, retries, timeouts |
| DB | Postgres 16 + pgvector + asyncpg | One store for metadata and embeddings |
| Cache + queue | Redis 7 | Cache, rate-limit counters, PII mappings |
| Background jobs | arq | Lightweight async Redis-backed worker |
| Embeddings | OpenAI / BGE / sentence-transformers (configurable) | Pluggable |
| Vector store | pgvector (default), Qdrant adapter | Simple → scalable |
| PII | Microsoft Presidio + plugin point for custom | Battle-tested baseline + drop-in extension |
| Tracing | OpenTelemetry SDK | Vendor-neutral |
| Metrics | prometheus-client | Standard |
| Containers | Docker, Compose for dev, Helm for k8s (P2) | Standard |
prompt-intelligence-layer/
├── app/
│ ├── main.py
│ ├── settings.py
│ ├── api/
│ │ ├── openai_compat.py # /v1/chat/completions
│ │ ├── chat.py # /api/v1/chat
│ │ ├── documents.py
│ │ ├── orgs.py
│ │ ├── replay.py
│ │ └── metrics.py
│ ├── core/
│ │ ├── pipeline.py # orchestrator
│ │ ├── intent_classifier.py
│ │ ├── pii/
│ │ │ ├── scrubber.py
│ │ │ ├── recognizers_in.py
│ │ │ └── reversible_store.py
│ │ ├── retrieval/
│ │ │ ├── chunker.py
│ │ │ ├── embeddings.py
│ │ │ ├── vector_store.py # adapter pattern
│ │ │ └── reranker.py
│ │ ├── prompt_builder.py
│ │ ├── token_manager.py
│ │ ├── cache.py # semantic cache
│ │ ├── router.py # YAML policy engine
│ │ ├── llm/
│ │ │ ├── base.py
│ │ │ ├── openai.py
│ │ │ ├── anthropic.py
│ │ │ ├── gemini.py
│ │ │ └── ollama.py
│ │ ├── validator.py
│ │ └── audit.py
│ ├── schemas/ # Pydantic + JSON Schemas
│ ├── db/
│ │ ├── models.py
│ │ ├── session.py
│ │ └── migrations/ # alembic
│ ├── observability/
│ │ ├── tracing.py
│ │ └── metrics.py
│ └── utils/
├── config/
│ ├── routing.yaml
│ ├── budgets.yaml
│ └── fail_policy.yaml
├── templates/
│ ├── general_chat.yaml
│ ├── summarization.yaml
│ ├── legal_risk_analysis.yaml
│ ├── contract_review.yaml
│ └── code_generation.yaml
├── eval/
│ ├── golden_set/ # versioned eval prompts
│ ├── run.py # eval harness
│ └── reports/
├── tests/
│ ├── unit/
│ ├── integration/
│ └── load/
├── docs/
│ ├── architecture.md
│ ├── api_reference.md
│ └── operating.md
├── docker-compose.yml
├── Dockerfile
├── pyproject.toml
├── .env.example
├── PRD.md
├── README.md
└── LICENSE
| Milestone | Window | Ships |
|---|---|---|
| M0 | Weeks 1–2 | FastAPI skeleton · OpenAI provider · auth · budgets · structured logs · OTEL · eval scaffolding |
| M1 | Weeks 3–5 | Intent classifier · YAML templates · token manager · YAML routing · all four providers |
| M2 | Weeks 6–7 | PII scrubber (incl. IN rules) · reversible mapping · semantic cache · per-stage fail-policies |
| M3 | Weeks 8–10 | RAG pipeline · pgvector · reranker · citation-grounded validator · legal-risk demo |
| M4 | Weeks 11–12 | Shadow mode · replay · dashboard · cost circuit-breakers · v1.0.0 |
| M5+ | Post-v1 | Helm chart · prompt registry with diffing · RBAC · eval-as-service · Kubernetes operator |
- Not a model host. PIL routes to providers; it does not serve models.
- Not a prompt IDE. Templates are YAML; authoring lives in your editor or whatever tool you like.
- Not a hallucination detector. That's an open research problem; we don't claim it. We do citation grounding.
- Not a vector database. We integrate with one.
- Not a fine-tuning platform.
- Not a chat UI. API + dashboards only in v1.
Pre-alpha. If you want to play, open an issue first — the surface area is still moving. Please don't open PRs that add a new module before the existing ones are stable.
Conventions:
- Python 3.12+,
ruff+black,mypy --strictonapp/core. - All new modules ship with unit tests and an entry in the eval harness.
- Anything user-visible needs a docs update in the same PR.
MIT. See LICENSE.
PIL stands on the shoulders of Presidio, pgvector, FastAPI, and the OSS LLM tooling community. The wedge — prompt-level intelligence over provider-level routing — was shaped by watching real production failures of LLM apps that already had a gateway, but no brain behind it.