Skip to content

Repository files navigation

Prompt Intelligence Layer (PIL)

A programmable LLM gateway with a brain. Drop it in front of OpenAI / Anthropic / Gemini / Ollama and get intent-aware routing, prompt templates, PII masking, semantic caching, structured-output validation, cost guardrails, and OpenTelemetry traces — behind a single OpenAI-compatible endpoint.

Status License FastAPI Postgres

        ┌──────────────────────────────────────────────────────────┐
Client ─┤ /v1/chat/completions       (OpenAI-compatible)           │
        │ /api/v1/chat               (PIL-native, extended)        │
        └────────────────────────┬─────────────────────────────────┘
                                 │
                ┌────────────────▼─────────────────┐
                │  Auth · Rate-limit · Budget      │
                └────────────────┬─────────────────┘
                                 │
   ┌─────────┐  ┌─────────┐  ┌───▼────┐  ┌────────┐  ┌──────────┐  ┌────────┐
   │ Intent  │─▶│  PII    │─▶│  RAG   │─▶│ Prompt │─▶│  Token   │─▶│ Cache  │
   │ Classify│  │ Scrubber│  │Retrieve│  │ Builder│  │ Budget   │  │ Lookup │
   └─────────┘  └─────────┘  └────────┘  └────────┘  └──────────┘  └───┬────┘
                                                                       │
                            ┌───────────┐  ┌─────────┐  ┌──────────┐  ┌▼─────────┐
                            │  Audit    │◀─│Validate │◀─│  LLM     │◀─│ Model    │
                            │  Logger   │  │Response │  │  Client  │  │ Router   │
                            └───────────┘  └─────────┘  └──────────┘  └──────────┘
                                 │
                                 ▼
                        Postgres · Redis · OTEL

Why PIL?

LLM apps die in production from a handful of repeatable causes: cost runaway, PII leakage to third parties, brittle structured outputs, no observability, no safe way to ship prompt or model changes. There are good tools for slices of this (LiteLLM, Langfuse, Presidio) — but no one ships the full path as one coherent system.

PIL is the layer that gives you:

  • Cost control — semantic cache, intent-aware routing to cheap models when possible, per-org daily/monthly budgets with circuit breakers.
  • Privacy — Presidio-based PII detection with a pluggable custom-recognizer point; reversible per-request masking before any external API call; privacy_mode pins requests to local Ollama with the fallback chain cleared.
  • Reliability — provider fallback chains, retry with backoff, structured-output validation, replay endpoint for debugging.
  • Observability — OpenTelemetry traces for every stage; Prometheus metrics; a dashboard for cost, latency, intent mix, model mix.
  • Safety to ship changes — hot-reloadable YAML routing policies, shadow mode for new models, golden-set evaluation harness.

How is this different from LiteLLM / Portkey / Helicone?

Feature LiteLLM Portkey Langfuse Helicone PIL
Provider routing
Cost / token tracking partial
Prompt templates (hot-reload) partial
Intent-aware routing
PII masking partial
Semantic cache partial
Structured-output validation
Shadow mode + replay partial
OpenTelemetry native partial partial
OSS self-host partial

The wedge: PIL is the only OSS gateway that treats the prompt itself as a first-class controllable artifact — not just the request envelope.


Table of Contents


Quick Start

Status: M3 (retrieval + validation). Everything M0–M2 ships, plus a document ingestion pipeline (PDF / DOCX / TXT / MD / HTML → chunks → embeddings) with status-tracked async ingestion, a retriever with optional hybrid (pgvector + tsvector) and an optional local cross-encoder reranker, a swappable Qdrant adapter behind the same VectorStore protocol, a JSON-Schema + citation-grounding response validator with one retry-on-fail, and a buffered-streaming path that activates automatically when stream=true collides with JSON output. Right-to-delete now purges documents + chunks too. Shadow mode and the dashboard land in M4.

Prerequisites

  • Docker + Docker Compose
  • An OpenAI-compatible API key (OpenAI direct, or set OPENAI_BASE_URL=https://openrouter.ai/api/v1 for OpenRouter, etc.)

Run it

git clone https://github.com/xRyuk/prompt-intelligence-layer.git
cd prompt-intelligence-layer
cp .env.example .env
# Edit .env: set OPENAI_API_KEY. To use OpenRouter, also set
#   OPENAI_BASE_URL=https://openrouter.ai/api/v1
docker compose up --build

You now have:

  • API at http://localhost:8000 (OpenAPI docs: /docs).
  • Jaeger UI at http://localhost:16686 for traces.
  • Prometheus exposition at http://localhost:8000/api/v1/metrics.

Issue an API key

ORG=$(curl -s -X POST localhost:8000/api/v1/orgs \
  -H "Content-Type: application/json" \
  -d '{"name":"acme","daily_budget_usd":5.0}' | jq -r .id)

KEY=$(curl -s -X POST localhost:8000/api/v1/orgs/$ORG/keys \
  -H "Content-Type: application/json" \
  -d '{"label":"dev"}' | jq -r .plaintext)

echo $KEY   # pil_sk_…  (returned exactly once)

Point your existing OpenAI SDK at PIL

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="pil_sk_…",                # the PIL key from above, NOT OpenAI's
)

resp = client.chat.completions.create(
    model="gpt-4o-mini",               # explicit model = byte-perfect passthrough
    messages=[{"role": "user", "content": "ping"}],
)
print(resp.choices[0].message.content)
# Cost, request_id, trace_id, and cache-hit live on the response headers:
#   x-pil-request-id, x-pil-trace-id, x-pil-cost-usd, x-pil-cache-hit

Let PIL pick the model (M1)

Pass model="auto" to drive the full pipeline (intent classify → template render → token budget → route → call). The response is still OpenAI-shaped; routing metadata rides on a non-standard pil block:

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarize the French Revolution in 3 bullets."}],
)
# resp.pil → {"intent": "summarization", "routing_rule": "default",
#             "provider_used": "openai", "fallback_used": false, ...}

Or call the PIL-native endpoint directly with extended fields:

curl -X POST localhost:8000/api/v1/chat \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"Summarize the French Revolution in 3 bullets."}'

Read the audit log

curl -s -H "Authorization: Bearer $KEY" localhost:8000/api/v1/requests | jq

See Docs/operating.md for how to read logs, find traces in Jaeger, and query Prometheus.


Core Concepts

Concept What it means
Intent The classified task type of a request (summarization, legal_risk_analysis, etc.).
Template A YAML system + task scaffold rendered per intent.
Routing rule A when → model mapping; first match wins. Hot-reloadable.
Pipeline stage One step in the request lifecycle. Each stage has a fail-open / fail-closed policy.
Shadow mode A candidate model receives a copy of traffic; its output is logged for comparison, not served.
Replay Re-run a past request_id with an overridden config. Diff appears in the dashboard.
Org / API key Every request is scoped to an org_id and api_key_id. Budgets and rate limits live here.

Architecture

PIL runs as a stateless FastAPI service backed by Postgres (with pgvector), Redis, and any LLM providers you configure.

Client → PIL Gateway → Pipeline
                       ├── Auth + rate-limit + budget
                       ├── Intent classifier        (skippable)
                       ├── PII scrubber             (fail-closed by default)
                       ├── Context retriever        (RAG, optional)
                       ├── Prompt builder           (YAML templates)
                       ├── Token budget manager
                       ├── Semantic cache lookup    (return on hit)
                       ├── Model router             (YAML policy)
                       ├── LLM client               (streaming)
                       ├── Response validator       (schema + grounding)
                       ├── PII restore              (if reversible mode)
                       └── Audit logger + OTEL span
            ↓
        LLM provider (OpenAI / Anthropic / Gemini / Ollama)

See Docs/prd.md §9 for the full request lifecycle and §10 for the data model.


API

Health

GET /health

OpenAI-compatible

POST /v1/chat/completions

Drop-in for openai-python, openai-node, langchain, etc. PIL adds x-pil-* response headers exposing the routing decision, cost, and trace ID.

PIL-native

POST /api/v1/chat

Extended fields enable the full pipeline:

{
  "query": "Review this contract and identify risky clauses.",
  "document_ids": ["contract_789"],
  "task_type": "auto",
  "output_format": "json",
  "privacy_mode": true,
  "max_cost_usd": 0.05
}

Response:

{
  "request_id": "req_01HZ...",
  "intent": "legal_risk_analysis",
  "model_used": "claude-opus-4-7",
  "fallback_used": false,
  "response": {
    "summary": "The document contains multiple clauses that may create legal risk.",
    "risks": [
      {
        "clause": "Termination",
        "risk_level": "Medium",
        "reason": "Allows termination without notice.",
        "suggestion": "Add a minimum notice period.",
        "evidence": "The company may terminate this agreement at any time without prior notice."
      }
    ]
  },
  "usage": {
    "prompt_tokens": 4210,
    "completion_tokens": 780,
    "total_tokens": 4990,
    "estimated_cost_usd": 0.032,
    "latency_ms": 2430,
    "cache_hit": false
  },
  "safety":     { "pii_detected": true, "pii_masked": true, "categories": ["PERSON", "PHONE"] },
  "validation": { "passed": true, "errors": [] },
  "trace_id":   "4bf92f3577b34da6a3ce929d0e0e4736"
}

Documents

POST   /api/v1/documents        # upload PDF / DOCX / TXT / HTML / MD
GET    /api/v1/documents/{id}
DELETE /api/v1/documents/{id}

Operations

POST /api/v1/replay/{request_id}        # re-run with optional config overrides
GET  /api/v1/requests/{request_id}      # full audit detail
GET  /api/v1/metrics                    # Prometheus
GET  /api/v1/dashboard                  # built-in UI (P1)

Admin

POST   /api/v1/orgs                     # create org
POST   /api/v1/orgs/{id}/keys           # issue API key
PUT    /api/v1/orgs/{id}/budget         # set daily / monthly budget
DELETE /api/v1/orgs/{id}/data           # right-to-delete

Routing Policy

Routing lives in config/routing.yaml. Edit it, save it, and PIL hot-reloads — no restart.

default_model: gpt-4o-mini
fallback_chain: [gpt-4o-mini, claude-haiku-4-5, llama3.1:8b]

rules:
  - name: privacy_mode_local_only
    when: { privacy_mode: true }
    model: llama3.1:8b
    provider: ollama

  - name: legal_to_strong_model
    when: { intent: legal_risk_analysis }
    model: claude-opus-4-7

  - name: code_gen_to_sonnet
    when: { intent: code_generation }
    model: claude-sonnet-4-6

  - name: summary_to_mini
    when:
      intent: [summarization, classification]
      max_tokens_lte: 800
    model: gpt-4o-mini

shadow:
  enabled: true
  candidate_model: claude-haiku-4-5
  traffic_percent: 5

Rules are evaluated top to bottom; first match wins. Anything that doesn't match falls through to default_model.


Prompt Templates

Templates live in app/templates/*.yaml and are keyed by intent. They are also hot-reloadable.

# app/templates/legal_risk_analysis.yaml
name: legal_risk_analysis
version: 3
system_prompt: |
  You are a legal-document analysis assistant.
  Use ONLY the provided context. Do not invent facts.
  If information is insufficient, say so clearly.

task_prompt: |
  Analyze the following document sections and identify legal risks.

output_schema: schemas/legal_risk_v3.json
output_format: json

few_shot_examples: examples/legal_risk_examples.yaml

Versioning is required. PIL stores the template version against every request so a regression can be traced to a prompt change.


Privacy: PII Masking

PII detection runs before any external API call. PIL ships with:

  • Presidio defaults for the international PII set: PERSON, EMAIL_ADDRESS, PHONE_NUMBER, IP_ADDRESS, IBAN_CODE, CREDIT_CARD, LOCATION, URL.
  • A plugin point at app/core/pii/recognizers/ — drop a .py file that exports RECOGNIZERS: list[EntityRecognizer] and PIL auto-registers it at startup. No locale-specific recognizers ship in core (recognizer packs are post-v1); see Docs/examples/recognizers/ for a working ORDER_ID example you can copy.

Three modes (set pii_mode on the request):

  • reversible (default) — the mapping {placeholder: original} is stored in Redis at pii:map:<request_id> with min(300s, request_timeout) TTL, and originals are restored into the response before it leaves the gateway. The key is deleted explicitly after the response is sent.
  • one_way — entities are replaced with stable placeholders (<PERSON_1>, <EMAIL_ADDRESS_1>); no mapping is stored; the caller sees the masked text.
  • off — no scrubbing. Use only when the request body is known PII-free.

Per-request scoping is the M2 guarantee — the mapping lives under a request-id-namespaced key and is unreachable without the request_id. There is no global mapping table.

privacy_mode: true is a stronger guarantee: the request is pinned to the local Ollama provider with the fallback chain cleared. If Ollama is unreachable PIL surfaces 503 PRIVACY_MODE_NO_LOCAL_PROVIDER rather than falling back to a remote provider — your data never leaves the local network.

The scrubber's default fail-policy is closed: if Presidio crashes, the request is aborted with 502 PII_SCRUBBER_UNAVAILABLE and no LLM call is made. Every other pipeline stage has its own configurable policy in config/fail_policy.yaml — hot-reloadable, no restart.


Semantic Cache

Embedding-similarity cache keyed by (org_id, intent, prompt_embedding). Default threshold cosine ≥ 0.97; configurable.

Per-org isolation is enforced — caches never leak across organizations. TTL is configurable; default 24h. Hit rate is exposed on the dashboard and as a Prometheus gauge.

Cache is bypassed when:

  • cache: false is passed on the request,
  • the intent is in the no_cache_intents list,
  • the request includes document_ids that have changed since the cached response was generated.

Documents and Retrieval (M3)

Upload a document, PIL parses + chunks + embeds it, and any future request with document_ids: [<id>] pulls relevant chunks into the prompt as {{ context }}. Supported formats: PDF, DOCX, TXT, MD, HTML.

# Upload — returns immediately with status="pending"
curl -X POST localhost:8000/api/v1/documents \
  -H "Authorization: Bearer $KEY" \
  -F "file=@contract.pdf"

# Poll until ready (status: pending → parsing → chunking → embedding → ready)
curl -H "Authorization: Bearer $KEY" \
  localhost:8000/api/v1/documents/$DOC_ID

# Ask the document a question. The template's `requires_context: true`
# field triggers retrieval automatically.
curl -X POST localhost:8000/api/v1/chat \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{
    "query": "List risky clauses in this contract.",
    "task_type": "legal_risk_analysis",
    "document_ids": ["'$DOC_ID'"]
  }'

Retrieval knobs

config/retrieval.yaml is hot-reloadable:

defaults:
  k: 8
  rerank: false
  hybrid: false
intents:
  legal_risk_analysis:
    k: 12
    rerank: true
    rerank_to_n: 4
  • Hybrid blends pgvector cosine NN with Postgres tsvector lexical search. Off by default; turning it on improves recall for keyword- shaped queries at the cost of a second SQL query per request.
  • Reranker is a local cross-encoder (BAAI/bge-reranker-base), lazy-loaded on first use. Off by default — paying the model load is the operator's call.
  • Backend switches between pgvector (default, reuses the M0 pg) and qdrant via VECTOR_BACKEND=qdrant. Both implement the same VectorStore protocol; the rest of the pipeline doesn't know which one is wired in.

End-to-end RAG demo

export PIL_BASE_URL=http://localhost:8000
export PIL_API_KEY=pil_sk_...
uv run python scripts/demo_legal_risk.py

Uploads tests/fixtures/sample_contract.md, polls until ready, asks legal_risk_analysis to find risky clauses, validates the structured response, prints the resolved citations, and exits 0 only if all M3 acceptance checks pass (validation OK, at least one grounded citation, no fabrications).


Validation

Two modes:

  1. Streaming pass-through — tokens stream straight back to the client. Best UX, no validation.
  2. Buffered + validated — full response is collected, then validated against the template's output_schema (JSON Schema). On failure, PIL retries once with a stricter prompt; on second failure, returns the validation error to the caller. Use for any structured output you depend on downstream.

When the caller passes stream: true but the resolved template's output_format is json, PIL automatically switches to buffered mode (streaming partial JSON would defeat the validator). The response header x-pil-streaming-buffered: true tells the client that happened.

Validators include:

  • JSON Schema conformance (Draft 2020-12)
  • Required-field presence
  • Enum / range validation (e.g. risk_level ∈ {Low, Medium, High})
  • Citation grounding — every cited chunk ID must exist in the retrieved context; flags fabricated citations
  • Length bounds

Retry is capped at one in code (MAX_VALIDATION_RETRIES = 1 in app/core/validator.py) — even if config/fail_policy.yaml sets a higher number, the second attempt's failure surfaces as 422 VALIDATION_FAILED.

PIL does not claim to detect hallucinations in free-form text. That problem is unsolved. Citation grounding is the closest we can honestly get.


Shadow Mode and Replay

Shadow mode sends a configurable percentage of live traffic to a candidate model in parallel. The candidate's response is logged but not served. The dashboard shows side-by-side cost, latency, and (if the request has a schema) validation pass rate. This is how you safely qualify a model change before flipping the switch.

Replay re-runs any past request with an optional config override:

curl -X POST http://localhost:8000/api/v1/replay/req_01HZ... \
  -H "Content-Type: application/json" \
  -d '{"override_model": "claude-haiku-4-5", "override_template_version": 4}'

The replay result and a diff against the original are stored against the original request_id for inspection.


Observability

  • OpenTelemetry traces — every pipeline stage is a span with semantic conventions (gen_ai.* attributes). Export to Jaeger, Tempo, Honeycomb, or any OTEL collector.
  • Prometheus metrics at /api/v1/metrics:
    • pil_requests_total{org, intent, model, status}
    • pil_request_duration_seconds{stage} (histogram)
    • pil_tokens_total{org, model, kind}
    • pil_cost_usd_total{org, model}
    • pil_cache_hits_total{org} / pil_cache_misses_total{org}
    • pil_validation_failures_total{intent, error_type}
  • Structured JSON logs to stdout — no file logging.
  • Built-in dashboard (P1) — cost, latency P50/P95/P99, intent mix, model mix, cache hit rate, error rate, top spending orgs.

Configuration

.env — only secrets and connection strings live here.

APP_ENV=development
APP_LOG_LEVEL=info

# Providers (at least one required)
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
GEMINI_API_KEY=
OLLAMA_HOST=http://ollama:11434

# Storage
DATABASE_URL=postgresql+asyncpg://pil:pil@postgres:5432/pil
REDIS_URL=redis://redis:6379/0
VECTOR_BACKEND=pgvector           # or: qdrant
QDRANT_URL=

# Toggles (defaults shown)
ENABLE_PII_MASKING=true
ENABLE_SEMANTIC_CACHE=true
ENABLE_RESPONSE_VALIDATION=true
PII_FAIL_POLICY=closed

# Observability
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
PROMETHEUS_ENABLED=true

Everything else — routing, templates, fail policies per stage, budget defaults — lives in config/ as hot-reloadable YAML.


Tech Stack

Layer Choice Why
HTTP / API FastAPI + Uvicorn Async-native, OpenAPI for free
Validation Pydantic v2 Speed + type safety
HTTP client httpx (async) Streams, retries, timeouts
DB Postgres 16 + pgvector + asyncpg One store for metadata and embeddings
Cache + queue Redis 7 Cache, rate-limit counters, PII mappings
Background jobs arq Lightweight async Redis-backed worker
Embeddings OpenAI / BGE / sentence-transformers (configurable) Pluggable
Vector store pgvector (default), Qdrant adapter Simple → scalable
PII Microsoft Presidio + plugin point for custom Battle-tested baseline + drop-in extension
Tracing OpenTelemetry SDK Vendor-neutral
Metrics prometheus-client Standard
Containers Docker, Compose for dev, Helm for k8s (P2) Standard

Repository Layout

prompt-intelligence-layer/
├── app/
│   ├── main.py
│   ├── settings.py
│   ├── api/
│   │   ├── openai_compat.py        # /v1/chat/completions
│   │   ├── chat.py                 # /api/v1/chat
│   │   ├── documents.py
│   │   ├── orgs.py
│   │   ├── replay.py
│   │   └── metrics.py
│   ├── core/
│   │   ├── pipeline.py             # orchestrator
│   │   ├── intent_classifier.py
│   │   ├── pii/
│   │   │   ├── scrubber.py
│   │   │   ├── recognizers_in.py
│   │   │   └── reversible_store.py
│   │   ├── retrieval/
│   │   │   ├── chunker.py
│   │   │   ├── embeddings.py
│   │   │   ├── vector_store.py     # adapter pattern
│   │   │   └── reranker.py
│   │   ├── prompt_builder.py
│   │   ├── token_manager.py
│   │   ├── cache.py                # semantic cache
│   │   ├── router.py               # YAML policy engine
│   │   ├── llm/
│   │   │   ├── base.py
│   │   │   ├── openai.py
│   │   │   ├── anthropic.py
│   │   │   ├── gemini.py
│   │   │   └── ollama.py
│   │   ├── validator.py
│   │   └── audit.py
│   ├── schemas/                    # Pydantic + JSON Schemas
│   ├── db/
│   │   ├── models.py
│   │   ├── session.py
│   │   └── migrations/             # alembic
│   ├── observability/
│   │   ├── tracing.py
│   │   └── metrics.py
│   └── utils/
├── config/
│   ├── routing.yaml
│   ├── budgets.yaml
│   └── fail_policy.yaml
├── templates/
│   ├── general_chat.yaml
│   ├── summarization.yaml
│   ├── legal_risk_analysis.yaml
│   ├── contract_review.yaml
│   └── code_generation.yaml
├── eval/
│   ├── golden_set/                 # versioned eval prompts
│   ├── run.py                      # eval harness
│   └── reports/
├── tests/
│   ├── unit/
│   ├── integration/
│   └── load/
├── docs/
│   ├── architecture.md
│   ├── api_reference.md
│   └── operating.md
├── docker-compose.yml
├── Dockerfile
├── pyproject.toml
├── .env.example
├── PRD.md
├── README.md
└── LICENSE

Roadmap

Milestone Window Ships
M0 Weeks 1–2 FastAPI skeleton · OpenAI provider · auth · budgets · structured logs · OTEL · eval scaffolding
M1 Weeks 3–5 Intent classifier · YAML templates · token manager · YAML routing · all four providers
M2 Weeks 6–7 PII scrubber (incl. IN rules) · reversible mapping · semantic cache · per-stage fail-policies
M3 Weeks 8–10 RAG pipeline · pgvector · reranker · citation-grounded validator · legal-risk demo
M4 Weeks 11–12 Shadow mode · replay · dashboard · cost circuit-breakers · v1.0.0
M5+ Post-v1 Helm chart · prompt registry with diffing · RBAC · eval-as-service · Kubernetes operator

Non-Goals

  • Not a model host. PIL routes to providers; it does not serve models.
  • Not a prompt IDE. Templates are YAML; authoring lives in your editor or whatever tool you like.
  • Not a hallucination detector. That's an open research problem; we don't claim it. We do citation grounding.
  • Not a vector database. We integrate with one.
  • Not a fine-tuning platform.
  • Not a chat UI. API + dashboards only in v1.

Contributing

Pre-alpha. If you want to play, open an issue first — the surface area is still moving. Please don't open PRs that add a new module before the existing ones are stable.

Conventions:

  • Python 3.12+, ruff + black, mypy --strict on app/core.
  • All new modules ship with unit tests and an entry in the eval harness.
  • Anything user-visible needs a docs update in the same PR.

License

MIT. See LICENSE.


Acknowledgements

PIL stands on the shoulders of Presidio, pgvector, FastAPI, and the OSS LLM tooling community. The wedge — prompt-level intelligence over provider-level routing — was shaped by watching real production failures of LLM apps that already had a gateway, but no brain behind it.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages