Repository navigation
FAQ
Yes, on port 8090, once an endpoint is configured. Point your client's base URL at the proxy and use a proxy API key as the bearer token. A fresh install boots in onboarding mode with no endpoints and returns 503 for inference until you add one.
The commercial ones plus anything OpenAI-compatible: LM Studio, vLLM, TGI, Ollama and similar can be declared directly in .env without editing YAML. Local endpoints with no authentication are supported by leaving the key blank.
Both. Requests go through the firewall and the analysers; responses go through sanitisation, which covers entropy anomalies, steganography such as bidi overrides and zero-width characters, and prompt leaks. That is a difference from aidlp, which is egress only.
The proxy stack costs single-digit milliseconds at p50 on an idle process. Against an LLM call measured in seconds, that is noise. See Scaling and state for the measured numbers and their conditions.
Rate limits and circuit breakers are per process, in memory, by design. With four workers, a limit of 100 admits up to 400. See Scaling and state.
Presidio is not installed by default, so detection falls back to regex: email, phone, SSN, credit card, IBAN, IP and API keys, and nothing else. Install presidio-analyzer and presidio-anonymizer for the NLP entity types.
The measured benign false-positive rate is 6%, and the known tripping case is meta-discussion of attacks: asking the model to explain how prompt injection works looks like prompt injection. That is deliberate. Full results and gaps are in docs/OWASP_LLM_COVERAGE.md.
Yes, security.firewall.enabled: false, and that is what it is for in development. Make sure it does not travel to production in a copied config, and read Before production for the related concern: an admin can disable both the firewall and authentication in one API call.
Three of the ten, thoroughly and with published numbers: prompt injection, sensitive information disclosure and system prompt leakage. The other seven are out of scope for a proxy because they live at build time, training time, in the caller or in the model, and the project documents them as not applicable rather than claiming them.
In /app/data: registered endpoints, budgets and the audit ledger, in SQLite by default and swappable for Postgres. Mount it as a volume or you lose the audit ledger on every restart.
HMAC-SHA256 response signing proves the response was not modified after leaving the proxy. Verify it client-side if that property matters to you; the signature is only useful to someone who checks it.
API keys, OIDC and JWT with Google, Microsoft and Apple, mTLS, and Tailscale zero-trust, with four RBAC roles: admin, operator, user and viewer. Note what an admin can do, in Before production.
Eight CI jobs on every push: ruff, mypy, pip-audit, a supply-chain scan for .pth malware and blocked packages, a syntax check, the test suite behind a 65% coverage gate, mathematical invariants, and a Docker image size check. Raising coverage on the untested security paths is tracked as #107.
LLMProxy · README · Documentation · Report a vulnerability privately
Operating it
Reference
Project