Skip to content

Releases: sochaty/llm-governance-engine

Release list

LLM Governance Engine v1.1.0

LLM Governance Engine v1.1.0 Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 10 Aug 10:01
v1.1.0
69ea763
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Quick start

tar -xzf llm-governance-engine-v1.1.0.tar.gz
cp .env.example .env          # fill in your API keys
bash start.sh
Service URL
Dashboard http://localhost:4200
API docs http://localhost:8000/docs

Docker images are published to GHCR:

  • ghcr.io/sochaty/llm-governance-engine/backend:v1.1.0
  • ghcr.io/sochaty/llm-governance-engine/frontend:v1.1.0

LLM Governance Engine v1.1.0

Faithfulness scoring for RAG — know when a response isn't grounded in the context you gave it, not just whether it leaked PII.

v1.0 stopped bad data going in. v1.1 starts checking whether what comes out is actually true to the source material you supplied.

What's in this release

🧠 RAGAS Faithfulness & Context Utilization Scoring

Paste retrieved context alongside your prompt — the same passages you'd hand a RAG pipeline — and every response is scored against it using RAGAS's Faithfulness and ContextUtilization metrics. Faithfulness measures whether the model's claims are actually supported by the context; context utilization measures whether the retrieved passages were relevant to the question at all.

Fully opt-in: skip the context field and nothing changes — no extra latency, no extra cost, same behavior as v1.0.

# What the benchmark endpoint now accepts (POST, was GET)
POST /api/benchmark/stream
{
  "prompt": "How many days do I have to request a refund?",
  "provider": "cloud",
  "context": "Returns are accepted within 30 days of purchase..."
}

🚨 Post-Response Governance: faithfulness_score_below

A new policy condition joins pii_detected, safety_score_below, cost_exceeds, and model_is — but with a twist: faithfulness can only be known after the model responds, so this is the first governance rule that evaluates post-response instead of pre-flight.

- id: low-faithfulness-warn
  name: "Low Faithfulness / Possible Hallucination"
  condition: faithfulness_score_below
  threshold: 0.6
  action: warn
  severity: medium

Because the response has already started streaming to the client by the time faithfulness is known, a block action on this condition is automatically recorded as alert instead — it can't stop delivery, but it's still logged, audited, and can still fire a webhook.

🗄️ Audit Vault: Faithfulness Columns

The History table and PDF export now show Faithfulness and Context Utilization alongside the existing PII/safety columns for every scored run — — when no context was supplied, a real percentage when it was.

📡 /benchmark/stream: GET → POST

The streaming endpoint moved from query-string params to a JSON body, so the (potentially long) context field isn't limited by URL length. If you're calling it directly rather than through the UI, update your client accordingly.

Known limitation

The original v1.0 roadmap described local Ollama as the faithfulness judge ("a free judge"). This release uses OpenAI (gpt-4o-mini) as the judge model instead — ragas's modern metric collection API needed a OPENAI_API_KEY either way for reliable structured-output scoring at this stage, so a local-judge option is deferred rather than shipped half-working. Track it for a follow-up release if you want scoring with zero marginal cost.

Quick start

# Download and extract the release bundle
tar -xzf llm-governance-engine-v1.1.0.tar.gz

# Configure your environment
cp .env.example .env
# OPENAI_API_KEY is now required even if you're only benchmarking
# other providers — it's used as the RAGAS judge model for faithfulness scoring.

# Start everything
bash start.sh
Service URL
Dashboard http://localhost:4200
API docs (Swagger) http://localhost:8000/docs

Upgrading from v1.0.0 on an existing Postgres volume? The new policy_violations.faithfulness_score column isn't added automatically (no Alembic migrations yet — schema is created once via create_all). Run this once against your existing database:

ALTER TABLE policy_violations ADD COLUMN IF NOT EXISTS faithfulness_score FLOAT;

Stack

Layer Technology
Backend FastAPI · SQLAlchemy async · PostgreSQL · Presidio · RAGAS 0.4
Frontend Angular 21 · Chart.js · Server-Sent Events
Inference OpenAI · Groq · Google Gemini · Anthropic · Ollama
Infrastructure Docker Compose · GHCR · GitHub Actions CI · Playwright e2e
Tests pytest · 92% backend coverage · Playwright e2e

What's next

  • v1.2 — FinOps dashboard: daily cost trends, Z-score anomaly detection, budget circuit breakers
  • v2.0 — Multi-tenant workspaces with JWT + RBAC

Reproduce this release

git clone https://github.com/sochaty/llm-governance-engine
git checkout governance-post-2
docker compose up

Feedback and PRs welcome — open an issue or start a Discussion.

Assets 3
Loading

LLM Governance Engine v1.0.0

LLM Governance Engine v1.0.0

Choose a tag to compare

View all tags
@sochaty sochaty released this 19 Jun 01:33
v1.0.0
5cdee87
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Quick start

tar -xzf llm-governance-engine-v1.0.0.tar.gz
cp .env.example .env          # fill in your API keys
bash start.sh
Service URL
Dashboard http://localhost:4200
API docs http://localhost:8000/docs

Docker images are published to GHCR:

  • ghcr.io/sochaty/llm-governance-engine/backend:v1.0.0
  • ghcr.io/sochaty/llm-governance-engine/frontend:v1.0.0

LLM Governance Engine v1.0.0

The first open-source PII firewall and policy engine for LLMs — self-hosted, one command.

Most LLM tools tell you what happened after the fact. This one stops the request before it reaches the model.

What's in this release

🔒 PII Firewall

Prompts are scanned by Microsoft Presidio before they leave your infrastructure. Names, emails, credit card numbers, SSNs, phone numbers — detected and blocked (or audited) at configurable confidence thresholds. Your data never reaches OpenAI unless you explicitly allow it.

📋 YAML Policy Engine

Write governance rules as code, not dashboards. Drop a YAML file, hit /api/v1/policies/reload, rules are live — no restart required.

rules:
  - id: pii-cloud-block
    condition: pii_detected
    threshold: 0.7
    models: [cloud, gpt-4o]
    action: block
    severity: critical

  - id: budget-circuit-breaker
    condition: cost_exceeds
    threshold: 0.05
    action: warn
    severity: medium

Four conditions: pii_detected, safety_score_below, cost_exceeds, model_is.
Three actions: block (returns 403), warn (audited, passes through), alert (fires webhook).
Starter templates included: policies/default.yaml, policies/gdpr.yaml, policies/hipaa.yaml.

⚡ Cloud vs Local Benchmark

Side-by-side streaming comparison — GPT-4o vs your local Llama model (via Ollama). Latency, cost per request, and a 10-dimension governance radar chart on every run. When GPT-4o charges $0.03 for a task Llama handles at $0.00, that number is in your audit log.

🗄️ Audit Vault

Every inference — prompt preview, provider, PII flag, safety score, latency, estimated cost — stored in PostgreSQL. Filterable, paginated, exportable to PDF. Answering a compliance audit is a query, not a fire drill.

🔀 Multi-Provider Routing

Route to OpenAI, Groq, Google Gemini, Anthropic, or any Ollama model from a single UI. API keys managed through the Settings UI — encrypted at rest with Fernet, no restarts needed when you rotate keys.

🔔 Webhook Alerts

Policy violations fire CloudEvents-compatible HTTP POST payloads to Slack, Microsoft Teams, or PagerDuty. Configurable per rule. 3-attempt exponential backoff.

Quick start

# Download and extract the release bundle
tar -xzf llm-governance-engine-v1.0.0.tar.gz

# Configure your environment
cp .env.example .env
# Add OPENAI_API_KEY (or any cloud provider key)
# OLLAMA_BASE_URL defaults to http://localhost:11434/v1

# Start everything
bash start.sh
Service URL
Dashboard http://localhost:4200
API docs (Swagger) http://localhost:8000/docs

No build step required — Docker images are pulled directly from GHCR.

Pull a local model (optional):

curl -X POST http://localhost:11434/api/pull -d '{"name":"llama3.2:latest"}'

Stack

Layer Technology
Backend FastAPI · SQLAlchemy async · PostgreSQL · Presidio
Frontend Angular 21 · Chart.js · Server-Sent Events
Inference OpenAI · Groq · Google Gemini · Anthropic · Ollama
Infrastructure Docker Compose · GHCR · GitHub Actions CI
Tests pytest · 91% backend coverage · Playwright e2e

What's next

  • v1.1 — RAGAS hallucination scoring on every request (local Ollama as free judge) Shipped — see RELEASE_NOTES_v1.1.0.md. Opt-in via a context field rather than every request, and judged by OpenAI rather than local Ollama — see that file's "Known limitation" section for why.
  • v1.2 — FinOps dashboard: daily cost trends, Z-score anomaly detection, budget circuit breakers
  • v2.0 — Multi-tenant workspaces with JWT + RBAC

Reproduce this release

git clone https://github.com/sochaty/llm-governance-engine
git checkout governance-post-1
docker compose up

Feedback and PRs welcome — open an issue or start a Discussion.

Loading