Skip to content

LLM Governance Engine v1.1.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 10 Aug 10:01
v1.1.0
69ea763
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Quick start

tar -xzf llm-governance-engine-v1.1.0.tar.gz
cp .env.example .env          # fill in your API keys
bash start.sh
Service URL
Dashboard http://localhost:4200
API docs http://localhost:8000/docs

Docker images are published to GHCR:

  • ghcr.io/sochaty/llm-governance-engine/backend:v1.1.0
  • ghcr.io/sochaty/llm-governance-engine/frontend:v1.1.0

LLM Governance Engine v1.1.0

Faithfulness scoring for RAG — know when a response isn't grounded in the context you gave it, not just whether it leaked PII.

v1.0 stopped bad data going in. v1.1 starts checking whether what comes out is actually true to the source material you supplied.

What's in this release

🧠 RAGAS Faithfulness & Context Utilization Scoring

Paste retrieved context alongside your prompt — the same passages you'd hand a RAG pipeline — and every response is scored against it using RAGAS's Faithfulness and ContextUtilization metrics. Faithfulness measures whether the model's claims are actually supported by the context; context utilization measures whether the retrieved passages were relevant to the question at all.

Fully opt-in: skip the context field and nothing changes — no extra latency, no extra cost, same behavior as v1.0.

# What the benchmark endpoint now accepts (POST, was GET)
POST /api/benchmark/stream
{
  "prompt": "How many days do I have to request a refund?",
  "provider": "cloud",
  "context": "Returns are accepted within 30 days of purchase..."
}

🚨 Post-Response Governance: faithfulness_score_below

A new policy condition joins pii_detected, safety_score_below, cost_exceeds, and model_is — but with a twist: faithfulness can only be known after the model responds, so this is the first governance rule that evaluates post-response instead of pre-flight.

- id: low-faithfulness-warn
  name: "Low Faithfulness / Possible Hallucination"
  condition: faithfulness_score_below
  threshold: 0.6
  action: warn
  severity: medium

Because the response has already started streaming to the client by the time faithfulness is known, a block action on this condition is automatically recorded as alert instead — it can't stop delivery, but it's still logged, audited, and can still fire a webhook.

🗄️ Audit Vault: Faithfulness Columns

The History table and PDF export now show Faithfulness and Context Utilization alongside the existing PII/safety columns for every scored run — — when no context was supplied, a real percentage when it was.

📡 /benchmark/stream: GET → POST

The streaming endpoint moved from query-string params to a JSON body, so the (potentially long) context field isn't limited by URL length. If you're calling it directly rather than through the UI, update your client accordingly.

Known limitation

The original v1.0 roadmap described local Ollama as the faithfulness judge ("a free judge"). This release uses OpenAI (gpt-4o-mini) as the judge model instead — ragas's modern metric collection API needed a OPENAI_API_KEY either way for reliable structured-output scoring at this stage, so a local-judge option is deferred rather than shipped half-working. Track it for a follow-up release if you want scoring with zero marginal cost.

Quick start

# Download and extract the release bundle
tar -xzf llm-governance-engine-v1.1.0.tar.gz

# Configure your environment
cp .env.example .env
# OPENAI_API_KEY is now required even if you're only benchmarking
# other providers — it's used as the RAGAS judge model for faithfulness scoring.

# Start everything
bash start.sh
Service URL
Dashboard http://localhost:4200
API docs (Swagger) http://localhost:8000/docs

Upgrading from v1.0.0 on an existing Postgres volume? The new policy_violations.faithfulness_score column isn't added automatically (no Alembic migrations yet — schema is created once via create_all). Run this once against your existing database:

ALTER TABLE policy_violations ADD COLUMN IF NOT EXISTS faithfulness_score FLOAT;

Stack

Layer Technology
Backend FastAPI · SQLAlchemy async · PostgreSQL · Presidio · RAGAS 0.4
Frontend Angular 21 · Chart.js · Server-Sent Events
Inference OpenAI · Groq · Google Gemini · Anthropic · Ollama
Infrastructure Docker Compose · GHCR · GitHub Actions CI · Playwright e2e
Tests pytest · 92% backend coverage · Playwright e2e

What's next

  • v1.2 — FinOps dashboard: daily cost trends, Z-score anomaly detection, budget circuit breakers
  • v2.0 — Multi-tenant workspaces with JWT + RBAC

Reproduce this release

git clone https://github.com/sochaty/llm-governance-engine
git checkout governance-post-2
docker compose up

Feedback and PRs welcome — open an issue or start a Discussion.

Assets 3
Loading