PassProof

Certification readiness as a calibrated probability: will this person pass, and what gets them there?

Learning dashboards count hours and quiz percentages. None of that answers the question a manager actually has: is this learner going to pass the exam? PassProof answers it with a calibrated P(pass), study material that cites its sources, a plan that fits the learner's real week, and an honest "not enough signal" when the data is too thin to judge.

target: AZ-204   practice avg 0.67   hours 18/35   skill_coverage 0.2
P(pass): 0.38   tier: NOT_READY   weakest: az204-storage   +16h to the gate

Built for the Microsoft Agents League (Reasoning Agents track) on Microsoft Foundry.

Demo

Watch the demo: add YouTube/Vimeo link here before submitting

See docs/SUBMISSION.md for the demo-video beat sheet and the full submission checklist.

What it is

Five agents wired into a LangGraph StateGraph with a conditional pass/fail loop-back:

Learning Path Curator → Study Plan Generator → Engagement → Assessment → Manager Insights.

The piece we care about most is the readiness model: a logistic P(pass) fit on synthetic exam outcomes and checked with Brier + AUC. It returns READY, ALMOST, NOT_READY, or INSUFFICIENT. That last tier means the system abstains instead of guessing.

Runs anywhere, by design

The track is judged 20% on Reliability & Safety, and the most reliable demo is one with nothing to break. So:

• The UI is vanilla HTML/CSS/JS with one vendored Alpine.js file. No npm, no build step.
• Fonts are local .woff2 files. No CDN, no network calls.
• The whole pipeline runs offline with no Azure credentials. Live Foundry paths are an explicit opt-in (PASSPROOF_LIVE=1 with .env configured) and fall back to cached providers on any failure. They never hard-fail.
• The full pytest suite runs with no network and no credentials. Live Azure paths are injected and stubbed in tests.

Quickstart (no credentials needed)

./scripts/quickstart.sh

Creates a venv, runs pip install -e ., then runs the offline demo end to end: fits the calibration model on the synthetic dataset, prints a cited readiness verdict for L-1001 and for a strong learner, prints a manager summary line, and finishes with the test suite.

Web UI: readiness on one screen

.venv/bin/python -m passproof.web        # → http://127.0.0.1:8000

One console, three views (Learner · Manager · Audit/Trace):

• Learner — the calibrated P(pass) front and center, the readiness tier, the weakest skill, hours to the gate, the cited study path, and a practice-question flow. The quiz is graded for practice; the verdict stays calibrated. After a failing assessment the loop-back re-plans more hours and shows the projected P(pass), labelled as a projection.
• Manager — team readiness roll-up, at-risk learners, capacity-constrained teams, breakdown by certification.
• Audit/Trace — the live pipeline stepper and the persisted decision trail: inputs, outputs, citations, model, latency per step.

Pick L-1046 · thin signal in the dropdown to watch the abstention end to end: the verdict is INSUFFICIENT, gamification earns nothing, and the what-if projection stays locked until there is enough evidence to calibrate.

Decision intelligence

Built on the same deterministic, offline model — no new math, just surfaced:

• Focus topics — an optional comma-separated input biases the curated path toward the topics a learner asks for (baseline-flow step 1).
• Fastest path to the gate — exact logistic counterfactuals (hours / skill / practice) ranked by resulting P(pass); arithmetic, not speculation.
• Cohort what-if — a manager slider projecting how many of a team cross the gate at +N hours per learner; abstained learners are excluded, as in the roll-up.
• Live readiness room — a roster-wide verdict stream (NDJSON) that shows each learner's tier as it computes, with abstention shown honestly, never rolled up.
• Agent-quality panel — the offline rubric harness (grounding, critic accept rate, abstention correctness, trace completeness) surfaced in the Audit view.

Three runtime modes

Mode	Command	What the LLM layer is
Offline (default)	PASSPROOF_OFFLINE=1 .venv/bin/python -m passproof.web	Deterministic templates over the cached synthetic corpus — nothing to break.
Replay	PASSPROOF_REPLAY=1 .venv/bin/python -m passproof.web	Real gpt-4o + live Foundry IQ output, recorded once and replayed deterministically (data/replay/live-session.json, recorded with scripts/record_live_session.py). No credentials needed.
Live	configure .env, then PASSPROOF_LIVE=1 .venv/bin/python -m passproof.web	gpt-4o via Foundry /openai/v1 + Azure AI Search agentic retrieval, live.

A fragile live demo and a mocked demo is a false choice: replay mode shows real model output with offline-mode reliability, and the header badge always states which mode you are looking at.

Configure live Azure via .env (AZURE_AI_FOUNDRY_ENDPOINT, MODEL_DEPLOYMENT=gpt-4o, FOUNDRY_IQ_*, CONTENT_SAFETY_*; the challenge brief calls these AZURE_AI_PROJECT_ENDPOINT/AZURE_AI_MODEL_DEPLOYMENT — PassProof reads AZURE_AI_FOUNDRY_ENDPOINT/MODEL_DEPLOYMENT, see .env.example). With those unset (or PASSPROOF_OFFLINE=1), every credential-free path still runs.

Built on Microsoft Foundry

Foundry capability	How PassProof uses it
Foundry IQ (required Microsoft IQ — live)	An Azure AI Search (Basic) Knowledge Base with agentic retrieval: query planning over the synthetic cert corpus, cited synthesis back. FoundryIQGroundingProvider maps those citations into the Curator's resources and the Assessment's questions. Falls back to a cached provider offline.
Fabric IQ	A semantic ontology (semantic/ontology.py): Role↔Cert↔Skill↔threshold↔hours. Drives hour allocation, skill coverage and gaps, the role ladder, and team roll-ups.
Work IQ	Synthetic work signals (meeting/focus hours, preferred learning slot) drive the Planner's capacity allocation and the Engagement agent's reminder timing.
Azure AI Content Safety	An RAI gate on generated Q&A. Flagged content is blocked before it reaches a learner.
Foundry Agent Service	The entry orchestration agent (foundry_agent.py) packages as a container for hosted deployment. Managed identity, no secrets in the image.
MCP	assess_readiness is exposed as an MCP tool (mcp_server.py); the Curator can also consume the MS Learn MCP server.

PassProof is also an MCP tool

Any MCP-capable agent (Claude, GitHub Copilot, Azure AI Foundry, a custom LangGraph agent) can call PassProof's readiness pipeline as a tool, no REST wiring needed:

# start the PassProof stdio MCP server
python -m passproof.mcp_server

The server exposes one tool, assess_readiness(learner_id, answers?). It runs the full multi-agent pipeline and returns tier, P(pass), confidence, weakest skill, and hours to the gate — the same verdict the web UI shows.

Example MCP client config (Claude Desktop or any stdio MCP host):

{
  "mcpServers": {
    "passproof": {
      "command": "python",
      "args": ["-m", "passproof.mcp_server"]
    }
  }
}

Example round-trip (see scripts/mcp_client_demo.py for the full demo):

# spawns the server, lists tools, calls assess_readiness for L-1002
PASSPROOF_OFFLINE=1 python scripts/mcp_client_demo.py

PassProof also consumes an external MCP server: the Microsoft Learn MCP (ms_learn_search via https://learn.microsoft.com/api/mcp), which the Curator uses to enrich curated content with cited Microsoft documentation when online.

The point: any agent or Copilot extension can ask "will this person pass?" over stdio and get a calibrated answer, without knowing anything about the pipeline behind it.

Verified live against Microsoft Foundry

PassProof was verified live end-to-end against real Azure on 2026-06-08 — a one-time smoke test against the author's subscription; the app and full test suite run offline by default:

• gpt-4o chat works via the Foundry OpenAI-compatible endpoint (<endpoint>/openai/v1, no api-version needed — v1 GA).
• Foundry IQ agentic retrieval works as a Basic-tier Azure AI Search Knowledge Base (/knowledgebases/<kb>/retrieve, api-version 2026-05-01-preview): gpt-4o query planning plus grounded citations over the synthetic cert corpus. CuratorAgent.curate(L-1001) returned 5/5 live-grounded resources from real Azure AI Search.
• One gotcha worth knowing: the KB's gpt-4o model authenticates via the Search service's system-assigned managed identity plus "Cognitive Services OpenAI User" RBAC. An API key gets rejected at retrieve time; managed identity is required.

To reproduce live: cp .env.example .env, fill in the real Azure values, then run the web UI in Live mode (PASSPROOF_LIVE=1 .venv/bin/python -m passproof.web) — or record the session once and serve it credential-free with Replay (see Three runtime modes above).

The calibration story

Readiness here is a calibrated probability, not a checkmark. A small logistic regression over three interpretable features (practice_score, hours_ratio, skill_coverage) is fit by seeded gradient descent on the synthetic exam_outcome labels, then checked with Brier score, AUC, and a reliability diagram (/api/calibration). We chose logistic on purpose: legible weights, deterministic fit, and calibration (not raw ranking) is what makes a P(pass) worth trusting. Below a confidence floor the verdict abstains (INSUFFICIENT), because a calibrated "not enough signal" beats a confident wrong answer.

Numbers on 39 labelled synthetic learners:

• In-sample fit: AUC ≈ 0.773, Brier ≈ 0.185 (vs 0.246 base-rate baseline)
• Held-out (leave-one-out CV): AUC ≈ 0.703, Brier ≈ 0.219 — every prediction made by a fit that never saw that learner, and it still beats the baseline

All data is synthetic

All data in PassProof is synthetic: fabricated ids (L-1001, EMP-001, TEAM-A), obviously fictional, no PII and no real customer data. Every synthetic knowledge document ends with > Synthetic document — fabricated for PassProof. Not real Microsoft content. Nothing in this repo is real Microsoft certification content or real learner data.

Repository layout

src/passproof/
  orchestrator.py    LangGraph StateGraph: Curator → Planner → Engagement → Assessment → Manager
  agents/            the five role-specialised agents + the question critic/verifier
  semantic/          Fabric-IQ-style ontology (Role↔Cert↔Skill↔threshold↔hours)
  readiness/         calibrated logistic P(pass) + verdict builder (abstention)
  intel/             Foundry IQ grounding (live, cited) + cached offline provider
  safety/            Azure AI Content Safety gate + input/output guardrails
  eval/              metrics + Brier/AUC calibration bench + rubric eval harness
  telemetry.py       Tracer + persisted AuditRecord trail + App Insights/OTel spans
  mcp_server.py      assess_readiness as an MCP tool (+ MS Learn MCP client)
  foundry_agent.py   entry agent for Foundry Agent Service
  web/               dual-persona console (stdlib server + vanilla UI)
  cli.py             creds-free offline demo entrypoint
data/synthetic/      learners, certs, roles, skills, work signals + the doc corpus
docs/                ARCHITECTURE.md · DESIGN.md · SUBMISSION.md · architecture-icons.png
tests/               one test file per module (offline; no network, no credentials)

Install & test

pip install -e .            # core; add ".[dev]" for tests, ".[foundry]" for live Foundry
python -m pytest -q

The suite runs with no network and no credentials. Live Azure paths are injected and stubbed in tests.

Team

• Eric Guimarães — Microsoft Learn: ericguimares-3344

License

MIT.