TrajectoryCal

Trajectory-level confidence calibration for AI agents — predicts when an agent run is likely to fail using process signals from the full execution trajectory (HTC + GAC inspired).

Educational, resume-grade implementation: hexagonal architecture, zero paid services by default, phased build prompts in docs/PROMPTS.md.

Demo LLM: mock (no API keys). Optional Groq or Google AI Studio — see docs/TECH_STACK.md.

What it does

1. Ingests standardized trajectory events (LangGraph demo + REST API).
2. Extracts HTC-style process features (tc_feat_*).
3. Calibrates with L1 logistic regression + optional isotonic calibration.
4. Transfers across domains via GAC (frozen robust standardization, no OOD refit).
5. Acts at runtime: continue, request_data, run_verifier, escalate_human, stop.
6. Reports ECE/Brier/AUROC, reliability diagrams, model card, optional Grafana/Jaeger.

Architecture

flowchart TB
    subgraph agents [Agent runtimes]
        LG[LangGraph demo]
    end

    subgraph ingest [Ingestion]
        API[FastAPI /v1/events]
        Adapter[LangGraph adapter]
    end

    subgraph core [TrajectoryCal core]
        Store[(SQLite store)]
        FE[Feature extractor]
        GAC[GAC standardizer]
        Cal[L1 calibrator]
        Pol[Policy engine]
    end

    subgraph ops [Ops optional]
        Prom[Prometheus]
        Graf[Grafana]
        Jaeger[Jaeger OTLP]
    end

    LG --> Adapter
    Adapter --> API
    API --> Store
    Store --> FE
    FE --> GAC
    GAC --> Cal
    Cal --> Pol
    API --> Prom
    API -.-> Jaeger
    Prom --> Graf

Loading

Quickstart (~20 minutes)

git clone <your-fork-url>
cd TrajectoryCal

uv venv .venv && source .venv/bin/activate
uv pip install -e ".[dev,demo]"

cp .env.example .env
mkdir -p data/synthetic artifacts reports/figures

# Train + evaluate (writes reports/metrics.json)
python scripts/generate_synthetic.py --count 100
python scripts/train_calibrator.py
python scripts/evaluate.py
python scripts/evaluate_ood.py
python scripts/generate_model_card.py

# Quality gates
ruff check src tests && mypy src/trajectorycal && pytest

# API
uvicorn trajectorycal.api.app:app --reload
# → http://127.0.0.1:8000/docs

# LangGraph demo (mock LLM, streams events to API)
python demos/langgraph_agent/run_demo.py --mock

# Interview showcase UI (one-click trajectory + score demo)
python demos/showcase/run.py
# → http://127.0.0.1:8765

Docker (optional): docs/SETUP_GUIDE.md

docker compose -f docker/compose.yml up --build -d trajectorycal-api
curl http://localhost:8000/health

Demo recording (terminal)

Record a short terminal cast for your portfolio (no GIF committed to the repo):

# Install: brew install asciinema
asciinema rec trajectorycal-demo.cast
# Inside the recording:
uv run python demos/langgraph_agent/run_demo.py --mock
curl http://127.0.0.1:8000/v1/trajectories/<trace_id_from_log>
# Ctrl+D to stop, then:
asciinema play trajectorycal-demo.cast
# Upload: asciinema upload trajectorycal-demo.cast

Evaluation metrics

From reports/metrics.json after python scripts/evaluate.py (synthetic validation split, n_val=20). Reproduce locally — numbers depend on your split seed.

Scorer	ECE ↓	Brier ↓	AUROC ↑
GAC + logistic calibrator	0.00	0.00	1.00
Last-step confidence baseline	0.53	0.38	0.67
Step-count heuristic	0.40	0.25	0.50

ECE improvement vs last-step baseline: 100% relative on this split (0.53 → 0.00).

OOD (domain synthetic_c, train A+B, no refit): ECE 0.00, Brier 0.00, AUROC 1.00 — see reports/ood_metrics.json from scripts/evaluate_ood.py.

Example templates committed at reports/templates/. Full model card: run scripts/generate_model_card.py → reports/model_card.md (generated locally).

Synthetic data yields strong separation; treat metrics as pipeline validation, not production guarantees. Document limitations in the generated model card.

Resume bullets (fill with your fork URL)

Copy after you re-run scripts/evaluate.py on your machine:

• Built TrajectoryCal, an open-source trajectory-level calibrator for LLM agents (HTC/GAC-inspired), with 12+ process features, GAC transfer, and FastAPI serving — repo.
• Reduced calibration ECE from 0.53 → 0.00 vs a last-step confidence baseline on a held-out synthetic validation split (AUROC 1.00); OOD domain eval without refit.
• Shipped LangGraph → event adapter, policy engine (5 actions), Docker Compose, and Grafana dashboards for ingest/score/policy metrics.

Documentation

Doc	Purpose
docs/SETUP_GUIDE.md	Prerequisites and verification checklist
docs/ARCHITECTURE.md	Modules and data flows
docs/PROMPTS.md	Phased implementation prompts (0–9)
docs/PRD.md	Requirements and scope
CONTRIBUTING.md	Phase-based PR workflow
docs/adr/001-sqlite-default.md	Why SQLite is default

Project status

Phases 0–9 complete (scaffold through docs/Docker). Optional extensions: docs/PROMPTS.md.

License

MIT — see LICENSE.

Citation

@article{zhang2026agentic,
  title={Agentic Confidence Calibration},
  author={Zhang, Jiaxin and Xiong, Caiming and Wu, Chien-Sheng},
  journal={arXiv preprint arXiv:2601.15778},
  year={2026}
}