Repository navigation
v0.2.0
Iteration 2 — v0.2.0 "AI core"
Must-include
| # | Feature | สถานะ |
|---|---|---|
| 1 | RAG / KB ingestion (pgvector + BM25, 36-position playbook) |
✅ |
| 2 | Router + MoE specialist agents (Classifier → IP / Liability / Termination / General) | ✅ |
| 3 | Real LLM calls + Pydantic structured output (5 providers via providers.py) |
✅ |
| 4 | LLM-as-a-Judge grounding check (judge.v1.jinja, excerpt ≥ 4 words verbatim) |
✅ |
| 5 | Eval script on gold subset (scripts/run_eval.py --limit 3, ~18 min) |
✅ |
| 6 | eval_report.md checked in |
✅ |
Eval Results (3 contracts / 22 clauses)
| Metric | Score |
|---|---|
segmentation_f1 |
100% |
citation_validity |
100% |
classification_accuracy |
45.45% |
risk_accuracy |
50.00% |
classification/risk ต่ำเพราะ gold label เปลี่ยน ไม่ใช่ pipeline แย่ลง
(CUAD annotate 41 คำถาม ไม่ได้จำแนก clause type ตรงๆ)
Pipeline
Upload → Parse → Segment → Router (MoE)
→ RAG (pgvector + BM25) → Risk Score → Judge → Report
Agents: MetadataExtractor · ClauseClassifier · RiskScorerAgent · JudgeAgent · OverrideAgent
RAG sources: data/playbook/positions.yaml (36 positions) + CUAD v1 (12 contracts)
Embedding: gemini-embedding-001 768 dim
Pass Bar
✅ Pipeline รันบน sample inputs ได้
✅ Eval metrics รายงานได้ (ไม่ต้องถึง final threshold)
✅ 268 offline tests + 11 live LLM tests ผ่าน