Repository navigation
JevGraph v0.2.0 — Three-model relation benchmark
What changed
- Added strict Vercel AI Gateway benchmark adapters for GPT-5.6 Luna and DeepSeek V4.1 Flash.
- Froze a same-sample, same-ontology FewRel comparison against Jev.
- Preserved model failures as evidence: DeepSeek required an 8 → 4 → 2 adaptive batch continuation after incomplete completions.
- Clarified the architecture boundary: the model only selects a relation for an existing candidate pair; it does not discover entities or pairs.
- Published aggregate evidence and a complete budget audit without committing FewRel source text.
Frozen pilot
| Model | Raw accuracy | Successful / attempted requests | p50 latency | Known cost |
|---|---|---|---|---|
| Jev | 85.0% | 20 / 21 | 444 ms | $0 reported |
| GPT-5.6 Luna | 80.0% | 20 / 20 | 5.15 s | $0.0144112 |
| DeepSeek V4.1 Flash | 83.75% | 51 / 53 | 4.4 s | $0.10255982 |
DeepSeek latency is a mixed-batch operational result with a 22.3 s p95. Jev alone returned the full probability/confidence contract, so its gated risk/coverage result is reported separately from raw accuracy.
Verification
- Python 3.11 and 3.13 CI passed
- 14 tests passed
- Ruff passed
- wheel and source distribution attached
Known billed chat-experiment cost was $0.150184 across exploratory and final calls. Including five unknown-cost failed requests across all experiments, a conservative ceiling remains below $0.315, inside the approved $5 cap.