Skip to content

JevGraph v0.2.0 — Three-model relation benchmark

Choose a tag to compare

@chenmingtang830 chenmingtang830 released this 20 Sep 10:05
· 24 commits to main since this release
85cebe0

What changed

  • Added strict Vercel AI Gateway benchmark adapters for GPT-5.6 Luna and DeepSeek V4.1 Flash.
  • Froze a same-sample, same-ontology FewRel comparison against Jev.
  • Preserved model failures as evidence: DeepSeek required an 8 → 4 → 2 adaptive batch continuation after incomplete completions.
  • Clarified the architecture boundary: the model only selects a relation for an existing candidate pair; it does not discover entities or pairs.
  • Published aggregate evidence and a complete budget audit without committing FewRel source text.

Frozen pilot

Model Raw accuracy Successful / attempted requests p50 latency Known cost
Jev 85.0% 20 / 21 444 ms $0 reported
GPT-5.6 Luna 80.0% 20 / 20 5.15 s $0.0144112
DeepSeek V4.1 Flash 83.75% 51 / 53 4.4 s $0.10255982

DeepSeek latency is a mixed-batch operational result with a 22.3 s p95. Jev alone returned the full probability/confidence contract, so its gated risk/coverage result is reported separately from raw accuracy.

Verification

  • Python 3.11 and 3.13 CI passed
  • 14 tests passed
  • Ruff passed
  • wheel and source distribution attached

Known billed chat-experiment cost was $0.150184 across exploratory and final calls. Including five unknown-cost failed requests across all experiments, a conservative ceiling remains below $0.315, inside the approved $5 cap.