JevGraph v0.5.1 — Visual direct relation benchmark

Result-first benchmark brief

v0.5.1 publishes the controlled direct relation-selection benchmark as a Proofpress-style two-page brief and self-contained HTML report. The first page leads with aligned bar charts for p95 latency, cost per 100 planned decisions, and planned-case accuracy; the second records protocol, failures, release readiness, and limits.

Jev is 3.94× lower in p95 latency than GPT-5.6 Luna and 2.88× lower than DeepSeek V4.1 Flash in this run. Its illustrative input list-price equivalent is 2.99–3.56× lower than the chat providers receipt costs. Accuracy is lower, so this remains an operational tradeoff rather than a general model ranking.

No new provider calls were made. The release preserves the same controlled 160-case evidence, including the Jev HTTP 503 and DeepSeek prediction_outside_criteria failure, with opaque provider-visible case IDs and no retries.

JevGraph v0.5.0 — Document-to-graph ingestion

Configured document to graph

v0.5.0 adds local PDF, DOCX, and PPTX ingestion through a commit-pinned DocJev + LiteParse adapter. It preserves source and canonical hashes, parser identity, canonical page spans, and page-local evidence offsets. Hosted DocJev classification and splitting are not invoked.

It also adds a digest-pinned configured E2E smoke benchmark over a fixed ontology, entity catalog, canonical text, and four repository-owned gold edges. The local regression run produces 15 candidates, proposes all four gold edges with no false positives, and reports candidate recall, edge precision/recall/F1, span integrity, latency, provider/list-price cost, and derived cost per candidate/proposed/correct edge.

This smoke result validates pipeline integrity only. It is not external evidence of general knowledge-graph quality, entity discovery, or model superiority. The existing FewRel comparison remains the relation-selection benchmark.

No new live provider calls were made for this release.

JevGraph v0.4.2

Derived cost per decision

v0.4.2 reports comparable cost per planned decision in direct-benchmark summaries and evidence. Chat values are derived from provider-reported receipts; Jev remains separately labeled as an illustrative input-token list-price equivalent, not an invoice.

It also documents why Jev’s native typed Choice output is larger: it returns the selected relation, a full probability distribution across every allowed relation, and confidence.

No additional model calls were made.

JevGraph v0.4.1

Controlled chat reasoning benchmark

v0.4.1 adds explicit --reasoning-effort to the direct chat benchmark and makes reasoning.effort=none the controlled primary comparison for GPT-5.6 Luna and DeepSeek V4.1 Flash.

The same 160 cases show that disabling provider-default reasoning materially reduces output-token use, tail latency, and cost. Provider-default v0.4.0 receipts remain in the evidence bundle for audit, but are no longer the primary comparison.

Artifacts: wheel, source distribution, sanitized evidence ZIP, and ZIP checksum.

JevGraph v0.4.0

Direct relation-selection benchmark

v0.4 makes the direct, one-case FewRel relation-selection protocol the primary benchmark. It uses opaque provider case IDs, 16 relations-only choices, 160 shared cases, batch size one, temperature zero, no retries/fallbacks, and a 4,096-token chat cap.

The primary result and full cost/failure audit are in docs/evidence/direct-2026-09-20.json. The evidence ZIP includes sanitized raw receipts, including all failures, plus SHA-256 manifest.

• v0.2 is superseded for capability claims because its provider-visible IDs exposed gold-bearing relation prefixes.
• v0.3 episodic validation is retained as an additional stress test.

Artifacts: wheel, source distribution, sanitized evidence ZIP, and ZIP checksum.

JevGraph v0.3.0 — Public FewRel episodic validation

What changed

• Adds deterministic public FewRel val_wiki N-way K-shot episodes for Jev, GPT-5.6 Luna, and DeepSeek V4.1 Flash.
• Hides property IDs and source indexes behind opaque per-episode labels and query IDs.
• Independently shuffles support and query order to remove label-order leakage.
• Preserves failed receipts and supports non-overlapping continuation without retries.
• Separates provider-reported Jev cost from an illustrative list-price equivalent.

Frozen 5-way 1-shot result

100 episodes, 500 planned queries, seed 17, one batched transductive request per episode.

Model	Planned accuracy	Coverage	Successful episodes	p50 latency	Known cost
Jev	85.6%	100%	100/100	368 ms	$0 reported; $0.009885 list equivalent
GPT-5.6 Luna	98.0%	100%	100/100	6,142 ms	$0.107803
DeepSeek V4.1 Flash	85.0%	86%	86/100	11,821 ms	$0.49140142 plus 2 unknown-cost timeouts

DeepSeek reached 98.84% on completed queries but had 12 output-ceiling truncations and two timeouts. No failed episode was retried. Its known-cost failures consumed $0.2393376, making reliability and orchestration cost part of the benchmark rather than discarded noise.

Scope

This is public validation for relation selection on supplied entity pairs. It is not the hidden official FewRel leaderboard, entity discovery, candidate-pair recall, negative-edge precision, graph truth, or human approval.

Evidence

The evidence ZIP contains aggregate records plus sanitized raw Track 1 and Track 2 run artifacts, including failed receipts and a SHA-256 manifest. It excludes FewRel sentences, provider prompts, source documents, credentials, and authorization headers.

Across all v0.2 and v0.3 calls, known Gateway cost was $0.77717750; the conservative upper bound including unknown-cost requests was $0.98180870, below the approved $5 cap.

JevGraph v0.2.0 — Three-model relation benchmark

What changed

• Added strict Vercel AI Gateway benchmark adapters for GPT-5.6 Luna and DeepSeek V4.1 Flash.
• Froze a same-sample, same-ontology FewRel comparison against Jev.
• Preserved model failures as evidence: DeepSeek required an 8 → 4 → 2 adaptive batch continuation after incomplete completions.
• Clarified the architecture boundary: the model only selects a relation for an existing candidate pair; it does not discover entities or pairs.
• Published aggregate evidence and a complete budget audit without committing FewRel source text.

Frozen pilot

Model	Raw accuracy	Successful / attempted requests	p50 latency	Known cost
Jev	85.0%	20 / 21	444 ms	$0 reported
GPT-5.6 Luna	80.0%	20 / 20	5.15 s	$0.0144112
DeepSeek V4.1 Flash	83.75%	51 / 53	4.4 s	$0.10255982

DeepSeek latency is a mixed-batch operational result with a 22.3 s p95. Jev alone returned the full probability/confidence contract, so its gated risk/coverage result is reported separately from raw accuracy.

Verification

• Python 3.11 and 3.13 CI passed
• 14 tests passed
• Ruff passed
• wheel and source distribution attached

Known billed chat-experiment cost was $0.150184 across exploratory and final calls. Including five unknown-cost failed requests across all experiments, a conservative ceiling remains below $0.315, inside the approved $5 cap.

JevGraph v0.1.1 — Validated live batch default

v0.1.1 is the recommended hackathon release.

This patch changes the default Jev and benchmark batch size from 32 to 8 and updates the examples and architecture notes. The v0.1 live evidence observed an HTTP 503 for one 32-question Gateway request, while eight-question batches supported the published pilot. This is a conservative operational default, not a provider throughput guarantee.

Everything from v0.1.0 remains included: bounded candidate generation, evidence spans, typed Jev decisions through Vercel AI Gateway, budget and no-retry gates, FewRel tooling, sharded benchmark merging, aggregate evidence, and CSV/Neo4j exports.

Validation: Python 3.11 and 3.13 CI passed; 12 tests, Ruff, wheel/sdist build, and clean-wheel install smoke passed.

JevGraph v0.1.0 — Hackathon release

JevGraph v0.1.0 turns relation extraction into a bounded, evidence-backed decision pipeline.

Highlights:

• YAML ontology with domain/range constraints
• exact mention and evidence spans
• local O(nk) candidate blocking instead of naive all-pairs evaluation
• Vercel AI Gateway adapter for typed Jev Choice decisions
• hard budget, call, size, timeout, no-retry, and no-fallback gates
• proposed/review/rejected edge states and sanitized request receipts
• JSON, CSV, and Neo4j exports
• pinned FewRel downloader, benchmark planning, sharding, and merge support

Pilot evidence:

• FewRel frozen sample: 136/160 correct (85.0%) across 16 relations
• pre-existing p>=0.85 and confidence>=0.80 gate: 95/160 selected, 95/95 correct on this positive-only sample
• 20 successful requests and one retained HTTP 503 receipt; p50/p95 request latency 444/625 ms
• successful Gateway responses reported $0.00 provider cost; one failed request has unknown billing status
• synthetic end-to-end demo proposed the four explicit relations and rejected unsupported directions

Scope: FewRel supplies entity pairs and positive labels. These results do not measure entity discovery, candidate recall, negative-edge precision, calibration, or general KG quality. The Gateway did not report a resolved model revision. See docs/RESULTS.md and the aggregate evidence JSON for the full boundaries.

Validation: CI passed on Python 3.11 and 3.13; 12 tests, Ruff, wheel/sdist build, and clean-wheel install smoke passed.