Repository navigation
JevGraph v0.3.0 — Public FewRel episodic validation
What changed
- Adds deterministic public FewRel val_wiki N-way K-shot episodes for Jev, GPT-5.6 Luna, and DeepSeek V4.1 Flash.
- Hides property IDs and source indexes behind opaque per-episode labels and query IDs.
- Independently shuffles support and query order to remove label-order leakage.
- Preserves failed receipts and supports non-overlapping continuation without retries.
- Separates provider-reported Jev cost from an illustrative list-price equivalent.
Frozen 5-way 1-shot result
100 episodes, 500 planned queries, seed 17, one batched transductive request per episode.
| Model | Planned accuracy | Coverage | Successful episodes | p50 latency | Known cost |
|---|---|---|---|---|---|
| Jev | 85.6% | 100% | 100/100 | 368 ms | $0 reported; $0.009885 list equivalent |
| GPT-5.6 Luna | 98.0% | 100% | 100/100 | 6,142 ms | $0.107803 |
| DeepSeek V4.1 Flash | 85.0% | 86% | 86/100 | 11,821 ms | $0.49140142 plus 2 unknown-cost timeouts |
DeepSeek reached 98.84% on completed queries but had 12 output-ceiling truncations and two timeouts. No failed episode was retried. Its known-cost failures consumed $0.2393376, making reliability and orchestration cost part of the benchmark rather than discarded noise.
Scope
This is public validation for relation selection on supplied entity pairs. It is not the hidden official FewRel leaderboard, entity discovery, candidate-pair recall, negative-edge precision, graph truth, or human approval.
Evidence
The evidence ZIP contains aggregate records plus sanitized raw Track 1 and Track 2 run artifacts, including failed receipts and a SHA-256 manifest. It excludes FewRel sentences, provider prompts, source documents, credentials, and authorization headers.
Across all v0.2 and v0.3 calls, known Gateway cost was $0.77717750; the conservative upper bound including unknown-cost requests was $0.98180870, below the approved $5 cap.