Skip to content

JevGraph v0.3.0 — Public FewRel episodic validation

Choose a tag to compare

@chenmingtang830 chenmingtang830 released this 20 Sep 11:10
7a6f7d0

What changed

  • Adds deterministic public FewRel val_wiki N-way K-shot episodes for Jev, GPT-5.6 Luna, and DeepSeek V4.1 Flash.
  • Hides property IDs and source indexes behind opaque per-episode labels and query IDs.
  • Independently shuffles support and query order to remove label-order leakage.
  • Preserves failed receipts and supports non-overlapping continuation without retries.
  • Separates provider-reported Jev cost from an illustrative list-price equivalent.

Frozen 5-way 1-shot result

100 episodes, 500 planned queries, seed 17, one batched transductive request per episode.

Model Planned accuracy Coverage Successful episodes p50 latency Known cost
Jev 85.6% 100% 100/100 368 ms $0 reported; $0.009885 list equivalent
GPT-5.6 Luna 98.0% 100% 100/100 6,142 ms $0.107803
DeepSeek V4.1 Flash 85.0% 86% 86/100 11,821 ms $0.49140142 plus 2 unknown-cost timeouts

DeepSeek reached 98.84% on completed queries but had 12 output-ceiling truncations and two timeouts. No failed episode was retried. Its known-cost failures consumed $0.2393376, making reliability and orchestration cost part of the benchmark rather than discarded noise.

Scope

This is public validation for relation selection on supplied entity pairs. It is not the hidden official FewRel leaderboard, entity discovery, candidate-pair recall, negative-edge precision, graph truth, or human approval.

Evidence

The evidence ZIP contains aggregate records plus sanitized raw Track 1 and Track 2 run artifacts, including failed receipts and a SHA-256 manifest. It excludes FewRel sentences, provider prompts, source documents, credentials, and authorization headers.

Across all v0.2 and v0.3 calls, known Gateway cost was $0.77717750; the conservative upper bound including unknown-cost requests was $0.98180870, below the approved $5 cap.