Skip to content

v0.4.4 — LRB v0.2.3 S3 publication-scale + cycle γ 4-bench infrastructure closure

Latest
Latest

Choose a tag to compare

View all tags
@Hashevolution Hashevolution released this 12 Jun 02:01
· 282 commits to main since this release
v0.4.4
b4486c0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v0.4.4 — LRB v0.2.3 S3 publication-scale + cycle γ 4-bench infrastructure closure

Date: 2026-06-12
Predecessor: v0.4.3 — 10.5281/zenodo.20625533 (deposited 2026-06-10)
Concept DOI: see Zenodo "Versions" tab of the parent record (chain-anchor across all versions)
Tag: v0.4.4
Commit: (this release tag)

What's new (11 PRs since v0.4.3)

LRB v0.2.3 — cross-scale reproducibility extension (6 PRs)

PR Type Summary
#823 feat S3 publication-scale generator (smoke / dev / publication presets; 100 / 300 / 1000 docs; deterministic-SHA programmatic vocabulary; 25 unit tests)
#824 feat S3 token-mode 4-point ladder measurement (V<N<J preserved across smoke / dev / publication)
#825 fix S3.1 contract title diversity (30 domains × 7 types = 210 unique pairs) + honest-framing self-correction (retracts pre-S3.1 over-tight ±0.05 verdict)
#826 docs preprint §4.6 Cross-scale + §5 pattern-vs-magnitude framing + §6.1 Limitations + §7.4 Future Work updates
#827 feat v0.2.3b cross-model LLM-grounded runner (operator-gated execution; pre-reg LOCKED)
#829 fix preprint typo corrections (leg 1 deltas + line 213 narrative) + abstract S3 integration (4 fixes)

Headline finding: R@1 V<N<J inequality preserved at every cell of a 4-point scale ladder spanning a 12.5× scale jump (S2 N=80 → S3 publication N=1000). JAMES − Naive gap remains above +0.10 at every cell.

Scenario N (init/events/queries) V R@1 N R@1 J R@1 J−N gap
S2 token (frozen v0.2.1) 200 / 564 / 80 0.225 0.5375 0.7125 +0.175
S3 smoke 100 / 282 / 100 0.510 0.730 0.930 +0.200
S3 dev 300 / 1206 / 300 0.530 0.737 0.913 +0.176
S3 publication 1000 / 5620 / 1000 0.502 0.721 0.845 +0.124

Pattern + gap are scale-robust (⭐⭐⭐). Absolute magnitudes are scenario-sensitive (⭐⭐; synthetic templated vocabulary has lower retrieval ambiguity than hand-curated S2 city-operations text). Cross-scenario claims are therefore limited to ordering and gap structure, not absolute magnitude.

Cycle γ 4-bench promise — infrastructure closure (3 PRs)

PR Type Summary
#819 chore LRB v0.2.1 claude S2 vanilla audit-trail closure (canonical R@1=0.6125 + reproducibility band ⭐ finding: claude API non-determinism produces ~10pp variance between two 1.5min-apart runs)
#820 feat D-alce research-tier NLI adapter (RoBERTa-large-MNLI primary + DeBERTa-v3-large-mnli-fever-anli-ling-wanli secondary; 19 unit tests; T5-XXL TRUE NLI Mixture deferred to GPU-attended cycle)
#821 feat D-2wiki supporting-fact-aware producer (WikiMultiCitedProducer with [Title #sent_id] citation prompt + tolerant parser; 24 unit tests)

After v0.4.4, the cycle γ 4-bench promise has research-tier-ready infrastructure for 4 of 4 cells (RGB + MuSiQue + ALCE + 2Wiki). v0.4.3's ⭐ infra-only caveat for ALCE / 2Wiki is closed.

Repository health (2 PRs)

PR Type Summary
#822 chore ruff F-class CI hygiene — 39 pre-existing F401/F541/F841 violations closed; 178 tests green
#828 docs Next-session entry handover doc + CLAUDE.md "Where to look next" pointer update

Self-correction example (12th wrong-fix-averted, first self-catch)

PR #824 originally claimed ⭐⭐⭐ "JAMES R@1 within ±0.05 of S2 token reference" (J = 0.673 at S3 publication vs J = 0.713 at S2 token, delta = 0.040). A per-category audit revealed current-contract R@10 = 0.0 across all 3 SUTs from single-template contract-title cluster collapse — the broken category coincidentally pulled J magnitude down to match S2.

PR #825 fixed the generator vocabulary, restored the broken category (R@10 = 1.0), revealed the honest J magnitude (0.845, delta +0.132 vs S2 token), and re-graded the verdict (⭐⭐⭐ pattern preserved + ⭐⭐ magnitude scenario-sensitive). PR #826 propagated the honest framing into preprint §5; PR #829 corrected residual abstract / leg 1 typos.

This is the 12th wrong-fix-averted in the JAMES cycle history and the first self-catch (the prior 11 were all user-catches). The S3.1 → preprint §5 → arXiv abstract self-correction loop is the strongest applied evidence of the feedback_oracle_phrase_artifacts measurement-side artefact rule: when overall magnitude looks like a coincidence, audit per-category before declaring ⭐⭐⭐.

What's NOT included (separate cycles)

  • LLM-grounded S3 publication run (v0.2.3b operator-gated; ~4-6h wall for 12 cells × 4 models)
  • D-alce research-tier measurement (operator-gated; HF cache ~1.5GB × 2, CPU ~3-5min × 2)
  • D-2wiki research-tier measurement (operator-gated; ~3-5min on local gemma4:e4b)
  • T5-XXL TRUE NLI Mixture integration (ALCE-official-grade verifier; GPU-attended)
  • HR full sweep n=100 (operator-attended)
  • TimeQA / TempReason / Microsoft GraphRAG SUT (operator data-download + license gated)
  • arXiv preprint LRB v0.2.5 submission (pre-flight in progress; this DOI will be cited in Acknowledgements)
  • Joint-piece collab arc with Robin Converse / Ali Afana (separate trajectory, not auto-linked per the eval-cycle-vs-collab-arc-separation rule)

Reproducing the results

# 1. Generate scenario fixtures (deterministic; no LLM; SHA-pinned)
python scripts/research/build_lrb_scenario_s1.py
python scripts/research/build_lrb_scenario_s2.py
python scripts/research/build_lrb_scenario_s3.py --scale smoke
python scripts/research/build_lrb_scenario_s3.py --scale dev
python scripts/research/build_lrb_scenario_s3.py --scale publication

# 2. Phase A (S1 current-only) + Phase B (S2 time-travel) token-mode
python scripts/research/lrb_run_phase_b.py --scenarios S1,S2 --k 10

# 3. v0.2.1 cross-model (S1+S2 × 4 models × token + llm-grounded)
PYTHONPATH=. python scripts/research/lrb_run_v021_cross_model.py \
  --scenarios S1,S2 \
  --modes token,llm-grounded \
  --models gemma4:e4b,gemma3:12b,mixtral:8x7b,claude-haiku-4-5

# 4. v0.2.3 S3 token-mode ladder (this release; deterministic; no LLM)
python scripts/research/lrb_run_s3.py --scale smoke
python scripts/research/lrb_run_s3.py --scale dev
python scripts/research/lrb_run_s3.py --scale publication

# 5. v0.2.3b S3 cross-model LLM-grounded (this release infrastructure;
#    operator-gated; expensive)
PYTHONPATH=. python scripts/research/lrb_run_v023b_s3_cross_model.py \
  --scale publication \
  --modes llm-grounded \
  --models gemma4:e4b,gemma3:12b,mixtral:8x7b,claude-haiku-4-5

# 6. v0.2.4 HR axis (RoBERTa primary + DeBERTa rescore)
PYTHONPATH=. python scripts/research/lrb_v024_hr_smoke.py
PYTHONPATH=. python scripts/research/lrb_v024_hr_rescore.py

# 7. Cycle γ 4-bench (RGB + MuSiQue + ALCE + 2Wiki)
python scripts/research/alce_smoke_run.py --verifier roberta-mnli --n 20
python scripts/research/alce_smoke_run.py --verifier deberta-v3-anli --n 20
python scripts/research/wiki2_smoke_run.py --cited --n 20

All result.json + bench.jsonl files land in reports/external/lrb/ and reports/external/{alce,2wiki,musique}/ deterministically (SHA-pinned per fixture; LLM-grounded mode inherits the ~10pp claude-API reproducibility band documented in PR #819).

Citation

@software{seo2026james044,
  author    = {Seo, Jiwon},
  title     = {PROJECT JAMES — v0.4.4 (LRB v0.2.3 S3 publication-scale +
               cycle γ 4-bench infrastructure closure)},
  year      = {2026},
  month     = {6},
  doi       = {10.5281/zenodo.XXXXXXXX},
  url       = {https://github.com/Hashevolution/James-RAG-Evol/releases/tag/v0.4.4},
  version   = {v0.4.4},
  publisher = {Zenodo}
}

(DOI populated by Zenodo after release publish.)

Pre-registrations (LOCKED before measurement, R5 rule)

Cycle Pre-reg doc
LRB Phase A (S1 current-only) docs/research/lrb-phase-a-smoke-preregistration-2026-06-11.md
LRB Phase B (S2 time-travel) docs/research/lrb-phase-b-time-travel-preregistration-2026-06-11.md
LRB v0.2.1 cross-model docs/research/lrb-v021-cross-model-preregistration-2026-06-11.md
LRB v0.2.4 HR axis docs/research/v024-hr-nli-axis-preregistration-2026-06-11.md
LRB v0.2.3 S3 scale (new) docs/research/lrb-v023-s3-publication-scale-preregistration-2026-06-12.md
LRB v0.2.3b S3 cross-model (new) docs/research/lrb-v023b-s3-cross-model-preregistration-2026-06-12.md
D-alce research-tier NLI (new) docs/research/cycle-gamma-d-alce-research-tier-nli-preregistration-2026-06-12.md
D-2wiki supporting-fact (new) docs/research/cycle-gamma-d-2wiki-supporting-fact-preregistration-2026-06-12.md

Each pre-reg is a Markdown LOCK doc committed before any measurement code or result.json artefact. All eight pre-regs (including the four inherited from v0.4.3) are committed in this repository and accessible at the v0.4.4 tag commit.

Test suite

After this release: ~200+ tests pass. Touched in v0.4.4:

  • tests/test_alce_nli_adapter.py — 19 tests (D-alce contract)
  • tests/test_wikimulti_cited_producer.py — 24 tests (D-2wiki parser + scorer round-trip)
  • tests/test_lrb_s3_generator.py — 27 tests (S3 vocabulary primitives + determinism + scale invariants)
  • Existing tests: 0 regression from any of the 11 PRs

Honest framing summary

The headline of v0.4.4 is the SCALE-axis extension of the V<N<J finding (LRB), not a JAMES win. Gap structure is the headline per LRB SPEC §4.6.

  • ✓ Pattern + gap scale-robust (publication-tier evidence)
  • ⚠ Absolute magnitudes scenario-sensitive (honest caveat; not a JAMES claim)
  • ✗ ActiveGraph (arXiv 2605.21997) is independent co-invention of the audit-native validity-window architecture; LRB does NOT claim novelty of the architecture, only of the benchmark and the cross-scale measurement
  • ✗ Multi-hop reasoning is saturated honest negative (3-SUT identical on MuSiQue)
  • ✗ EU AI Act references are anchor / framing, not compliance certification

This release is the data-availability anchor for the LRB v0.2.5 arXiv preprint. The preprint's Acknowledgements section will cite the v0.4.4 DOI assigned at Zenodo mint.

Assets 2
Loading