Evidence

Evidence

Quality here is checked in two ways: a test suite for the code, and a gold-set evaluation for the behavior. All results below come from the committed results/*.json files or from a run on 2026-11-06, as marked. The five mock evaluations were repeated on a clean copy on that date, and each one reproduced the recall, MRR, and QA figures in its committed file exactly. The Qwen evaluation was not repeated; its figures come from results/eval_qwen.json.

Tests

From the repository's own .venv, on Windows with Python 3.12.10:

$ .venv\Scripts\python -m pytest -q
.........................................................                [100%]
57 passed in 19.06s

File	Tests	Covers
tests/test_llm.py	14	Mock provider, JSON extraction, thinking text, fences, and retry
tests/test_qa.py	9	The six QA checks
tests/test_report.py	7	The generated results page
tests/test_retrieve.py	7	BM25 and reciprocal rank fusion
tests/test_index.py	5	Index build, search by meaning, and a missing index
tests/test_search_options.py	4	Target-language filter and hybrid option
tests/test_tmx.py	4	TMX parsing
tests/test_tbx.py	3	TBX parsing
tests/test_cli.py	2	A smoke run of the commands, and honest reporting of a QA failure
tests/test_readme.py	2	README and results page agree with results/*.json

The last row is unusual and deliberate. test_readme_and_results_page_numbers_match_results_json fails when a figure from results/*.json goes missing from the README or the results page, which ties the published tables to the files that produced them.

What was measured

The gold set holds 41 paraphrased English queries, each naming the unit it should find, and data/gold_zh.jsonl adds 10 Chinese queries searched against the English sources. Recall@k is the share of queries with the expected unit in the top k, and MRR is the mean reciprocal rank. The QA step then runs the full pipeline on the expected unit's source segment and applies the six rules.

Configuration	File	R@1	R@3	R@5	R@10	MRR	QA pass
Baseline	eval.json	0.5122	0.6829	0.7317	0.8049	0.6146	0.8049 (mock)
zh-CN filter	eval_zhcn.json	0.5854	0.7317	0.7561	0.8293	0.6755	0.8293 (mock)
Hybrid RRF and BM25	eval_hybrid.json	0.5366	0.5610	0.6098	0.6585	0.5632	0.8780 (mock)
zh-CN and hybrid	eval_zhcn_hybrid.json	0.5366	0.5854	0.6098	0.6585	0.5665	0.8780 (mock)
Chinese queries	eval_crosslang.json	0.5000	0.5000	0.7000	0.8000	0.5593	0.7000 (mock)
Baseline retrieval, Qwen	eval_qwen.json	0.5122	0.6829	0.7317	0.8049	0.6146	0.9756 (Qwen)

The Qwen run used qwen2.5-coder:14b through local Ollama with prompt version 2.0. It took 544.25 seconds for 41 items, about 13 seconds each, and it produced 0 bad JSON replies. Per-check pass rates were 1.0 for placeholders, tags, character limit, numbers, and forbidden terms, and 0.9756 for required terms.

What the 97.6% can and cannot support

The headline comparison is 40 of 41 for Qwen against 33 of 41 for the retrieval-only baseline, and those are the numbers the run produced. However, three facts narrow what they mean, and all three follow from the committed code and results. No model ran to establish them.

The ceiling is 97.6%. Running the QA rules on the memory's own approved translations of the 41 gold items gives 40 passes out of 41. The rules therefore cannot award more than 97.6% on this gold set, and Qwen reached that ceiling; no room exists above it for any provider to show improvement.

The two rows do not face the same task. The mock copies the top match for the paraphrased query, and it never sees the source it is scored against. Of its 8 failures, 6 copied the wrong unit, 1 copied the right unit in the wrong script, and 1 is the QA defect described below. In eval.json, for example, heal my wounds retrieved item-health-potion and copied 生命药水 for the source Restored %d HP. The model, by contrast, receives the exact source text in its prompt. The gap therefore compares "copy the nearest match" with "read the source and translate", and it is not a measure of how much a model adds to a translation memory on equal terms.

The answer was often in the prompt. The expected unit appeared among the two few-shot examples for 27 of the 41 items, and Qwen's output matched the approved translation exactly on 33 of 41, including 24 of those 27. The remaining 8 outputs differ in small ways, such as 加载游戏 for 读取游戏 and 背包 for 物品栏, which the QA rules accept because no term entry covers them. The result shows that this model keeps placeholders and uses listed terms under these conditions. It says little about unseen text, and it says nothing about translation quality in the sense a human reviewer would mean.

Known failures

One QA rule has a defect, and it produced the only Qwen failure. The failed item is item-health-potion, with source Health Potion. Qwen returned 生命药水, which is the approved term and is identical to the memory's own translation, yet the check failed with missing term targets: ['生命值']. The cause is in required_terms(), which matches by substring: the source contains both Health Potion and Health, so the rule demands both targets. The approved translation fails its own QA for the same reason, and it is the only one of the 120 zh-CN units that does. EXPERIMENTS.md describes this failure as the model not using 生命药水; the detail string shows that the missing term is 生命值, so that sentence needs correcting.

Hybrid retrieval was not worse at every k. EXPERIMENTS.md and the README say so, and the summary holds for recall at 3, 5, and 10 and for MRR. At k = 1 the baseline is 0.5122 and hybrid is 0.5366, a gain of one query. Against the zh-CN filter, which is the fair comparison, hybrid loses at every k. The decision to keep it as an option stands; the wording overstates the case.

Eight queries miss entirely at k = 10. Queries such as open my bag and pick how hard the game is do not retrieve their units in the baseline. The zh-CN filter rescues one of the eight and leaves seven, and the best recall@1 in any configuration is 0.5854.

The mock's higher QA under hybrid is an artifact. Hybrid scored 0.8780 on QA with worse recall, because a different top match happened to be copied. The experiment log records this and does not count it as an improvement.

Check it yourself in 5 minutes

The test suite and the mock evaluations need no model. The first index run downloads about 90 MB of embedding weights. Results go to a separate folder here, so the committed files stay untouched.

git clone https://github.com/dboles99/tm-rag-copilot.git
cd tm-rag-copilot
python -m venv .venv
.venv\Scripts\python -m pip install -r requirements.txt
.venv\Scripts\python -m pytest -q
mkdir out
.venv\Scripts\python -m tm_rag eval --k 1 3 5 10 --target-lang zh-CN --results out\zhcn.json

The last command should print MRR: 0.6755 and QA pass rate (mock provider): 0.8293.

To confirm the ceiling and the exact-match counts from the committed results, with no model:

import json
from tm_rag.qa import all_passed, run_qa
from tm_rag.tbx import parse_tbx
from tm_rag.tmx import parse_tmx

units = {u.id: u for u in parse_tmx("data/tm.tmx")}
terms = parse_tbx("data/terms.tbx")
items = json.load(open("results/eval_qwen.json", encoding="utf-8"))["items"]
refs = [units[i["expected_id"]] for i in items]
print(sum(all_passed(run_qa(u.source, u.target, terms)) for u in refs))  # 40
print(sum(i["translation"] == u.target for i, u in zip(items, refs)))    # 33
print(sum(bool(i["rank"]) and i["rank"] <= 2 for i in items))            # 27

To follow the experiments in order, read EXPERIMENTS.md, keeping in mind the two corrections above.