Repository navigation
Design Decisions
Five decisions shaped this project, and two of them record things that did not work. The sources are EXPERIMENTS.md, the README, and the committed results. The method throughout was small changes, measured on committed data, with no gold set or threshold tuned to improve a number.
Problem. Each English source has up to three variants in the memory, and identical sources embed identically, so the zh-TW and Thai variants of one sentence crowded the top ranks and pushed other units out.
Options. One collection per target language; a query-time payload filter on tgt_lang; or de-duplication by source.
Choice. A single collection with a Qdrant payload filter, exposed as --target-lang. Every recall figure and the MRR improved over the baseline, with recall@10 rising from 0.8049 to 0.8293 and MRR from 0.6146 to 0.6755. This is the best measured configuration.
Trade-off. The filter needs the caller to state the target language, and it does little for queries that miss for reasons of meaning rather than crowding: 7 of the 41 still miss at k = 10. The gain is modest, because it removes one cause of failure and not the main one.
What would change it. Per-language collections would make sense at a scale where a shared index becomes slow; 133 units is nowhere near that point.
Problem. Keyword search is the standard complement to embeddings, and the obvious next experiment was to combine them.
Options. Vector search alone, or reciprocal rank fusion of vector search with BM25. BM25 and RRF were written in pure Python in retrieve.py, so the behavior stays easy to test.
Choice. Keep it as the --hybrid option and leave it off by default. Recall@10 fell from 0.8049 to 0.6585 and MRR from 0.6146 to 0.5632, and with the zh-CN filter added, recall@10 fell from 0.8293 to 0.6585. Recall@1 did rise from 0.5122 to 0.5366 against the unfiltered baseline, by one query. The explanation recorded in EXPERIMENTS.md is that the gold queries are paraphrases with little keyword overlap, so BM25 ranks mostly noise and fusion dilutes the vector signal.
Trade-off. Fusion gives each ranking equal weight, and on this gold set that is the wrong weight. The result depends on the gold set, however: a query set that shared vocabulary with the memory could reverse it. The negative result is therefore specific rather than general, and the option stays available for that reason.
What would change it. A second gold set built from realistic keyword-style queries would show whether hybrid earns a place.
Problem. Every output needs a check, and a model judge would add cost, delay, and a second source of unexplained variation.
Options. A language-model judge; string rules; or both.
Choice. Six pure string checks in qa.py. They are reproducible, fast, and need no model, and they cover the failures that break a shipped game: a lost placeholder, a lost tag, a changed number, an unapproved term.
Trade-off. The rules know nothing about meaning. A translation can be wrong and pass, and Evidence shows that the rules can also fail a correct one. The substring match in required_terms() treats Health inside Health Potion as a second required term, which is the single defect behind the only Qwen failure. Simple rules remain the right foundation, because that defect was easy to find and is easy to fix, which would not be true of a judge's opinion.
What would change it. A review stage for meaning would need a human or a judge model beside these rules, not in place of them.
Problem. A demo that needs a model to run cannot be checked by a stranger in five minutes, whereas a demo that fakes a model misleads the reader.
Options. Require a model; ship canned outputs; or include a deterministic stand-in that says what it is.
Choice. MockProvider copies the closest match and adds a note saying so. The tests, the retrieval evaluation, and the mock QA figures all run offline. Real providers run through an OpenAI-compatible client, with --provider ollama for a local server.
Trade-off. The mock's QA figures are easy to misread as a translation baseline, when they mainly reflect retrieval, as Evidence explains. The "retrieval-only baseline" label carries that caveat in its name, but only for a careful reader.
What would change it. A real baseline, such as the same model with no retrieved examples, would replace the mock in the headline comparison.
Problem. A run with a real model produces malformed replies, and a report written by hand drifts from its data.
Options. Drop bad replies, retry until something parses, or count them; write results by hand, or generate them.
Choice. Thinking text and fenced JSON are stripped, a bad reply gets one retry, and a reply that is still bad counts as a failure in bad_json_count. The results page is generated from results/*.json, and a test checks that the README and the page carry the same figures.
Trade-off. One retry is a small allowance, and a less cooperative model would score worse for it, which is the intended behavior. The test covers numbers and not prose, so it could not catch the two wording errors listed in Evidence.
What would change it. Those two errors argue for generating the summary sentences from the data as well.
Designed and authored by Daniel Boles; implemented with coding agents. Apache-2.0.