Limits and Next Steps

Limits and Next Steps

Limits

The most important limit is that this is a small demo on invented data, and its numbers describe that demo and nothing wider. The memory holds 133 synthetic units, the gold set holds 41 queries written for this repository, and the model ran once. The figures are deterministic for this data and code, which makes them reproducible; however, reproducible is a different property from general, and no claim here extends to real translation memories.

The second limit concerns what QA can see. Six string checks confirm that a translation kept its placeholders, tags, and numbers and used the listed terms, and they say nothing about whether it reads well or means the same thing. Because the highest score these rules allow on this gold set is 97.6%, and because the model often saw the answer, the Qwen result is evidence of mechanical compliance. It does not show translation quality.

Area	Limit
Retrieval	Bounded by a 384-dimensional MiniLM model and 133 segments. The best recall@1 is 0.5854
Hybrid	Worse than plain vector search at k of 3 or more on this gold set; it helps only when queries share keywords with the sources
Term base	Covers zh-CN only. zh-TW and Thai units exist in the memory, but no rule enforces their terms
Term matching	Substring match, so a term inside a longer term counts as required. This wrongly fails Health Potion
Mock provider	Copies the nearest match and does not translate
Baseline	No run compares the model with and without retrieved examples
Model providers	The translate command accepts mock, openai, and openai-compatible. The ollama option exists only on eval
Qdrant	Local mode, single process, with no persistence guarantees beyond the directory
Platforms	Tested on Windows with Python 3.12

Next steps

These are ordered by how much each one would improve what the evaluation can honestly say.

1. Fix the term rule. A required term should be dropped when a longer matched term contains it, and that one change would remove the false failure on Health Potion. The two wording errors in EXPERIMENTS.md and the README should be corrected at the same time.
2. Run a true baseline. The same model with no retrieved examples would measure what retrieval adds, which the current mock comparison cannot show.
3. Hold out the answer. Excluding the expected unit from the few-shot examples would test the model on text it has not been shown, and that would make the 97.6% mean something stronger.
4. Add a meaning check. A human review sample, or a reviewed judge model beside the rules, would address the largest thing QA cannot see.
5. Build a keyword gold set. Hybrid retrieval deserves a fair trial on queries that share vocabulary with the memory, which would settle whether it should stay.
6. Extend terms to zh-TW and Thai. The memory already holds these variants, and the QA rules should cover them.

A production system would need more than this list: a larger, versioned memory with metadata, batch ingestion, authentication on the vector store, a human review workflow, and continuous evaluation against a held-out set. No dates are attached to any of this, because the value of the project lies in the method, and the method should stay small enough to check by hand.