How It Works

How It Works

The idea

Retrieval-augmented translation treats a translation memory as evidence rather than as a lookup table. A traditional fuzzy match compares strings, so a rephrased sentence with the same meaning can score poorly, whereas an embedding compares meaning, which lets a new segment find approved work even when the words differ. The difficulty is that a model given that evidence can still ignore it, and a retrieval step can still return the wrong evidence. Therefore the pipeline measures retrieval and generation separately, and it checks every output with rules that do not depend on any model.

All data is synthetic and invented for this repository: the fantasy game "Emberveil Legends", with 133 translation units (120 zh-CN, 8 zh-TW, and 5 Thai) and 32 term entries, each with a forbidden alternative.

Architecture

flowchart LR
    TMX[data/tm.tmx] --> PARSE[tmx.py]
    TBX[data/terms.tbx] --> TERMS[tbx.py]
    PARSE --> EMB[index.py: embed with MiniLM]
    EMB --> QD[(Qdrant local mode)]
    QD --> RET[search: vector, optional filter, optional hybrid]
    RET --> PROMPT[prompts.py: v1.0 text or v2.0 JSON]
    TERMS --> PROMPT
    PROMPT --> LLM[llm.py: mock or Ollama]
    LLM --> RES[schema.py: TranslationResult]
    RES --> QA[qa.py: deterministic checks]
    RET --> EV[evaluate.py: recall, MRR, QA, wall time]
    QA --> EV
    EV --> OUT[results/*.json]
    OUT --> REP[report.py: docs/index.html]

Loading

Main parts

Part	Code	Role
Parsers	tmx.py, tbx.py	Read TMX 1.4 and TBX into translation units and term entries
Index	index.py	Embeds source segments with paraphrase-multilingual-MiniLM-L12-v2 and stores them in Qdrant local mode
Hybrid ranking	retrieve.py	Pure-Python Okapi BM25 (k1 = 1.5, b = 0.75) and reciprocal rank fusion (k = 60)
Prompts	prompts.py	Versioned few-shot prompts: the top two matches as examples, plus the approved terms
Providers	llm.py	A deterministic mock that copies the top match, and an OpenAI-compatible client for local Ollama
QA	qa.py	Six pure string checks on a source and translation pair
Evaluation	evaluate.py	Recall@k, MRR, QA pass rate, and wall time, written to a results file
Report	report.py	Builds one self-contained static page from results/*.json

The six QA checks are placeholder preservation, tag preservation, character limit, required term used, forbidden term absent, and number preservation. A translation passes only when all six pass, and because each check is a pure function of its inputs, two runs on the same text cannot disagree.

Walkthrough: one query through three configurations

One gold item shows why the project measures each stage on its own. The query is a checkpoint was reached while exploring, and the expected unit is dlg-016, whose source is Checkpoint reached. The memory holds two translations of that source, one for zh-CN and one for zh-TW, and the term base requires 检查点. Every value below comes from the committed results/*.json files.

Configuration	Expected unit's rank	Top match	Output	QA
Baseline, mock (eval.json)	2	dlg-016@zh-TW, 0.8686	已到達檢查點	Fails required_term_used
zh-CN filter, mock (eval_zhcn.json)	1	dlg-016, 0.8686	已到达检查点	Passes
Baseline retrieval, Qwen (eval_qwen.json)	2	dlg-016@zh-TW, 0.8686	已到达检查点	Passes

Baseline. The vector search finds the right source and ranks its zh-TW variant first, because the two variants share one English source and therefore one embedding. The mock copies whatever sits on top, so it returns Traditional Chinese, and the required-term check fails because 檢查點 is not the approved form 检查点. The retrieval was close to correct; however, the failure is real, because the output would have shipped the wrong script.

zh-CN filter. A payload filter on tgt_lang removes the other variants before ranking, so dlg-016 rises from second to first and the same mock now passes. Nothing changed except which candidates competed. This pattern repeated across the gold set, which is why the filter improved every recall figure and became the best measured configuration.

Qwen. The real model received the same unfiltered retrieval as the baseline, with both variants in its two examples, yet it returned the Simplified form, because the prompt asks for zh-CN and lists the approved term. The model repaired a retrieval weakness that the mock could not. That is a fair result; even so, the example also shows how much the prompt gave the model, a theme that Evidence takes up in detail.

The three rows together explain the structure of the project. Retrieval quality and output quality are separate questions, and a single end-to-end score would have hidden the difference between them.