AI Translation Methods

🤖 AI Translation Methods & Pipeline Strategy

This project serves as an empirical engineering study to evaluate:

• Quality: How accurately can publicly available Large Language Models (LLMs) translate complex Sanskrit grammatical treatises across 35+ diverse target languages?
• Cost Efficiency: How can consumer-grade hardware and local LLM engines be combined to achieve zero-cost mass translation while preserving publication-grade accuracy?
• Format Integrity: Can legacy HTML manuscripts be algorithmically converted into standardized Markdown while strictly preserving Devanāgarī scripts, IAST transliterations, and nested grammar tables?

---

🌐 Target Language Matrix

The project encompasses 48 languages (with 35 fully completed at 100% clean files):

• Master Source: German (de)
• Global Major Languages: English (en), Chinese (zh-CN, zh), Hindi (hi), Indonesian (id), Spanish (es), Russian (ru), Arabic (ar)
• Complex Scripts & RTL: Tamil (ta), Telugu (te), Punjabi (pa), Hebrew (he), Persian (fa), Thai (th)
• Swiss National Languages: German (de), Italian (it), French (fr), Rumantsch (rm)
• Classical & Historical Languages: Latin (la), Ancient Greek (grc), Modern Greek (el), Ge'ez (gez)
• European & Slavic Languages: Dutch (nl), Afrikaans (af), Lithuanian (lt), Albanian (sq), Bulgarian (bg), Turkish (tr), Vietnamese (vi), Polish (pl), Czech (cs), Slovak (sk), Hungarian (hu), Finnish (fi), Romanian (ro), Ukrainian (uk)

---

1. Pipeline Architecture and Orchestration

The translation pipeline operates through strictly modularized components:

1. Orchestrator (lan_translate.py): Main entry point. Iterates through target languages according to priority rankings, manages file queues, controls runtime concurrency, and invokes file processing.
2. Single Source of Truth & QA Gate (translation_qa.py): Enforces absolute status criteria. A file is queued for translation/re-translation if:
   • The target file does not exist.
   • The target file is older than the master source (mtime check).
   • The target file contains German/English fallback remnants, unapproved Latin text, or syntax violations.
3. File Processor (file_processor.py): Separates metadata (YAML frontmatter) from markdown body, orchestrates chunking, manages the Translation Memory (TM) cache, applies post-processing sanitizers, and triggers auto-healing.
4. Chunking Engine (chunker.py): Splits documents into syntax-safe blocks under 1,500 characters, calculates cryptographic chunk hashes, and emits progress heartbeats (/tmp/payer_progress_heartbeat.json).
5. Inference Client (client.py): Connects to the dedicated local LLM server (http://nyx.local:8000) hosting the quantized Mixture-of-Experts engine.

---

2. Translation Memory (TM) & Caching Architecture

To eliminate redundant compute and prevent degradation of already clean translations, the system utilizes a deterministic caching layer:

2.1. Cryptographic Hashing

• Every German source chunk is stripped of boundary whitespace and hashed using MD5 (hashlib.md5(chunk.strip().encode('utf-8')).hexdigest()).
• Frontmatter is translated in a single atomic pass and keyed independently from body content.

2.2. Persistent Storage (.payer/tm/<lang>.json)

• Each target language maintains an isolated JSON key-value store in .payer/tm/<lang>.json.
• Key: Source chunk MD5 hash (32-character hexadecimal string).
• Value: Validated target language translation string.
• In-flight updates are atomic: each completed chunk updates the dictionary in memory and flushes to disk immediately.

2.3. Pre-Seeding & Dynamic Cache Invalidation

Before dispatching an inference request, the pipeline evaluates the cache:

• Pre-Seeding: When re-processing an existing file, clean chunks that pass residue scanning are automatically backfilled into the TM.
• Verification Gate: A cache hit is only accepted if:
  1. The entry does NOT start with ERROR:.
  2. The entry passes scan_german_residues() (verifying 0 residual German tokens).
• Dynamic Invalidation: If a cached entry is flagged as dirty or contains German remnants, the cache hit is rejected on the fly, a warning is logged ([!] TM Cache Invalidation), and the chunk is scheduled for fresh translation.

2.4. Vault Archiving

• During CI/CD runs on nataraja.local, .payer/tm/ is automatically archived to /home/marco/payer_backups/tm_backup_<timestamp>.tar.gz to ensure zero data loss.

---

3. The 5-Stage Escalation Hierarchy ("Stufen 1–5")

To balance zero operational cost with absolute translation fidelity, the pipeline implements a 5-stage escalation model:

[Stufe 1: Local MoE Inferenz]
       │ (Residuen oder Fehlversuch)
       ▼
[Stufe 2: Chirurgischer Chunk-Repass / Sub-Chunking]
       │ (Format- oder Lexik-Abweichung)
       ▼
[Stufe 3: Lokales Auto-Healing via Lingua]
       │ (Struktureller Kollaps / Persistente Fehler)
       ▼
[Stufe 4: Cloud-Modell Fallback (Sonnet)]
       │ (TM-Vergiftung / Translation Deadlock)
       ▼
[Stufe 5: Autonomes Stufe-5-Manöver (TM Purge & Force Rebuild)]

Stufe 1: Primary Local MoE Inference

• Model: mlx-community/Qwen3.6-35B-A3B-4bit-DWQ running on nyx.local:8000 (Apple Silicon M4, 32GB Unified Memory).
• MoE Efficiency: The "A3B" architecture activates only 3 billion parameters per token out of 35B total parameters. This bypasses memory bandwidth bottlenecks and achieves steady throughput of ~20 tokens/second.
• Single-Process Constraint: A hard operating system constraint ensures that only one translation worker runs across the network (ps aux | grep lan_translate). This guarantees 100% VRAM allocation without GPU context-switching penalties.
• Cost: 100% cost-free, private local inference.

Stufe 2: Surgical Chunk-Level Fallback & Adaptive Sub-Chunking

• Chunk-by-Chunk Diffing: Large documents such as wortliste.md or glossar.md contain thousands of lines and up to 50+ chunks. Rather than discarding entire files when a single section degrades, Stufe 2 isolates the exact chunk index that failed.
• Adaptive Sub-Chunking: If an individual chunk fails or returns an error, the chunk is automatically bisected into smaller units (down to 500 characters) and retried sequentially before escalating. Clean chunks remain untouched in the TM.

Stufe 3: Statistical Residue Detection & Local Auto-Healing

• Two-Tier Residue Scanner (scan_german_residues):
  1. Regex Filter: Checks for unallowed German grammatical terms (e.g., Akkusativ, Passiv, Stammabstufung, Wortbildung).
  2. Statistical Language Detection: Evaluates paragraphs using the Lingua natural language classifier to detect residual German prose against target language baselines.
• Targeted Auto-Heal Prompt: When residues are detected, the system does not re-translate the entire file. It extracts the flagged segments and issues a targeted patch prompt instructing the model to repair only the identified remnants while preserving existing Devanāgarī and structural markers.
• Back-Sync: The repaired chunk is verified and written directly back into .payer/tm/<lang>.json.

Stufe 4: Cloud Frontier Fallback

• Escalation: Reserved for complex edge cases where local open-weight inference repeatedly collapses (e.g., highly nested 12-column Markdown tables or complex morphological Sanskrit breakdowns in extreme low-resource target scripts).
• Model: Anthropic Claude Sonnet (via OpenRouter API), configured as the Stufe 4 fallback engine.
• Granular Invocation: Dispatched strictly at the chunk level to minimize API consumption.

Stufe 5: The "Stufe 5 Manöver" (Autonomous TM Deadlock Recovery)

• Deadlock Condition: In automated mass translation runs, a language can enter a deadlock state when:
  • Untranslated German fragments or ERROR: strings were inadvertently persisted into .payer/tm/<lang>.json during earlier interrupted sessions.
  • The orchestrator reads the dirty TM, writes a target file that fails translation_qa.py, and the QA gate rejects the file, creating an infinite retry loop.
• The Stufe 5 Protocol:
  1. Detection: Identified when a language reaches an execution threshold without progress or is flagged by QA with persistent remnants.
  2. Automated TM Purge (scratch/llm_fix_tms.py): The script loads .payer/tm/<lang>.json, iterates over all cached values, identifies any residual German terminology using ALL_TERMS and LICENSES_PHRASES, and translates/purges the dirty tokens directly inside the TM database.
  3. Forced Compilation: The pipeline executes python3 scripts/lan_translate.py --lang <lang> --force, forcing reassembly from the sanitized TM entries.
  4. Verification Gate: get_translation_queue(lang) in translation_qa.py is invoked to verify that queue length drops to 0 (140/140 files clean, 100.0%).

---

4. Quality Control & Protection Invariants

All translation tiers must strictly uphold the project's formatting invariants:

1. Devanāgarī Script Rule: Red signal tags (⟪...⟫) and signal styling (sig[...]) are applied exclusively to Devanāgarī text. Latin transliteration (IAST), German/English glosses, and punctuation must remain unstyled. Devanāgarī text must never be italicized (*...*).
2. Container Colon Depth Parity: Container syntax (::: grammar-box, ::: indent, ::: note-box) must maintain proper nesting depth. Outer containers enclosing inner blocks must automatically upgrade colons (:::: grammar-box).
3. Table Formatting: Multi-line table cells must use the :br token on a single line. Raw HTML tags (<br>, <div>) are strictly forbidden and stripped by scripts/purge_html.py.
4. TOTALBREMSE Invariant: Master source files (docs/lektionen/*.md) and languages that have achieved 100% completion (0 fallbacks, queue length 0) are strictly read-only. Automated scripts are blocked from modifying completed language directories.

4.5. Engineering Governance: Regular Code Reviews via Claude Opus

The translation pipeline infrastructure (lan_translate.py, translation_qa.py, file_processor.py, chunker.py) is maintained under continuous engineering governance. To prevent edge-case regressions and race conditions during high-volume translation runs, regular, rigorous code reviews are conducted using Anthropic Claude Opus. These reviews focus on:

• Boundary conditions in text chunking and sentence boundary recognition.
• Thread-safety, atomic cache updates, and Translation Memory corruption defenses.
• Statistical language detection thresholds and target-script regex bypasses.
• Strict enforcement of the TOTALBREMSE invariant protecting completed translations.

---

5. End-to-End Sequence Diagram

The following diagram details the lifecycle of a translation request across all 5 escalation stages:

sequenceDiagram
    autonumber
    participant CLI as lan_translate.py
    participant QA as translation_qa.py
    participant Proc as file_processor.py
    participant TM as TM Cache (.payer/tm/)
    participant L1 as Stufe 1: Local MoE (nyx.local)
    participant L3 as Stufe 3: Auto-Heal (Lingua)
    participant L4 as Stufe 4: Sonnet Cloud Fallback
    participant L5 as Stufe 5: llm_fix_tms.py

    CLI->>QA: get_translation_queue(lang)
    QA-->>CLI: List of unverified / fallback files
    
    loop For each queued file
        CLI->>Proc: translate_file(file, force=False)
        Proc->>Proc: Split YAML Frontmatter & Body
        Proc->>Proc: chunk_content(body) -> List[Chunks <= 1500 chars]
        
        loop For each Chunk
            Proc->>Proc: Compute MD5(chunk)
            Proc->>TM: Lookup MD5 in <lang>.json
            
            alt Cache Hit & Clean (Passes QA)
                TM-->>Proc: Return cached translation
            else Cache Miss or Dirty (Residues detected)
                Proc->>L1: Stufe 1: Query Qwen3.6-35B-A3B
                L1-->>Proc: Raw translated chunk
                
                alt Sub-Chunking needed (Stufe 2)
                    Proc->>Proc: Stufe 2: Bisect into <= 500 char sub-chunks & retry
                end
                
                Proc->>Proc: sanitize_translation_output()
                Proc->>QA: scan_german_residues(chunk)
                
                alt Residues detected (Stufe 3)
                    QA-->>Proc: Detected German remnants
                    Proc->>L3: Stufe 3: Targeted Auto-Heal patch query
                    L3-->>Proc: Cleaned translation
                end
                
                alt Structural collapse persists (Stufe 4)
                    Proc->>L4: Stufe 4: Query Claude Sonnet Fallback
                    L4-->>Proc: High-precision formatted chunk
                end
                
                Proc->>TM: Write clean chunk (MD5 -> Translation)
            end
        end
        
        Proc->>Proc: Reassemble Frontmatter + Chunks
        Proc->>Proc: Write target file to disk
        Proc->>QA: check_has_de_phrases(target_file)
        
        alt Pipeline Deadlock (Persistent TM Poisoning)
            QA-->>CLI: File rejected by QA
            CLI->>L5: Stufe 5: Run llm_fix_tms.py (Purge dirty TM entries)
            L5->>TM: Cleanse & overwrite .payer/tm/<lang>.json
            CLI->>Proc: Re-run translate_file(file, force=True)
            Proc->>QA: Re-verify against clean TM -> PASS (Exit 0)
        end
    end

Loading